Metadata-Version: 1.0
Name: mrq
Version: 0.0.2
Summary: Mongo Redis Queue
Home-page: http://github.com/pricingassistant/mrq
Author: Pricing Assistant
Author-email: contact@pricingassistant.com
License: MIT
Description: MRQ
        ===
        
        Mongo Redis Queue - A distributed worker task queue in Python
        
        /!\ MRQ is not yet ready for public use. Soon!
        
        Why?
        ====
        
        MRQ is an opinionated task queue. It aims to be simple and beautiful like http://python-rq.org while having performance close to http://celeryproject.org
        
        MRQ was first developed at http://pricingassistant.com and its initial feature set matches the needs of worker queues with heterogenous jobs (IO-bound & CPU-bound, lots of small tasks & a few large ones).
        
        The main features of MRQ are:
        
         * **Simple code:** We originally switched from Celery to RQ because Celery's code was incredibly complex and obscure ([Slides](http://www.slideshare.net/sylvinus/why-and-how-pricing-assistant-migrated-from-celery-to-rq-parispy-2)). MRQ should be as easy to understand as RQ and even easier to extend.
         * **Great dashboard:** Have visibility and control on everything: queued jobs, current jobs, worker status, ...
         * **Per-job logs:** Get the log output of each task separately in the dashboard
         * **Gevent worker:** IO-bound tasks can be done in parallel in the same UNIX process for maximum throughput
         * **Supervisord integration:** CPU-bound tasks can be split across several UNIX processes with a single command-line flag
         * **Job management:** You can retry, requeue, cancel jobs from the code or the dashboard.
         * **Performance:** Bulk job queueing, easy job profiling
         * **Easy configuration:** Every aspect of MRQ is configurable through command-line flags or a configuration file
         * **Job routing:** Like Celery, jobs can have default queues, timeout and ttl values.
         * **Thorough testing:** Edge-cases like worker interrupts, Redis failures, ... are tested inside a Docker container.
         * **Builtin scheduler:** Schedule tasks by interval or by time of the day
         * **Greenlet tracing:** See how much time was spent in each greenlet to debug CPU-intensive jobs.
        
        Performance
        ===========
        
        On a MacbookPro, we see 1300 jobs/second in a single worker process with very simple jobs that store results, to measure the overhead of MRQ. However what we are really measuring there is MongoDB's write performance.
        
        Tests
        =====
        
        Testing is done inside a Docker container for maximum repeatability.
        We don't use Travis-CI or friends because we need to be able to kill our process dependencies (MongoDB, Redis, ...) on demand.
        
        Therefore you need to ([install docker](https://www.docker.io/gettingstarted/#h_installation)) to run the tests.
        If you're not on an os that supports natively docker, don't forget to start up your VM and ssh into it.
        
        ```
        $ make test
        ```
        
        You can also open a shell inside the docker (just like you would enter in a virtualenv) with:
        
        ```
        $ make docker (if it wasn't build before)
        $ make ssh
        ```
        
        Use in your application
        =======================
        
        add MRQ to your environment
        
        
        PyPy
        ====
        
        Earlier in its development MRQ was tested successfully on PyPy but we are waiting for better PyPy+gevent support to continue working on it, as performance was worse than CPython.
        
        
        Useful third-party utils
        ========================
        
        * http://superlance.readthedocs.org/en/latest/
        
        
        TODO
        ====
        
        **alpha**
        
         * Max Retries
         * MongoDB/Redis disconnect tests in more contexts (long-running queries, ...)
        
        **public beta**
        
         * Full linting
         * Code coverage
         * Public docs
         * PyPI
        
        **whishlist**
        
         * task progress
         * ETAs / Lag stats for each queue + graphes
         * uniquestarted/uniquequeued via bulk sets?
         * Base cleaning/retry tasks: move
         * Current greenlet traces in dashboard
         * Move monitoring in a thread to protect against CPU-intensive tasks
         * Bulk queues
         * Tasksets
         * Full PyPy support
         * Search in dashboard
        
        Credits
        =======
        
        Inspirations:
         * RQ
         * Celery
        
        JS libraries used in the Dashboard:
         * http://backbonejs.org
         * http://underscorejs.org
         * http://requirejs.org
         * http://momentjs.com
         * http://jquery.com
         * http://datatables.net
         * https://github.com/Jowin/Datatables-Bootstrap3/
         * https://github.com/twbs/bootstrap
        
        ... as well as all the Python modules in requirements.txt!
Keywords: worker,task,distributed,queue,asynchronous
Platform: any
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 2.7
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Other Environment
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Topic :: Utilities
