Pipelines
=========

This runner's purpose in life is to process messages that have been accepted
for posting, applying any modifications and also sending copies of the message
to the archives, digests, nntp, and outgoing queues.  Pipelines are named and
consist of a sequence of handlers, each of which is applied in turn.  Unlike
rules and chains, there is no way to stop a pipeline from processing the
message once it's started.

    >>> from mailman.app.lifecycle import create_list
    >>> mlist = create_list(u'xtest@example.com')
    >>> mlist.web_page_url = u'http://lists.example.com/archives/'
    >>> mlist.pipeline
    u'built-in'
    >>> from mailman.app.pipelines import process


Processing a message
--------------------

Messages hit the pipeline after they've been accepted for posting.

    >>> msg = message_from_string("""\
    ... From: aperson@example.com
    ... To: xtest@example.com
    ... Subject: My first post
    ... Message-ID: <first>
    ...
    ... First post!
    ... """)
    >>> msgdata = {}
    >>> process(mlist, msg, msgdata, mlist.pipeline)

The message has been modified with additional headers, footer decorations,
etc.

    >>> print msg.as_string()
    From: aperson@example.com
    To: xtest@example.com
    Message-ID: <first>
    Subject: [Xtest] My first post
    X-BeenThere: xtest@example.com
    X-Mailman-Version: ...
    Precedence: list
    List-Id: <xtest.example.com>
    List-Unsubscribe:
        <http://lists.example.com/archives/listinfo/xtest@example.com>,
        <mailto:xtest-leave@example.com>
    List-Archive: <http://www.example.com/pipermail/xtest@example.com>
    List-Post: <mailto:xtest@example.com>
    List-Help: <mailto:xtest-request@example.com?subject=help>
    List-Subscribe:
        <http://lists.example.com/archives/listinfo/xtest@example.com>,
        <mailto:xtest-join@example.com>
    <BLANKLINE>
    First post!
    <BLANKLINE>

And the message metadata has information about recipients and other stuff.
However there are currently no recipients for this message.

    >>> sorted(msgdata.items())
    [('original_sender', u'aperson@example.com'),
     ('origsubj', u'My first post'),
     ('recips', set([])),
     ('stripped_subject', <email.header.Header instance at ...>)]

And the message is now sitting in various other processing queues.

    >>> from mailman.tests.helpers import get_queue_messages
    >>> from mailman.configuration import config
    >>> messages = get_queue_messages(config.ARCHQUEUE_DIR)
    >>> len(messages)
    1
    >>> print messages[0].msg.as_string()
    From: aperson@example.com
    To: xtest@example.com
    Message-ID: <first>
    Subject: [Xtest] My first post
    X-BeenThere: xtest@example.com
    X-Mailman-Version: ...
    Precedence: list
    List-Id: <xtest.example.com>
    List-Unsubscribe:
        <http://lists.example.com/archives/listinfo/xtest@example.com>,
        <mailto:xtest-leave@example.com>
    List-Archive: <http://www.example.com/pipermail/xtest@example.com>
    List-Post: <mailto:xtest@example.com>
    List-Help: <mailto:xtest-request@example.com?subject=help>
    List-Subscribe:
        <http://lists.example.com/archives/listinfo/xtest@example.com>,
        <mailto:xtest-join@example.com>
    <BLANKLINE>
    First post!
    <BLANKLINE>
    >>> print sorted(messages[0].msgdata.items())
    [('_parsemsg', False), ('original_sender', u'aperson@example.com'),
     ('origsubj', u'My first post'),
     ('received_time', ...), ('recips', set([])),
     ('stripped_subject', <email.header.Header instance at ...>),
     ('version', 3)]

This mailing list is not linked to an NNTP newsgroup, so there's nothing in
the outgoing nntp queue.

    >>> messages = get_queue_messages(config.NEWSQUEUE_DIR)
    >>> len(messages)
    0

This is the message that will actually get delivered to end recipients.

    >>> messages = get_queue_messages(config.OUTQUEUE_DIR)
    >>> len(messages)
    1
    >>> print messages[0].msg.as_string()
    From: aperson@example.com
    To: xtest@example.com
    Message-ID: <first>
    Subject: [Xtest] My first post
    X-BeenThere: xtest@example.com
    X-Mailman-Version: ...
    Precedence: list
    List-Id: <xtest.example.com>
    List-Unsubscribe:
        <http://lists.example.com/archives/listinfo/xtest@example.com>,
        <mailto:xtest-leave@example.com>
    List-Archive: <http://www.example.com/pipermail/xtest@example.com>
    List-Post: <mailto:xtest@example.com>
    List-Help: <mailto:xtest-request@example.com?subject=help>
    List-Subscribe:
        <http://lists.example.com/archives/listinfo/xtest@example.com>,
        <mailto:xtest-join@example.com>
    <BLANKLINE>
    First post!
    <BLANKLINE>
    >>> print sorted(messages[0].msgdata.items())
    [('_parsemsg', False), ('listname', u'xtest@example.com'),
     ('original_sender', u'aperson@example.com'),
     ('origsubj', u'My first post'), ('received_time', ...),
     ('recips', set([])),
     ('stripped_subject', <email.header.Header instance at ...>),
     ('version', 3)]

There's now one message in the digest mailbox, getting ready to be sent.

    >>> from mailman.tests.helpers import digest_mbox
    >>> digest = digest_mbox(mlist)
    >>> sum(1 for mboxmsg in digest)
    1
    >>> print list(digest)[0].as_string()
    From: aperson@example.com
    To: xtest@example.com
    Message-ID: <first>
    Subject: [Xtest] My first post
    X-BeenThere: xtest@example.com
    X-Mailman-Version: ...
    Precedence: list
    List-Id: <xtest.example.com>
    List-Unsubscribe:
        <http://lists.example.com/archives/listinfo/xtest@example.com>,
        <mailto:xtest-leave@example.com>
    List-Archive: <http://www.example.com/pipermail/xtest@example.com>
    List-Post: <mailto:xtest@example.com>
    List-Help: <mailto:xtest-request@example.com?subject=help>
    List-Subscribe:
        <http://lists.example.com/archives/listinfo/xtest@example.com>,
        <mailto:xtest-join@example.com>
    <BLANKLINE>
    First post!
    <BLANKLINE>
    <BLANKLINE>

    >>> digest.clear()
