Metadata-Version: 1.1
Name: mongo-mail-server
Version: 0.1.1
Summary: Python SMTP server with Gevent for recording messages in MongoDB
Home-page: https://github.com/radical-software/mongo-mail-server
Author: Stéphane RAULT
Author-email: stephane.rault@radicalspam.org
License: BSD
Description: =================
        Mongo Mail Server
        =================
        
        **Gevent SMTP Server with MongoDB storage** 
        
        |Build Status| |pypi downloads| |pypi version| |pypi licence| |requires status|
        
        **Demo:**
        
        - URL: http://188.165.254.60:8083
        - Login: admin@example.net
        - Password: password
        
        **Features:**
        
        - SMTP Server high performance with Gevent Coroutine_ ( fork of Gsmtpd_ )
        - Postfix XFORWARD_ extension
        - Record messages in MongoDB_
        - Ability to use a custom python plugin to edit the message before recording
        
        **The use in production is not yet guaranteed**
        
        .. contents:: **Table of Contents**
            :depth: 1
            :backlinks: none
        
        Quarantine mode example
        =======================
        
        **Mode for quarantine or statistics only**
        
        .. image:: http://espace-groupware.com/docs/mongo-mail/img/mongo-mail-quarantine.jpg
           :alt: Mongo Mail Quarantine schema
           :align: center
        
        - Network: smtp sender or recipient (with internet or local network)
        - Content Filter: amavisd-new with xxx_quarantine_to (smtp:) parameters
        - Configuration in amavisd.conf
        - Zero configuration for Postfix
        
        Proxy mode example
        ==================
        
        **Mode for statistics or honey pot**
        
        .. image:: http://espace-groupware.com/docs/mongo-mail/img/mongo-mail-proxy.jpg
           :alt: Mongo Mail Proxy schema
           :align: center
           
        - Network: smtp sender or recipient (with internet or local network) 
        - Optional: filtering after delivery to postfix by mongo mail
        
        Filter mode example
        ===================
        
        **Mode for spam/virus filtering and statistics**
        
        .. image:: http://espace-groupware.com/docs/mongo-mail/img/mongo-mail-filter.jpg
           :alt: Mongo Mail Filter schema
           :align: center
           
        - Network: smtp sender or recipient (with internet or local network) 
        - Filtering: clamav/spamassassin with TCP connection (without amavisd-new)
        - Out filter: delivered so clean else
        
        Important Notes
        ===============
        
        - **Message is not transformed (unless in quarantine mode if MMS_REAL_RCPT=1)**
        
        - **Gross message is stored in GridFS after being compressed (zlib) and converted to base64**
        
        - **In proxy mode, the raw message is sent before the registration in MongoDB and is not saved if an error occurs while sending or all recipients are rejected** 
        
        Tested With
        -----------
        
        - Docker_ 1.4.1
        - Ubuntu_ 14.04
        - MongoDB_ 2.6.5
        - Python_ 2.7.6
        - Gevent_ 1.0
        - Pymongo2_8_ 2.8 and Pymongo_ 3.0 
        - Postfix_ 2.5.5
        - Amavisd-new_ 2.6.4
        
        Message Data
        ============
        
        Before sent to MongoDB
        ----------------------
        
        .. code:: python
        
            {'client_address': '139.129.236.68',
             'message': ObjectId('55252ae62d4b25262070a176'),
             'rcpt': ['jean91@example.com'],
             'rcpt_count': 1,
             'rcpt_refused': {},
             'received': datetime.datetime(2015, 4, 8, 13, 19, 34, 579000, tzinfo=tzutc()),
             'sender': 'acollet@example.org',
             'server': '127.0.0.1',
             'store_key': '77bd8b356cf2c593e61a6c0a7cbc5572eb357a7b857adca402ee40021db34fa6',
             'xforward': {'ADDR': '139.129.236.68',
                          'HELO': 'mx.example.org',
                          'NAME': 'mx.example.org'}}   
                          
            'message': ObjectId('55252ae62d4b25262070a176') is reference to data in Gridfs                                 
                              
        After record in MongoDB - Read from mongo-mail-web
        --------------------------------------------------
        
        .. code:: python
        
            {'_id': ObjectId('55252ae62d4b25262070a178'),
             'client_address': u'139.129.236.68',
             'completed': 0,
             'errors_count': 0,
             'events': [],
             'files': [],
             'files_count': 0,
             'group_name': u'DEFAULT',
             'headers': {},
             'internal_field': 0,
             'is_banned': 0,
             'is_bounce': 0,
             'is_in': 1,
             'is_spam': 0,
             'is_unchecked': 0,
             'is_virus': 0,
             'mark_for_delete': 0,
             'message': ObjectId('55252ae62d4b25262070a176'),
             'parsing_errors': [],
             'queue': 1,
             'rcpt': [u'jean91@example.com'],
             'rcpt_count': 1,
             'rcpt_refused': {},
             'received': datetime.datetime(2015, 4, 8, 13, 19, 34, 579000, tzinfo=<bson.tz_util.FixedOffset object at 0x02B54E10>),
             'sender': u'acollet@example.org',
             'server': u'127.0.0.1',
             'size': 0L,
             'store_key': u'77bd8b356cf2c593e61a6c0a7cbc5572eb357a7b857adca402ee40021db34fa6',
             'tags': [],
             'xforward': {u'ADDR': u'139.129.236.68',
              u'HELO': u'mx.example.org',
              u'NAME': u'mx.example.org'}}
        
        After parsing with mongo-mail-web (completed task)
        --------------------------------------------------
        
        .. code:: python
        
            {'_id': ObjectId('55252ae62d4b25262070a178'),
             'client_address': u'139.129.236.68',
             'completed': 1,
             'country': u'CN',
             'errors_count': 0,
             'events': [],
             'files': [],
             'files_count': 0,
             'group_name': u'DEFAULT',
             'headers': {u'Content-Transfer-Encoding': [u'base64', {}],
              u'Content-Type': [u'text/plain', {u'charset': u'utf-8'}],
              u'Date': u'Wed, 08 Apr 2015 13:19:34 UTC',
              u'From': u'"Bertrand Auger" <acollet@example.org>',
              u'Message-Id': u'<20150408131934.10264.63423@admin-VAIO>',
              u'Mime-Version': u'1.0',
              u'Subject': u'Provident tempora ad quasi enim in ratione excepturi. Optio soluta culpa voluptas labore in. Voluptatem aliquid est rerum in est adipisci dolore.',
              u'To': u'"Thierry Leleu" <jean91@example.com>',
              u'X-Mailer': u'MessageFaker'},
             'internal_field': 0,
             'is_banned': 0,
             'is_bounce': 0,
             'is_in': 1,
             'is_spam': 0,
             'is_unchecked': 0,
             'is_virus': 0,
             'mark_for_delete': 0,
             'message': ObjectId('55252ae62d4b25262070a176'),
             'message_id': u'20150408131934.10264.63423@admin-VAIO',
             'parsing_errors': [],
             'queue': 1,
             'rcpt': [u'jean91@example.com'],
             'rcpt_count': 1,
             'rcpt_refused': {},
             'received': datetime.datetime(2015, 4, 8, 13, 19, 34, 579000, tzinfo=<bson.tz_util.FixedOffset object at 0x02AC4E10>),
             'sender': u'acollet@example.org',
             'sent': datetime.datetime(2015, 4, 8, 13, 19, 34, tzinfo=<bson.tz_util.FixedOffset object at 0x02AC4E10>),
             'server': u'127.0.0.1',
             'size': 636L,
             'store_key': u'77bd8b356cf2c593e61a6c0a7cbc5572eb357a7b857adca402ee40021db34fa6',
             'subject': u'Provident tempora ad quasi enim in ratione excepturi. Optio soluta culpa voluptas labore in. Voluptatem aliquid est rerum in est adipisci dolore.',
             'tags': [],
             'xforward': {u'ADDR': u'139.129.236.68',
              u'HELO': u'mx.example.org',
              u'NAME': u'mx.example.org'}}    
        
        Original Message
        ----------------
        
        ::
        
            Content-Type: text/plain; charset="utf-8"
            MIME-Version: 1.0
            Content-Transfer-Encoding: base64
            X-Mailer: MessageFaker
            Message-ID: <20150408131934.10264.63423@admin-VAIO>
            From: "Bertrand Auger" <acollet@example.org>
            To: "Thierry Leleu" <jean91@example.com>
            Subject: Provident tempora ad quasi enim in ratione excepturi. Optio soluta
             culpa voluptas labore in. Voluptatem aliquid est rerum in est adipisci
             dolore.
            Date: Wed, 08 Apr 2015 13:19:34 UTC
            
            U2l0IHZvbHVwdGF0ZSByZXJ1bSBjb3Jwb3JpcyBkb2xvcmlidXMgZW9zLiBRdWFzIGVvcyBub24g
            bW9kaSBxdWlzLiBBbGlhcyB2ZWwgbGF1ZGFudGl1bSBtYWduaSBzdXNjaXBpdC4gRnVnaWF0IGV0
            IHF1aXMgZXQgaW4gYWNjdXNhbXVzLg==
        
        
        Installation
        ============
        
        Without Docker
        --------------
        
        Required
        ::::::::
        
        - MongoDB Server
        - Postfix or Amavisd-new
        - Python 2.7.6+ (< 3.x)
        - python-gevent 1.0+
        - recent setuptools and pip installer
        
        Installation
        ::::::::::::
        
        .. code:: bash
        
            $ pip install mongo-mail-server
        
            $ mongo-mail-server --help 
        
        
        With Docker
        -----------
        
        Required
        ::::::::
        
        - Docker 1.4+
        - MongoDB Server
            
        MongoDB Server example
        ::::::::::::::::::::::
        
        Contenair based on Ubuntu 14.04 - Python 2.7
        
        Image from Dockerfile_
        
        .. code:: bash
        
            $ docker pull dockerfile/mongodb
            
            $ docker run -d -p 27017:27017 --name mongodb dockerfile/mongodb mongod --smallfiles
            
            # Persist mongodb
            $ docker run -v /home/persist/mongodb:/data/db -d -p 27017:27017 --name mongodb dockerfile/mongodb mongod --smallfiles
        
        Build Mongo Mail Server image
        :::::::::::::::::::::::::::::
        
        .. code:: bash
        
            $ git clone https://github.com/radical-software/mongo-mail-server.git
            
            $ cd mongo-mail-server && docker build -t mongo-mail-server .
            
            # help and verify
            $ docker run -it --rm mongo-mail-server --help
        
        Run Mongo Mail Server
        :::::::::::::::::::::
        
        .. code:: bash
        
            $ mongodb_ip=$(docker inspect -f '{{.NetworkSettings.IPAddress}}' mongodb)
        
            # start for test
            $ docker run -it --rm -e MMS_MONGODB_URI=mongodb://$mongodb_ip/message -p 172.17.42.1:14001:14001 mongo-mail-server
        
            # start of background (optional: bind of docker0 interface)
            # Add --restart=always for automatic restart 
            $ docker run -d --name mms -e MMS_MONGODB_URI=mongodb://$mongodb_ip/message -p 172.17.42.1:14001:14001 mongo-mail-server
        
            # Logs
            $ docker logs mms
            2015-02-12 07:35:36 rs_smtpd_server: [INFO] - Starting SMTP Server - server[mongo-quarantine] - on 0.0.0.0:14001 (PID:1)
        
        Configuration
        =============
        
        MMS_SERVER
        ----------
        
        Server mode: mongo-quarantine | mongo-proxy | mongo-proxy | debug
        
        *Default*: mongo-quarantine
        
        .. code:: bash
        
            # with command mode
            $ export MMS_SERVER=mongo-quarantine
            
            # with docker environ
            $ docker run -e MMS_SERVER=mongo-quarantine
            
            # with command arguments
            $ mongo-mail-server --server mongo-quarantine 
        
        MMS_HOST
        --------
        
        **Host bind**
        
        *Default*: 0.0.0.0
        
        .. code:: bash
        
            # with command mode
            $ export MMS_HOST=0.0.0.0
            
            # with docker environ
            $ docker run -e MMS_HOST=0.0.0.0
            
            # with command arguments
            $ mongo-mail-server --host 0.0.0.0 
        
        MMS_PORT
        --------
        
        **Port bind**
        
        *Default*: 14001
        
        .. code:: bash
        
            # with command mode
            $ export MMS_PORT=14001
            
            # with docker environ
            $ docker run -e MMS_PORT=14001
            
            # with command arguments
            $ mongo-mail-server --port 14001
            
        MMS_MONGODB_URI
        ---------------
        
        *Default*: mongodb://localhost/message
        
        http://docs.mongodb.org/manual/reference/connection-string/
        
        .. code:: bash
        
            # with command mode
            $ export MMS_MONGODB_URI=mongodb://localhost/message
            
            # with docker environ
            $ docker run -e MMS_MONGODB_URI=mongodb://localhost/message
            
            # with command arguments
            $ mongo-mail-server --mongo-host mongodb://localhost/message
        
        
        MMS_MONGODB_DATABASE
        --------------------
        
        **DB Name for recording mails**
        
        *Default*: message
        
        .. code:: bash
        
            # with command mode
            $ export MMS_MONGODB_DATABASE=message
            
            # with docker environ
            $ docker run -e MMS_MONGODB_DATABASE=message
            
            # with command arguments
            $ mongo-mail-server --mongo-database message
        
        
        
        MMS_MONGODB_COLLECTION
        ----------------------
        
        **Collection Name for recording mails**
        
        *Default*: message
        
        .. code:: bash
        
            # with command mode
            $ export MMS_MONGODB_COLLECTION=message
            
            # with docker environ
            $ docker run -e MMS_MONGODB_COLLECTION=message
            
            # with command arguments
            $ mongo-mail-server --mongo-collection message
             
        MMS_TIMEOUT
        -----------
        
        **Timeout for smtp transaction from Postfix**
        
        *Default: 600 (seconds)*
        
        MMS_DATA_SIZE_LIMIT
        -------------------
        
        **Size limit of message (in bytes)**
        
        *Default: 0 (no limit)*
        
        MMS_REAL_RCPT (for amavisd-new < 2.7.0)
        ---------------------------------------
        
        **Replace smtp recipient by real recipients (for quarantine with amavisd-new)** 
        
        *Default*: disable
        
        .. code:: bash
        
            # with command mode
            $ export MMS_REAL_RCPT=1
            
            # with docker environ
            $ docker run -e MMS_REAL_RCPT=1
            
            # with command arguments
            $ mongo-mail-server --real-rcpt
            
        Usecase - Quarantine Mode configuration - with Amavis
        =====================================================
        
        **caution**
        
        ::
        
            Before amavisd-new 2.7.0 the recipient envelope is replaced by xxx_quarantine_to parameters
            
            Starting from 2.7.0, use macro '%a' in xxx_quarantine_to parameters
        
        **caution**
        
        ::
            
            About IP Address of smtp sender:
            
            Amavis does not use the extension SMTPD FORWARD to send mails in quarantine. The original IP address is lost.
            
            The solution might be to use postfix to amavis output for quarantine and postfix then return the message to mongo-mail       
            
        
        For Archiving only
        ------------------
        
        .. code:: bash
        
            $ vi amavisd.conf
            
            # ip address and port of Mongo Mail Server
            $archive_quarantine_method      = 'smtp:[172.17.42.1]:14001';
            
            # Any valid email address. Domain few not exist
            $archive_quarantine_to          = 'archive-quarantine@localhost.net';
            
            # reload amavis
        
        For Quarantine and Archiving
        ----------------------------
        
        .. code:: bash
        
            $ vi amavisd.conf
        
            $archive_quarantine_method      = 'smtp:[172.17.42.1]:14001';
            $archive_quarantine_to          = 'archive-quarantine@localhost.net';
        
            $virus_quarantine_method        = $archive_quarantine_method;
            $banned_files_quarantine_method = $archive_quarantine_method;
            $spam_quarantine_method         = $archive_quarantine_method;
            
            # Not quarantine for clean mail - already stored with archive_quarantine_method
            $clean_quarantine_method        = undef;
            
            # Not quarantine for bad header mail
            $bad_header_quarantine_method   = undef;
        
            $virus_quarantine_to            = $archive_quarantine_to;
            $banned_quarantine_to           = $archive_quarantine_to;
            $spam_quarantine_to             = $archive_quarantine_to;
            
            #OR
            $virus_quarantine_to            = 'virus-quarantine@localhost.net';
            $banned_quarantine_to           = 'banned-quarantine@localhost.net';
            $spam_quarantine_to             = 'spam-quarantine@localhost.net';
            
        Usecase - Proxy Mode - Honey pot
        ================================
        
        **Dedicate a postfix server for this purpose**
        
        .. code:: bash
        
            # main.cf - ip:port of Mongo Mail
            smtpd_proxy_filter=127.0.0.1:14001
            
            # or with command line
            $ postconf -e 'smtpd_proxy_filter=127.0.0.1:14001'
            
            # reload postfix
            $ postix reload
            
        
        Using a plugin
        ==============
        
        **The module must be in a package**
        
        .. code:: python
        
            # just required apply(metadata=None, data=None) method
        
            # examples/plugins/dummy_plugin.py - modify server field and print message
            
            import pprint
            def apply(metadata=None, data=None):
                metadata['server'] = "1.1.1.1"
                pprint.pprint(metadata)    
            
            # Use:
            $ mongo-mail-server --server debug --host 127.0.0.1 --port 14001 --plugin contrib.dummy_plugin start
        
            # Use multiple plugins - run in the order of arguments
            $ mongo-mail-server --server --plugin myplugin1 --plugin myplugin2 ...
            
        SMTP Tests - With Telnet
        ========================
        
        .. code:: bash
        
            # Use 172.17.42.1 is binding of docker0 else:
            $ mms_ip=$(docker inspect -f '{{.NetworkSettings.IPAddress}}' mms)
        
            $ telnet $mms_ip 14001
            
            Trying 172.17.1.19...
            Connected to 172.17.1.19.
            Escape character is '^]'.
            220 a88632d9a311 SMTPD at your service
            
            ehlo me.com
            250-a88632d9a311 on plain
            250-XFORWARD NAME ADDR PROTO HELO SOURCE PORT
            250 HELP
            
            XFORWARD NAME=mail.test.fr ADDR=1.1.1.1 HELO=test.fr
            250 Ok
            
            MAIL FROM:<contact@test.fr>
            250 Ok
            
            RCPT TO:<contact@localhost.net>
            250 Ok
            
            DATA
            354 End data with <CR><LF>.<CR><LF>
            Subject: Test
            From: contact@test.fr
            To: contact@localhost.net
            
            mytest
            .
            250 Ok: queued as ab80249748e0496b812b13c489a88002fbe102fc9c263b02a8b52101491f0128
            
            QUIT
            221 Bye
            Connection closed by foreign host.
            
        Use mongofiles command
        ======================
        
        .. code:: bash
        
            $ mongofiles -d message list
            72c0f4898db56d5e10037e3f7f0c2af68704c8b86a2405d98a3e44e89bb56481        2188
            571329a72c31a914251fd6fdecb160403345ee143c194cfc442ab5bee6118918        2188
            a8de0206f9978346326cbcc9ffd5df647728268c19e8564dd1c2790b6c1404f3        2192
            ...    
            
            # Extract and write message to disk
            $ mongofiles -d message get 75e3896c1c5d98a21fc14e9408e1b9be91ced60f2bc224416de63c975c9c2915
            
            # Convert with python
            python -c "import zlib,base64; print(str(zlib.decompress(base64.b64decode(open('75e3896c1c5d98a21fc14e9408e1b9be91ced60f2bc224416de63c975c9c2915', 'rb').read()))))"
        
            # Parse to email.Message and print as_string()
            python -c "import zlib,base64,email; print(email.message_from_string(str(zlib.decompress(base64.b64decode(open('75e3896c1c5d98a21fc14e9408e1b9be91ced60f2bc224416de63c975c9c2915', 'rb').read())))).as_string())"
                
            
        Tips
        ====
        
        SMTP timeout
        ------------
        
        Use MMS_TIMEOUT  in environment or --timeout
        
        Size of messages
        ----------------
        
        Use MMS_DATA_SIZE_LIMIT in environment or --data-size-limit
        
        Open Message with Python
        ------------------------
        
        .. code:: python
        
            >>> import os, zlib, base64
            >>> from pprint import pprint as pp
            >>> from email.parser import Parser, HeaderParser
            >>> from pymongo import MongoClient
            >>> from gridfs import GridFS
            >>> client = MongoClient(os.environ.get('MMS_MONGODB_URI'))
            >>> db = client['message']
            >>> col = db['message']
            >>> doc = col.find_one() 
            >>> fs = GridFS(db)
            >>> msg_base64 = fs.get(doc['message']).read()
            >>> msg_string = zlib.decompress(base64.b64decode(msg_base64))
            >>> msg = Parser().parsestr(msg_string)
            >>> msg
            <email.message.Message instance at 0x7ff5e4054560>    
        
        
        TODO
        ====
        
        - More tests
        - Travis tests
        - Monitoring with psutil
        - Filter tasks
        - Documentation of mongo-mail-reader command
        - Documentation en Français
        
        Ideas
        =====
        
        - Record to ElasticSearch
        - Sends statistics to graphite, statsd, influxdb
        
        Contributing
        ============
        
        To contribute to the project, fork it on GitHub and send a pull request, all contributions and suggestions are welcome.
        
        .. _Gsmtpd: https://github.com/34nm/gsmtpd
        .. _MongoDB: http://mongodb.org/
        .. _Docker: https://www.docker.com/
        .. _Ubuntu: http://www.ubuntu.com/
        .. _Dockerfile: http://dockerfile.github.io/#/mongodb
        .. _Python: http://www.python.org/
        .. _Pymongo2_8: http://api.mongodb.org/python/2.8/
        .. _Pymongo: http://api.mongodb.org/python/current/index.html
        .. _Gevent: http://www.gevent.org/
        .. _Postfix: http://www.postfix.org
        .. _XFORWARD: http://www.postfix.org/XFORWARD_README.html
        .. _Amavisd-new: http://www.ijs.si/software/amavisd/
        .. _Clamav: http://clamav.net/
        .. _SpamAssassin: http://spamassassin.org/
        .. _Coroutine: http://en.wikipedia.org/wiki/Coroutine
         
        .. |Build Status| image:: https://travis-ci.org/radical-software/mongo-mail-server.svg?branch=master
           :target: https://travis-ci.org/radical-software/mongo-mail-server
           :alt: Travis Build Status
        
        .. |pypi downloads| image:: https://img.shields.io/pypi/dm/mongo-mail-server.svg
            :target: https://pypi.python.org/pypi/mongo-mail-server
            :alt: Number of PyPI downloads
            
        .. |pypi version| image:: https://img.shields.io/pypi/v/mongo-mail-server.svg
            :target: https://pypi.python.org/pypi/mongo-mail-server
            :alt: Latest Version
        
        .. |pypi licence| image:: https://img.shields.io/pypi/l/mongo-mail-server.svg
            :target: https://pypi.python.org/pypi/mongo-mail-server
            :alt: License
        
        .. |pypi py_versions| image:: https://pypip.in/py_versions/mongo-mail-server/badge.svg
            :target: https://pypi.python.org/pypi/mongo-mail-server
            :alt: Supported Python versions
        
        .. |pypi dev_status| image:: https://pypip.in/status/mongo-mail-server/badge.svg
            :target: https://pypi.python.org/pypi/mongo-mail-server
            :alt: Development Status        
            
        .. |requires status| image:: https://requires.io/github/radical-software/mongo-mail-server/requirements.svg?branch=master
             :target: https://requires.io/github/radical-software/mongo-mail-server/requirements/?branch=master
             :alt: Requirements Status    
Platform: UNKNOWN
Classifier: Topic :: Communications :: Email
Classifier: Topic :: Communications :: Email :: Mail Transport Agents
Classifier: Development Status :: 4 - Beta
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 2
Classifier: Programming Language :: Python :: 2.7
Classifier: License :: OSI Approved :: BSD License
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: System Administrators
