imappop - Getting PigeonDeliver handle IMAP and POP3

The first choice is what usually happens with any other MTA software: you install the MTA, decide the directory structure for users' home directories, decide a common mailbox format and figure out, one way or another, how to make the choosen pop3/imap server access that user database, directory structure and mailbox format.

The main disadvantage of this approach is that you often need to figure out how to make a given MTA with a well defined structure work well with a pop3/imap server with a completely different structure in mind. The main problems that are usually encountered are due to:

• Since most servers use either proprietary configuration files or databases, to make two different softwares work together you will probably end up using a shared database. While this is not bad by itself and even be something you were wishing, there are at least two drawbacks...

• ... first of all, your choice of database to use is limited to the databases working both with your MTA and POP3/IMAP server, which may well leave you just a couple choices...

• ... additionally, you need to figure out how two different softwares, which may use a completely different internal structure, can use one same database for their purposes. Usually, when configuring such a solution, problems like: "hey, the mta wants the full path of the home directory in one field...", "the pop3/imap server builds the path using two different fields...", "the uid / gid of the MTA is set from the configuration file...", "the pop3 / imap server wants two fields to be added...", "one crypts the passwords with ...", "the other wants..." and so on. Figuring out how to make things work smoothly in such conditions is not often easy, and usually leads to a database more complex than necessary and with weird relations/fields to be kept consistent/... to be used. Which means: something harder to maintain, and something harder to modify...

• ... even if you get a nice looking database without using hacks or SQL queries like CONCAT(field1, '/', ...) or strange constants, some softwares may just make different assumptions. Who creates the users' home directory? is that directory writeable by whom? if the directory is created automagically, how do we handle privileges? ...

• How do we handle privileges of different softwares having to access the same user base? What if some day or another you want to switch to a different software? What if the two softwares have slightly different semantics for things that should be standard? How do we handle quotas? ...

• How do we manage a cluster? how do we create one? how do we solve concurrency issues on a database that one software does not even lock when modifying it, or where relations are so weird that we cannot guarantee atomicity when updating user configurations... (and believe me or not, there are very few open source solutions that worry about the atomicity of updating user configurations).

PigeonAir and PigeonDeliver internally provide support for using a couple of POP3 and IMAP servers.

At time of writing, support for the Courier IMAP and Courirer POP3 server (http://www.courier-mta.org) has already been completed, while Dovecot support is on its way to completion.

The basic idea is quite simple: PigeonDeliver and PigeonAir must take care of handling all integration issues between the MTA and the POP3/IMAP server being used (user database, authentication, directory structure, mailbox format) and must take care to abstract out clusters of mail servers like if they were a single machine both to the end user, to the administrator, and to the softwares being used.

From a more technical point of view, the idea is quite simple: PigeonDeliver internally implements a "modular" IMAP/POP3 proxy server. The proxy server itself takes care of authenticating users and fetching their configuration from a database, while its modular nature allows it to either transparently redirect the connections to other machines in a mail cluster or to internally call and load the mail server of your choice (courier or dovecot, at time of writing, Fri Feb 24 21:30:06 CET 2006), without any performance loss (and probably, with some gain). What's new about the PigeonDeliver IMAP/POP3 proxy modular structure is:

• authentication and fetching of users configurations are performed only once. Almost all IMAP/POP3 proxy we are aware of need to perform authentication twice: one on the machine the user connects to, and one on the machine where the user is redirected. And, on both machines, user configurations need to be fetched from the database.

  The PigeonDeliver POP3/IMAP proxy can use a configurable authentication mechanism when talking to its own peers. One of these mechanisms (which is just one which you can choose among others...) establishes some kind of trust path based on cryptographic authentication of hosts. Once the host is authenticated, the IMAP/POP3 proxy can pass authentication data (is the user who claims to be? who is he?) and configurations directly to the other peer, avoiding any additional lookup. No sensitive data (like passwords, certificates or private keys) is ever sent on the network, not even on encrypted channels.

• when proxying is avoidable, proxying is actually avoided. At time of writing, the PigeonDeliver IMAP/POP3 proxy can be used just as a simple IMAP/POP3 authentication module. For example, the "courier-imapd" module allows PigeonDeliver to internally load the courier imapd daemon. When courier is loaded internally, however, authentication has been carried on already by PigeonDeliver and all user configurations have already been fetched from the database. So, no need to enable strange courier authentication modules, no need at all to configure any courier authentication or database. Everything is taken care efficiently by the PigeonDeliver module, which also caches connections to the database as much as possible. In short: by using PigeonDeliver IMAP/POP3 proxy you should see a speedup in user authentications. Both on single setup and on clustered solutions.

• If you start with one single machine, PigeonDeliver IMAP/POP3 proxy will just act as a very simple authentication module for your IMAP/POP3 server of choice, without adding any overhead to the standard mechanism used by Courier/Dovecot.

  If you then switch to a mail cluster, PigeonDeliver IMAP/POP3 proxy will simply continue to handle "local connections" like before, by internally loading Courier or Dovecot, while redirecting connections that need to be handled by remote servers, thus becoming a very fast, smart and efficient POP3/IMAP proxy.

• This same modular structure also allows for many other mechanisms to be used. If you want a "normal and standard" IMAP/POP3 proxy, you can use the PigeonDeliver one just to avoid having to mess up with the database one more time. Right, because if you have a MTA, a POP3/IMAP server, an IMAP/POP3 proxy based on different softwares, it is quite easy to end up with a good headache when configuring one single database to be used by all of them.

• Another nice feature is that it can really be integrated in any mail server. As an example, if you use the module "pd-forker-inet" you can run the IMAP/POP3 server from your inetd.conf file. If you use "pd-forker-stdio", you can load it from the command line. If you use "pd-forker-postfix", you can run it from the "master.cf" postfix configuration file, ending up with a single place for your logs and with all daemons involved with "mail management" monitored by the postfix master, which is extremely reliable and performant, and which should simplify the task of administering the machine. If you use "pd-forker-listen", you will be able to run the IMAP/POP3 server as a standalone daemon... and you are free to add more modules to support more advanced connection handling/forking mechanisms or to integrate PigeonDeliver IMAP/POP3 server in any other software you may like.