Brian Roy’s Blog

The great IMAP migration…

Tuesday, August 26th, 2008 | Experimentation, Vendors | Comments

Late last week GoDaddy finally pushed me over the edge… I just couldn’t take it anymore. I’m all for combatting spam… I really am, but when 60% plus of the emails sent to me bounce as spam/virus something has gone dreadfully awry.

So, I had to search for a new email hosting provider - and while I was at it I could right one of the major (non-email bounce) issues I had with GoDaddy - no IMAP support.

I reviewed several options… and came to the conclusion that Google Apps (for MY domain) would be the right choice. I only had one serious issue with this. I wanted gmail to provide my IMAP store. The challenge? How to migrate 20 accounts with all of the associated IMAP folders and emails to gmail.

Turns out there is no simple answer unless you opt for a premier account with Google - which will run you $50/user/year. The premier account has an IMAP migration tool (but no backup… keep reading). But here is what I settled on:

imapsync would handle moving my current IMAP store to Gmail. To save you some time here is the full command I used:

/usr/local/imapsync/imapsync –syncinternaldates –host1 <my_imap_server> –user1 <username> –ssl1 –password1 <pwd> –host2 imap.gmail.com –port1 993 –port2 993 –ssl2 –user2 <gmail user> –password2 <gmail pwd> –prefix2 ‘[Gmail]/’

Now that I had all of the old email moved to Gmail I only had one other issue… What if Gmail “went away”.

Minor Tanget:

An associate of mine said “You don’t have to worry about it… Google will never go away. Backing up Google is silly.” - to which I respond:

Isn’t that what we thought about aol?

End of Tangent.

So I needed to find a solution that would allow me to keep all of my Gmail IMAP folders and emails copied in a reasonably secure way - ensuring that if I ever needed to move away from Gmail I could without losing anything.

Initially, I tried imapsync again to sync my Gmail data back down to my IMAP server. This, however, was not practical. imapsync’s primary attribute is not speed. It is not a high speed backup/sync solution… it is a bullet-proof solution. That made it:

1. Way too much overhead for my needs
2. Way too slow for my needs
3. it is written in Perl - personal thing here but ICK.

I also looked at offlineimap (which I had looked at for the initial migration as well… but it has no ability to “dry run”- which was too big a chance to take). The advantage of offlineimap is that it “syncs” your IMAP to a Maildir (which - conveniently is my dovecot IMAP server’s file structure). This makes it much faster than doing IMAP to IMAP synchronization. Here are some other advantages:

1. Runs as a daemon with defined intervals at which to sync accounts
2. Supports multiple accounts
3. Supports “quick sync” to only sync if a maildir has changed
4. Python… giant step up from perl

NOTE: Why wasn’t I concerned about no “dry run”? Because I was creating a backup of the GMail IMAP folders and emails in a simple directory on my server. If the run went bad… just delete the directory and try again. I was also certain I had told offlineimap NOT to delete or expunge anything from my GMail account.

Again, to save you some time, here is my offlineimap config file:

[general]

accounts = brian.cosinity, brian.brianandkelly, brian.briantroy

metadata = /etc/offlineimap/.offlineimap

maxsyncaccounts = 3

ui = Noninteractive.Basic

[Account brian.cosinity]

localrepository = Local.brian.cosinity

remoterepository = brian.cosinity.gmail

autorefresh = 120

quick = 10

[Repository Local.brian.cosinity]

type = Maildir

localfolders = /u02/gmail-imap-backups/brian.cosinity

[Repository brian.cosinity.gmail]

folderfilter = lambda foldername: foldername not in ['[Gmail]/All Mail’]

type = IMAP

remotehost = imap.gmail.com

remoteuser = <gmail_account>

remoteport = 993

ssl = yes

remotepass = <gmail_password>

[Account brian.briantroy]

localrepository = Local.brian.briantroy

remoterepository = brian.briantroy.gmail

autorefresh = 120quick = 10

[Repository Local.brian.briantroy]

type = Maildir

localfolders = /u02/gmail-imap-backups/brian.briantroy

[Repository brian.briantroy.gmail]

folderfilter = lambda foldername: foldername not in ['[Gmail]/All Mail’]

type = IMAP

remotehost = imap.gmail.com

remoteuser = <gmail_account>

remoteport = 993

ssl = yes

remotepass = <gmail_password>

[Account brian.brianandkelly]

localrepository = Local.brian.brianandkelly

remoterepository = brian.brianandkelly.gmail

autorefresh = 120

quick = 10

[Repository Local.brian.brianandkelly]

type = Maildir

localfolders = /u02/gmail-imap-backups/brian.brianandkelly

[Repository brian.brianandkelly.gmail]

folderfilter = lambda foldername: foldername not in ['[Gmail]/All Mail’, ‘[Gmail]/System Email’]

type = IMAP

remotehost = imap.gmail.com

remoteuser = <gmail_account>

remoteport = 993

ssl = yes

remotepass = <gmail_password>

What this does is backup 3 gmail accounts (brian.cosinity, brian.briantroy, and brian.brianandkelly) to /u02/gmail-imap-backup/<folder>. These accounts are backed up every 120 minutes. Also, every 10th sync is a full sync - so only once per day (every 20 hours) does offlineimap fully sync the accounts - the other syncs are “quick”.
You can check out the offlineimap wiki for more information about each directive in this config file.
This solution works for me. If you are looking for true dual synchronized IMAP servers you would have to go a bit further that what I’ve done here.
Now - with any luck - Google Gmail will never go down and I’ll never need any of this… but I’ll sleep better knowing it is there.

Tags: gmail, google