The great IMAP migration…

Late last week GoDaddy finally pushed me over the edge… I just couldn’t take it anymore. I’m all for combatting spam… I really am, but when 60% plus of the emails sent to me bounce as spam/virus something has gone dreadfully awry.

So, I had to search for a new email hosting provider – and while I was at it I could right one of the major (non-email bounce) issues I had with GoDaddy – no IMAP support.

I reviewed several options… and came to the conclusion that Google Apps (for MY domain) would be the right choice. I only had one serious issue with this. I wanted gmail to provide my IMAP store. The challenge? How to migrate 20 accounts with all of the associated IMAP folders and emails to gmail.

Turns out there is no simple answer unless you opt for a premier account with Google – which will run you $50/user/year. The premier account has an IMAP migration tool (but no backup… keep reading). But here is what I settled on:

imapsync would handle moving my current IMAP store to Gmail. To save you some time here is the full command I used:

/usr/local/imapsync/imapsync –syncinternaldates –host1 <my_imap_server> –user1 <username> –ssl1 –password1 <pwd> –host2 imap.gmail.com –port1 993 –port2 993 –ssl2 –user2 <gmail user> –password2 <gmail pwd> –prefix2 ‘[Gmail]/’

Now that I had all of the old email moved to Gmail I only had one other issue… What if Gmail “went away”.

Minor Tanget:

An associate of mine said “You don’t have to worry about it… Google will never go away. Backing up Google is silly.” – to which I respond:

Isn’t that what we thought about aol?

End of Tangent.

So I needed to find a solution that would allow me to keep all of my Gmail IMAP folders and emails copied in a reasonably secure way – ensuring that if I ever needed to move away from Gmail I could without losing anything.

Initially, I tried imapsync again to sync my Gmail data back down to my IMAP server. This, however, was not practical. imapsync’s primary attribute is not speed. It is not a high speed backup/sync solution… it is a bullet-proof solution. That made it:

  1. Way too much overhead for my needs
  2. Way too slow for my needs
  3. it is written in Perl – personal thing here but ICK.

I also looked at offlineimap (which I had looked at for the initial migration as well… but it has no ability to “dry run”- which was too big a chance to take). The advantage of offlineimap is that it “syncs” your IMAP to a Maildir (which – conveniently is my dovecot IMAP server’s file structure). This makes it much faster than doing IMAP to IMAP synchronization. Here are some other advantages:

  1. Runs as a daemon with defined intervals at which to sync accounts
  2. Supports multiple accounts
  3. Supports “quick sync” to only sync if a maildir has changed
  4. Python… giant step up from perl

NOTE: Why wasn’t I concerned about no “dry run”? Because I was creating a backup of the GMail IMAP folders and emails in a simple directory on my server. If the run went bad… just delete the directory and try again. I was also certain I had told offlineimap NOT to delete or expunge anything from my GMail account.

Again, to save you some time, here is my offlineimap config file:

[general]

accounts = brian.cosinity, brian.brianandkelly, brian.briantroy

metadata = /etc/offlineimap/.offlineimap

maxsyncaccounts = 3

ui = Noninteractive.Basic

[Account brian.cosinity]

localrepository = Local.brian.cosinity

remoterepository = brian.cosinity.gmail

autorefresh = 120

quick = 10

[Repository Local.brian.cosinity]

type = Maildir

localfolders = /u02/gmail-imap-backups/brian.cosinity

[Repository brian.cosinity.gmail]

folderfilter = lambda foldername: foldername not in [‘[Gmail]/All Mail’]

type = IMAP

remotehost = imap.gmail.com

remoteuser = <gmail_account>

remoteport = 993

ssl = yes

remotepass = <gmail_password>

[Account brian.briantroy]

localrepository = Local.brian.briantroy

remoterepository = brian.briantroy.gmail

autorefresh = 120quick = 10

[Repository Local.brian.briantroy]

type = Maildir

localfolders = /u02/gmail-imap-backups/brian.briantroy


[Repository brian.briantroy.gmail]

folderfilter = lambda foldername: foldername not in [‘[Gmail]/All Mail’]

type = IMAP

remotehost = imap.gmail.com

remoteuser = <gmail_account>

remoteport = 993

ssl = yes

remotepass = <gmail_password>

[Account brian.brianandkelly]

localrepository = Local.brian.brianandkelly

remoterepository = brian.brianandkelly.gmail

autorefresh = 120

quick = 10

[Repository Local.brian.brianandkelly]

type = Maildir

localfolders = /u02/gmail-imap-backups/brian.brianandkelly

[Repository brian.brianandkelly.gmail]

folderfilter = lambda foldername: foldername not in [‘[Gmail]/All Mail’, ‘[Gmail]/System Email’]

type = IMAP

remotehost = imap.gmail.com

remoteuser = <gmail_account>

remoteport = 993

ssl = yes

remotepass = <gmail_password>

What this does is backup 3 gmail accounts (brian.cosinity, brian.briantroy, and brian.brianandkelly) to /u02/gmail-imap-backup/<folder>. These accounts are backed up every 120 minutes. Also, every 10th sync is a full sync – so only once per day (every 20 hours) does offlineimap fully sync the accounts – the other syncs are “quick”.
You can check out the offlineimap wiki for more information about each directive in this config file.
This solution works for me. If you are looking for true dual synchronized IMAP servers you would have to go a bit further that what I’ve done here.
Now – with any luck – Google Gmail will never go down and I’ll never need any of this… but I’ll sleep better knowing it is there.

  • Daniel Kraeer

    I think godaddy has imap. They say they don’t, but i was looking around my domain settings and noticed an imap adress at imap.domain.com

    set it up and it worked.

  • briantroy

    Daniel –

    The thing is – even if it “works” right now there is no guarantee it will tomorrow. An unsupported solution is pretty much the same thing as no solution in my book.

    Brian

  • jlane

    I am working on moving a non-profit over to Gmail. One concern is that the existing IMAP folders will be lost? I was thinking about signing up for the 30 day free trial of premium and syncing via IMAP? Once I get all the mail over… and in the right 'folder' my work is done. Any tips on moving? Do you think the 30 day trial idea is worth a shot?

  • jlane

    I am working on moving a non-profit over to Gmail. One concern is that the existing IMAP folders will be lost? I was thinking about signing up for the 30 day free trial of premium and syncing via IMAP? Once I get all the mail over… and in the right 'folder' my work is done. Any tips on moving? Do you think the 30 day trial idea is worth a shot?

  • MT

    hi,

    were you ever able to imapsync from godaddy to gmail? if so what imaphostname you used for godaddy?

  • http://www.briantroy.com/blog briantroy

    GoDaddy doesn't support IMAP. There are, however, plenty of tools to migrate from POP to IMAP, or even local mailbox files (from your client) to IMAP.

    Good luck…

  • Schwieb

    AWESOME! This post led us to be able to do a mass migration from our legacy email system to the Google Education accounts! FYI, Google's migration tool is far less compatible with different IMAP servers than imapsync.

  • http://www.briantroy.com/blog briantroy

    Glad to be of assistance! Good luck.

  • Ramon

    Out of curiosity, did you go back and ? I've been trying to use imapsync to move 10 years worth of emails from an IMAP server to GMail, and it seems to do a great job 99% of the time. However, the log almost always shows a small handful of messages that were “skipped”, and tracking those down is a bear. Did you see complete migration (i.e. no messages skipped) when you did your move?

    Ramon

  • http://commutertriathlon.com/ Jonathan

    Super helpful post, thanks! One thing to note, however, this doesn't seem to work (stalled on my imap server, didn't recognize a number of the params you used) with imapsync 1.99. After I'd downloaded it and tried it, I discovered I magically had a copy of imapsync 1.267 ALSO downloaded in my home dir, which worked like a charm. Very odd.

  • http://www.briantroy.com/blog briantroy

    Yes, this post is getting pretty old, but I'm glad you were able to figure out how to get it to work with newer versions of the tools.

  • http://commutertriathlon.com/ Jonathan

    Haha, no! The trick was that it *didn't* work with the newer version of the tool! Not sure what version you used, but the latest version seemed to remove the –sslX flag as well as the “works” feature :)

  • http://commutertriathlon.com/ Jonathan

    Super helpful post, thanks! One thing to note, however, this doesn't seem to work (stalled on my imap server, didn't recognize a number of the params you used) with imapsync 1.99. After I'd downloaded it and tried it, I discovered I magically had a copy of imapsync 1.267 ALSO downloaded in my home dir, which worked like a charm. Very odd.

  • http://www.briantroy.com/blog briantroy

    Yes, this post is getting pretty old, but I'm glad you were able to figure out how to get it to work with newer versions of the tools.

  • http://commutertriathlon.com/ Jonathan

    Haha, no! The trick was that it *didn't* work with the newer version of the tool! Not sure what version you used, but the latest version seemed to remove the –sslX flag as well as the “works” feature :)

  • http://www.kelvinism.com Kelvin Nicholson

    Cheers for posting this. I'm currently migrating many years worth of email from my personal server to Gmail, and wanted to know there was a way out “just in case”.

  • Pingback: ImapSync vs. MigrationWiz « MigrationWiz Blog()