Module 8 · 8.2

English

Nederlands English Deutsch Français Español Italiano

IMAP synchronisation and the MX switch

Module 8: Moving and getting your domain in order · Reading time: 12 minutes

Learning objective: After this lesson, you can copy mailboxes with an IMAP sync, carry out the MX switch, and use DNS lookups to confirm that new mail is arriving in the right place.

Key concepts: IMAP sync (imapsync), source and destination, linking folders (Sent/Verzonden), first and second round, avoiding duplicates, Message-ID, quota and large attachments, MX change, TTL and caches, transition period, reaching the old server temporarily by server name, switching over devices, mail.<domain> already points to the new server

How an IMAP sync works

An IMAP sync is a program that logs in to two mailboxes at the same time: the source (the old mailbox) and the destination (the new one). It reads the list of folders in the source, creates those folders in the destination, and copies, folder by folder, every message that's still missing from the destination. The best known program is imapsync, an open-source program that many providers use behind their migration tool.

Good to know:

It copies, it doesn't move. Nothing changes in the source. That's why you can copy calmly while the old mailbox just keeps working.
A second round skips what's already there. The program recognises messages that are already in the destination, usually by the Message-ID header plus a few other headers. That means you can sync as often as you like without ending up with duplicates.
Flags come along too: read, replied, flagged. That means the new mailbox looks the same.
The destination needs to be big enough. If the sync stops halfway, the new mailbox is often full. Increase the quota and start again: it carries on where it left off.

Linking folders: Sent isn't always Sent

Every provider names the special folders differently. With one it's called Sent, with another Sent Items, Verzonden items or INBOX.Sent. Gmail doesn't have real folders but labels, and shows them via IMAP as folders such as [Gmail]/Verzonden berichten and [Gmail]/Alle berichten.

Without linking, you end up with a Sent Items folder in the new mailbox alongside your own Sent folder, and your mail program only uses one of the two (lesson 3.1). Good migration tools link the special folders automatically; imapsync has the --automap option for this, and with --regextrans2 you can rename folders yourself. After the first round, check that your sent mail is in the right folder.

Two things to watch for with Gmail:

All Mail ([Gmail]/All Mail) contains all mail, including everything that's in other labels. If you copy that folder too, you'll end up with everything twice. Usually skip it, unless you have mail that's only archived.
A message with three labels appears via IMAP in three folders, and so gets copied three times. That's by design, but it does count towards your storage.

Folders with calendar items or contacts (with Microsoft 365 and Exchange you sometimes see these via IMAP too) can't be copied as mail. A good tool skips them and reports this.

The first round

Start the first round well before the MX switch, once the new mailboxes are ready. For each mailbox, you'll need:

the IMAP server of the source (for example imap.mailprovider.example, or the server name of your old hosting package);
the address and password (or app password, or a Microsoft login, lesson 8.1) of the source;
the address and password of the destination, and the server name of the new provider.

A pitfall: is your old IMAP server set up as mail.example.com, and have you already moved the DNS? Then mail.example.com may already point to the new server, and the program would copy from the new mailbox to the new mailbox. So for the source, use the fixed server name of your old provider (you'll find it in your old welcome email or client panel), not a name under your own domain.

Large mailboxes with lots of attachments take a long time: expect hours for many gigabytes. Just let the program run; most tools carry on where they left off after an interruption.

The MX switch

Have the mailboxes been copied, and are the records for SPF, DKIM and DMARC ready (lesson 8.1, step 3)? Then change the MX records to the values given by the new provider. After that:

Check at the authoritative nameserver whether the new MX is there. You'll see this immediately, without any cache:

$ dig +short NS example.com
ns1.dns.example.
ns2.dns.example.
$ dig +norecurse MX example.com @ns1.dns.example.
;; ANSWER SECTION:
example.com.     300   IN   MX   10 mx1.newprovider.example.
example.com.     300   IN   MX   20 mx2.newprovider.example.

Check at a public resolver what the rest of the internet sees. If you still get the old value there, that resolver still has it in its cache until the TTL runs out:

$ dig MX example.com @9.9.9.9
example.com.     212   IN   MX   10 mx.oldprovider.example.

The number (212) is the remaining time in seconds. After that time, the resolver asks again.

Send a test email from an address outside your domain (for example a free mail service, or from your phone over mobile internet) and check whether it arrives in the new mailbox. In the headers (lesson 1.5), the topmost Received lines show you which server it came in through.

The transition period

During the transition, mail sometimes still arrives at the old server: from senders whose resolver still knows the old MX, and from servers that had a message queued for the old server. That's not a problem, as long as the old mailbox still exists. Two ways to make sure you don't miss anything:

A second sync round a day after the switch (or later, depending on the old TTL). That only picks up the new messages. With a domain move that also involves a nameserver change, it's wise to do another round after two or three days.
Or keep both mailboxes temporarily in your mail program, with the old one set up using the fixed server name of the old provider. That way you'll spot any stragglers straight away.

Mail you send during this period goes out via the outgoing server set up on your device. Make sure SPF includes both providers during this period, otherwise your mail might score worse at the recipient by going out via the "wrong" server.

Switching over devices

On each device, set up the new mailbox as a new account with IMAP and the details of the new provider (module 2). Then remove the old account. Why not just change the existing account? Because the program can then sometimes try to sync the old local copy with the new mailbox, which can lead to duplicates or missing folders.

Don't forget:

Remove old accounts quickly. A phone in a drawer that keeps trying to log in with an old password can trigger a temporary IP block at the old or the new provider (lesson 3.4). Then suddenly nothing works on that network.
Everything that sends mail via SMTP: website, webshop, scanner, till system (lesson 8.1).
Signatures, rules and autoresponders need setting up again.

Only cancel afterwards

Only cancel the old mailboxes once everything has been working well for a few weeks and the last sync round is done. After that, clean up the DNS: remove the old provider from SPF, remove the old DKIM selector, and set the TTL back to a normal value.

In practice: what this looks like with a provider

With WebOké, you can copy using the email move tool in My WebOké (it picks the server name of well-known providers itself, also logs in to Microsoft, and carries on where it left off after an interruption) or with IMAPSync Migrations in DirectAdmin (fill in the source host, username and password, choose the destination mailbox). In both cases, a second round skips what's already been copied. Other providers have similar migration tools; the concepts of source, destination and second round are the same everywhere.

Summary

An IMAP sync copies, folder by folder, whatever is missing from the destination; a second round doesn't create duplicates.
Link special folders (Sent, Trash) and with Gmail watch out for labels and All Mail.
For the source, use the fixed server name of the old provider, not mail.<your domain>.
After the MX switch: first check authoritatively, then publicly (TTL), then a test email from outside.
Second round after the old TTL, set up devices as new accounts, remove old accounts, only cancel after a few weeks.

Quiz

Exercise

Exercise Z8.2: Checking MX and TTL around the switch. Enter your own domain (or use wikipedia.org if you don't have your own domain). Fill in: the MX host with the lowest priority according to the authoritative nameserver, the TTL of the MX record at that nameserver, and the party your mail arrives with (free text). The program looks up NS, then MX at an authoritative nameserver without a cache, and MX at a public resolver. It puts the answers side by side and flags it if the public resolver still gives a different answer (meaning the switch isn't visible everywhere yet).

Cookies at WebOké

Cookie settings

Necessary

Marketing