What I Learned Cutting Over Our First Domain (Including Three Bugs Apple Mail Found)

On September 1st I moved my own company domain from Amazon WorkMail onto inbox.camp. Not a test domain. The one I actually use. This is what happened, including the parts that didn't work, because those are the useful parts.

The rule: no cutover without a round trip

The engine will not touch a customer's MX record until it has proven the new server works end to end. Before the switch it had delivered a probe over SMTP to the new server, read it back over IMAP, and sent a reply out through the customer's SES relay. Then it copied every mailbox from WorkMail, and, this is the part I'm proudest of, counted what actually landed by logging in as the user and asking the server, rather than trusting the migration tool's own tally.

Only then did it publish the MX and the autodiscover records, send one more probe through the public internet to the new MX, and wait. The probe arrived in five seconds. My old MX value was written into the change comment so it could be put back with one command.

The data loss that wasn't

A week earlier I'd hit something that looked like a disaster: after a migration test, a mailbox that should have held six messages held two. The messages' bodies were still in S3; the index entries were gone. Two restarts of the mail server per management operation looked like the culprit, and I spent a day removing every restart from the engine's normal path. Good change regardless.

But restarts weren't the cause. The customer's own AWS account keeps a record of every command the engine runs on the server, with the full script, for thirty days. Reading that history showed exactly what happened: an earlier test session had deliberately wiped the mailbox, and the migration tool's duplicate detection then wrongly reported the wiped messages as "already present," so they were never copied again. The bug was a regular expression that matched the words message-id: inside DKIM signature headers, giving unrelated messages the same key.

Two lessons. First, when a system claims data loss, get three independent views of the data before theorizing; I ended up counting by EXISTS, by UID SEARCH, and by UID FETCH, and they agreed. Second, that command history is not just a compliance feature. It's the audit log that solved the case, and it lives in the customer's account, not mine.

I then tested the restart theory directly, on purpose: append messages, wait, restart, append more, delete some, restart again, count. Nothing was lost at any step. The engine now runs that kind of check as a matter of course.

Three bugs only a phone could find

Everything the engine tested passed. Then I added the account to iOS Mail.

1. "Username or password is wrong." It wasn't. The server log showed the phone logging in over IMAP successfully, twice. What failed was iOS checking outgoing mail: it tries port 587 first, and the server didn't listen on 587, only on 465. iOS fell through to port 25, which correctly refuses to authenticate, and reported that as a bad password. Fix: a standard 587 STARTTLS listener, now created for every tenant.
2. Sending bounced. The relay rejected the message: Duplicate header 'DKIM-Signature'. The server signs outgoing mail with two keys by default, RSA and Ed25519, and Amazon SES treats the second signature as a duplicate header. Fix: RSA only. SES adds its own signature on top, so recipients still see two valid, aligned signatures.
3. The certificate. The server orders one Let's Encrypt certificate covering five hostnames, including autodiscover and mta-sts. Before the cutover those names still pointed at WorkMail or didn't exist, so every order failed and Let's Encrypt rate-limited the retries. The cutover itself publishes exactly those names, so the fix was to nudge the renewal right after switching MX. The engine now does that.

None of these would have shown up in a test harness. All three are now defaults, which is what a pilot is for.

Where it stands

Mail for my domain arrives on a server in my own AWS account, is backed up hourly there, and sends through my own SES. I read it on my phone and in the new webmail. If inbox.camp disappeared tonight, the server would keep running. That was the whole idea, and it turns out to be true.