216 passing tests. A feature that was completely broken. Here is the gap between those two facts, and what I changed afterwards.
I am building an encrypted messenger. Messages are end to end encrypted, and the server relaying them cannot read anything. That part worked.
What I added was offline delivery. If you message someone whose app is closed, the server should hold the message, hand it over when they come back, then delete it. Nothing kept longer than it needs to be.
I wrote it. I wrote tests for it: unit tests for the storage layer, integration tests against a real Postgres, end to end tests over real WebSocket connections. Every one passed.
Then I ran it against the deployed build, closed one browser, sent two messages, and reopened.
The server hands over held messages the instant the connection opens. The client, meanwhile, loads its decryption keys from browser storage, which is asynchronous.
So the messages arrived before there was anything to decrypt them with, and were dropped.
My tests never saw it because in tests the key loading was effectively instant. The window between "connected" and "ready to decrypt" existed only on a real machine doing real I/O.
The lesson: if your test setup completes instantly and production does not, you are not testing the same system. Anywhere your code says await between "we are live" and "we are ready", something can arrive in between.
The server deletes a held message once the client confirms it. My client confirmed on arrival.
Arrival is not delivery. The message had arrived at the socket, but the app had not decrypted it, had not stored it, had not shown it to anyone. The server deleted it anyway. The message was gone from both sides.
The fix was to confirm only after the message had actually been handled, and to leave anything unhandled with the server so it comes again next time.
There is one deliberate exception: a message this device can never read, because it belongs to a conversation whose keys are gone, is still confirmed. Asking for it again would not help.
The lesson: "received" and "handled" are different events. Only one of them is safe to delete on. If you are building any at-least-once delivery, be precise about which one you are acting on.
Once messages were flowing again, I looked in the database and found a copy of every message I had sent, including ones delivered instantly while both people were online.
The logic was: store every message, delete it when the recipient confirms. But confirmation only happens for messages delivered from storage. A message delivered live goes straight to the other person, so nothing ever confirms it, so nothing ever deletes it.
An active conversation was quietly accumulating a server side copy of itself, and only a weekly sweep cleared it.
The lesson: a delete that only runs on one code path is not a delete. When you write "we clean this up later", check that every path reaches the cleanup, not just the one you had in mind.