Seven green deploys that never shipped

For about twenty-four hours, nothing we merged reached production. Seven deploys ran, all seven went green, and the API kept serving the build from the night before. Nobody noticed because nothing was down: the old container was still answering. We found it by accident while applying production migrations.

Two bugs, each hiding the other

The new container could not start

The container runs ./migrate && exec ./server. Drizzle's migrator decides what is pending by name, and it skips rows that have no name.

One migration had been applied out of band with drizzle-kit, which recorded it without a name. On every boot, the migrator therefore believed it was still pending and ran its ALTER TABLE ... ADD COLUMN again. Postgres answered that the column already existed, migrate exited 1, the && stopped there, and the server never started listening. The health check failed, so the deploy tool kept the previous container — exactly as it should.

The failed rollout passed CI

haloy deploy exits 0 even when a rollout fails its health check. Our GitHub Actions step was a bare run: haloy deploy, so the job went green. Seven times.

The first bug stopped every release. The second reported each one as shipped. Together, they let production stay healthy enough to hide that it was a day behind.

Why not delete the row?

Deleting the nameless row, or rewriting the migration with IF NOT EXISTS, would also have let the container boot. Both answer the only question that matters here — was this migration applied? — with a guess on the production database.

We repaired the state instead:

• Before migrating, the script now matches nameless rows to our migrations by hash and restores their names. A matching hash is evidence that the migration already ran; recording that fact avoids rerunning the DDL or modifying application data.
• The deploy workflow reads haloy's output and fails when the rollout fails or never reports a healthy check.

We verified the repair by reproducing the exact state in development — a nameless row for a migration that had already run — and executing the same binary as the container. It exited 0, repaired the row and ran no DDL twice.

What I take away

• Migration state is verified by hash, never by name. A name is a label; the hash is the evidence.
• A deploy that cannot fail CI is not a verified deploy. If the tool exits 0 on a failed rollout, read its output and fail the job yourself.
• A green health check is not proof that new code is running. The cheap confirmation is to ask the API for something only the new version can answer.