# Seven green deploys that never shipped

- Date: 13 July 2026
- Project: restoman
- Tags: DevOps, Postgres, CI

> Restoman's API served the previous night's build for a day while seven deploys reported success. A stuck migration and a deploy command that exited 0 on failure hid each other.

For about twenty-four hours, nothing we merged reached production. Seven
deploys ran, all seven went green, and the API kept serving the build from the
night before. Nobody noticed because nothing was down: the old container was
still answering. We found it by accident while applying production migrations.

## Two bugs, each hiding the other

### The new container could not start

The container runs `./migrate && exec ./server`. Drizzle's migrator decides
what is pending **by name**, and it skips rows that have no name.

One migration had been applied out of band with `drizzle-kit`, which recorded
it without a name. On every boot, the migrator therefore believed it was still
pending and ran its `ALTER TABLE ... ADD COLUMN` again. Postgres answered that
the column already existed, `migrate` exited 1, the `&&` stopped there, and the
server never started listening. The health check failed, so the deploy tool
kept the previous container — exactly as it should.

### The failed rollout passed CI

`haloy deploy` exits **0** even when a rollout fails its health check. Our
GitHub Actions step was a bare `run: haloy deploy`, so the job went green.
Seven times.

The first bug stopped every release. The second reported each one as shipped.
Together, they let production stay healthy enough to hide that it was a day
behind.

## Why not delete the row?

Deleting the nameless row, or rewriting the migration with `IF NOT EXISTS`,
would also have let the container boot. Both answer the only question that
matters here — _was this migration applied?_ — with a guess on the production
database.

We repaired the state instead:

- Before migrating, the script now matches nameless rows to our migrations
  **by hash** and restores their names. A matching hash is evidence that the
  migration already ran; recording that fact avoids rerunning the DDL or
  modifying application data.
- The deploy workflow reads haloy's output and **fails** when the rollout fails
  or never reports a healthy check.

We verified the repair by reproducing the exact state in development — a
nameless row for a migration that had already run — and executing the same
binary as the container. It exited 0, repaired the row and ran no DDL twice.

## What I take away

- **Migration state is verified by hash, never by name.** A name is a label; the
  hash is the evidence.
- **A deploy that cannot fail CI is not a verified deploy.** If the tool exits 0
  on a failed rollout, read its output and fail the job yourself.
- **A green health check is not proof that new code is running.** The cheap
  confirmation is to ask the API for something only the new version can answer.

---

Page: https://eitanf.com/notes/seven-green-deploys · Español: https://eitanf.com/es/notes/seven-green-deploys.md · llms.txt: https://eitanf.com/llms.txt