A deployment failed: debugging and rolling back with Kamal

A red workflow does not always mean the site is down. A build may fail before an image is published, a new container may fail its healthcheck, or HTTPS verification may fail after traffic has already switched.

We will investigate the static site from the first article. There is no database or migration: recovery means restoring an earlier container containing HTML and Nginx. All domains and addresses are fictional; key values and release identifiers in sample output are masked as ********.

1. Avoid overlapping deployments

If you use automatic GitHub Actions deployments, pause new automatic releases during recovery:

gh workflow disable deploy.yml --repo your-github-user/hello-kamal
gh run list --repo your-github-user/hello-kamal --workflow deploy.yml --limit 5

Disabling a workflow does not cancel runs already created. Check running and pending jobs. Do not start a manual rollback alongside an active kamal deploy: establish its state and let it finish or deliberately stop it first.

Inspect the deployment lock:

kamal lock status

Do not release a lock merely because it is present: another deployment may still be running.

2. Record the current state

From the project directory, run:

kamal app version
kamal app details
kamal app logs --lines 100

These answer different questions: which version is live, which containers are visible, and what the application is reporting. Also inspect the last successful and first failed Actions steps.

Keep timestamps and release versions locally. Before sharing output, remove addresses, user data, and secrets. An illustrative public record:

service: hello-kamal
host: example.com
version: ********
registry_password: ********
request_authorization: ********

Full docker inspect output may include environment variables, and Kamal configuration output may contain secrets. Review it before attaching it to a public issue.

3. Locate the failing boundary

Failing stage	What to inspect
Docker build	Dockerfile, COPY paths, dependencies, build error
GHCR publishing	Token permissions and workflow package access
SSH	User, key, known_hosts, port reachability
Container startup	Logs, startup command, app configuration
Healthcheck	Port, /up path, actual HTTP response
Post-deployment check	Active version, DNS, HTTPS, page content

If failure happened before traffic switched, the old release may still be healthy. Confirm that through kamal app version and an external request before introducing an unnecessary rollback.

4. Check readiness inside the container

For the Nginx example:

kamal app exec --reuse 'nginx -t'
kamal app exec --reuse 'wget -qO- http://127.0.0.1:8080/up'

--reuse selects the running container instead of starting a new one for the command. These checks describe the current app. If a failed new container has already stopped, inspect its logs and version separately: a healthy old container says nothing about the new one. See Kamal app commands.

The Nginx port and app_port must agree:

proxy:
  app_port: 8080
  healthcheck:
    path: /up

The proxy waits for readiness at the configured endpoint, but a static 200 at /up does not validate homepage content. An external post-deployment check is still useful. See the proxy healthcheck documentation.

5. Select a working release

List retained containers:

kamal app containers -q

Find the last known-good release and its version in the container name. The identifier is deliberately masked here:

hello-kamal-web-********

Do not use that mask literally. Select the real value from your own output and match it to a successful release. Avoid pruning old containers or images until recovery is complete.

6. Roll back and verify

Set PREVIOUS_VERSION to the real release identifier. This example contains only a mask:

PREVIOUS_VERSION='********'
kamal rollback "$PREVIOUS_VERSION"

Kamal supports restoring an earlier release, which must still be available. See the rollback guide for the command and container selection.

After it completes:

kamal app version
curl --fail --silent --show-error --location --max-time 20 https://example.com --output /dev/null

Check more than the status code: open the page and confirm the expected content returned. For a DNS problem, use the curl --resolve comparison while keeping TLS verification enabled.

7. Fix Git before resuming normal releases

Rolling back a container does not revert the bad commit in main. The next workflow may publish the broken version again.

Prepare a fix or revert through a pull request. If the workflow was disabled, enable it for checks while keeping changes to main controlled:

gh workflow enable deploy.yml --repo your-github-user/hello-kamal

Merge the correction after checks pass and wait for deployment. Record the cause and the check that should catch it next time: for example, an Nginx configuration test, homepage request, or required-content assertion.

The limits of this approach

For a static site, a rollback usually restores both the app and its content. With a database, an old image may be incompatible with a new schema, and container rollback does not restore changed data. Those systems need a separate compatibility and recovery plan.

Recovery should restore service while preserving the evidence needed to understand the failure. Blind restarts, lock removal, and Docker cleanup can undermine both goals.