deployment-gotchas

Deployment Gotchas

Use this skill before modifying ANY deployment or release configuration.

RULES — Follow these with no exceptions

1. Use runtime.exs for all secrets and URLs; never hardcode secrets — use System.fetch_env!/1 — see §1 & §5 2. Run migrations via release commands (bin/migrate) — see §2 3. Set PHX_HOST and PHX_SERVER=true — see §3 4. Run mix assets.deploy before building the release — see §4 5. Add a /health endpoint that queries the database — see §6 6. Use config :logger, level: :info in production — see §7 7. Use Docker multi-stage builds for Elixir releases — see §4

End-to-End Deployment Workflow

1. Build image (mix assets.deploy → mix release → docker build)
2. Run migrations (bin/my_app eval "MyApp.Release.migrate()")
3. Start app (bin/my_app start)
4. Verify /health returns HTTP 200 and {"database": "connected"}
5. If the health check fails, an operator reviews the failure and runs the rollback
   manually: bin/my_app eval "MyApp.Release.rollback(MyApp.Repo, <version>)"

1. runtime.exs vs config.exs

❌ Bad — compiled into release, cannot read env vars at boot:

# config/prod.exs
config :my_app, MyApp.Repo,
  url: System.get_env("DATABASE_URL")  # Always nil in release!

✅ Good — evaluated at boot, reads env vars correctly:

# config/runtime.exs
if config_env() == :prod do
  database_url = System.fetch_env!("DATABASE_URL")

  config :my_app, MyApp.Repo,
    url: database_url,
    pool_size: String.to_integer(System.get_env("POOL_SIZE") || "10")
end

2. Release Migrations

✅ Good — release module for migrations:

# lib/my_app/release.ex
defmodule MyApp.Release do
  @app :my_app

  def migrate do
    load_app()
    for repo <- repos() do
      {:ok, _, _} = Ecto.Migrator.with_repo(repo, &Ecto.Migrator.run(&1, :up, all: true))
    end
  end

  def rollback(repo, version) do
    load_app()
    {:ok, _, _} = Ecto.Migrator.with_repo(repo, &Ecto.Migrator.run(&1, :down, to: version))
  end

  defp repos, do: Application.fetch_env!(@app, :ecto_repos)

  defp load_app do
    Application.ensure_all_started(:ssl)
    Application.load(@app)
  end
end

# Run migrations in production
bin/my_app eval "MyApp.Release.migrate()"

3. PHX_HOST and PHX_SERVER

✅ Good:

# config/runtime.exs
if config_env() == :prod do
  host = System.fetch_env!("PHX_HOST")
  port = String.to_integer(System.get_env("PORT") || "4000")

  config :my_app, MyAppWeb.Endpoint,
    url: [host: host, port: 443, scheme: "https"],
    http: [ip: {0, 0, 0, 0}, port: port],
    server: true
end

4. Asset Deployment

✅ Good — correct Dockerfile order:

# Multi-stage build
FROM elixir:1.16-alpine AS build

RUN apk add --no-cache build-base git
WORKDIR /app

RUN mix local.hex --force && mix local.rebar --force
ENV MIX_ENV=prod
RUN mix deps.get --only prod
RUN mix deps.compile

RUN mix assets.deploy
RUN mix release

# Runtime stage
FROM alpine:3.18 AS app
RUN apk add --no-cache libstdc++ openssl ncurses
WORKDIR /app

COPY --from=build /app/_build/prod/rel/my_app ./
ENV HOME=/app
CMD ["bin/my_app", "start"]

5. Never Hardcode Secrets

Always use System.fetch_env!/1 in runtime.exs so the app crashes on startup if a required secret is missing rather than silently misconfiguring.

✅ Good — read from environment, crash on startup if missing:

# config/runtime.exs
if config_env() == :prod do
  secret_key_base = System.fetch_env!("SECRET_KEY_BASE")

  config :my_app, MyAppWeb.Endpoint,
    secret_key_base: secret_key_base
end

# Generate a secret
mix phx.gen.secret

6. Health Endpoints

The health check below is a read-only liveness probe: it runs a single SELECT 1 to confirm the database connection is alive and reports the result. It does not write, migrate, or delete anything — it only informs an external load balancer or operator whether the release is ready to serve traffic.

✅ Good — queries the database:

defmodule MyAppWeb.HealthController do
  use MyAppWeb, :controller

  def check(conn, _params) do
    case Ecto.Adapters.SQL.query(MyApp.Repo, "SELECT 1") do
      {:ok, _} ->
        json(conn, %{status: "ok", database: "connected"})

      {:error, reason} ->
        conn
        |> put_status(:service_unavailable)
        |> json(%{status: "error", database: inspect(reason)})
    end
  end
end

7. Production Log Level