Production deployment

A real deployment never builds on the box. It pulls tagged images from GHCR and applies docker-compose.prod.yml, which is what binds the api and frontend to 127.0.0.1, removes the Postgres, Redis, search and optional-profile port publishes entirely, and adds restart policies, log caps, resource limits and TLS.

Images are ghcr.io/yegamble/{vidra-core,vidra-user,vidra-search}:vX.Y.Z, built only from tags. The current release line is v0.5.0.

The env file

env/production.env is generated by vidra setup and is docker-compose format, not shell.

Never source env/production.env

vidra setup generates VIDRA_COMPOSE_PROFILES=core frontend — unquoted, with a space. Compose's --env-file parser reads that as the literal string core frontend, which is correct. A shell that sources the same file sets VIDRA_COMPOSE_PROFILES=core and then tries to execute frontend.

Parse it as KEY=VALUE with the value taken literally, or hand it to docker compose --env-file. The same applies to any value with a space, such as a multi-word INSTANCE_NAME.

The failure is quiet, too: a set -e script that pipes its output to tail reports the pipeline's exit status, so a run that died on line 1 exits 0 and looks like a run that found nothing to do.

Set it up by hand — this is exactly what the installer automates:

git clone https://github.com/yegamble/vidra.git /opt/vidra && cd /opt/vidra
./bootstrap.sh                          # clones the three component repos
cp env/production.env.example env/production.env
$EDITOR env/production.env              # JWT_SECRET, POSTGRES_PASSWORD, REDIS_PASSWORD,
                                        # MFA_KEY_KEK, SEARCH_INTERNAL_SECRET, SMTP_*,
                                        # STORAGE_S3_*, INSTANCE_NAME, PUBLIC_BASE_URL,
                                        # VIDRA_*_TAG, REGISTRATION_ENABLED=false
git check-ignore -v env/production.env  # MUST match, or stop and fix .gitignore

vidra setup --template env/production.env.example --yes
                                        # renders deploy/Caddyfile.local from the template
                                        # + PUBLIC_BASE_URL/VIDRA_TLS_MODE

./deploy/compose.sh config -q           # render check — catches missing required vars
./deploy/compose.sh pull
./deploy/compose.sh run --rm migrate && ./deploy/compose.sh run --rm search-migrate
./deploy/compose.sh up -d --no-build

curl -fsS http://127.0.0.1:8080/readyz  # {"status":"ok"} incl. postgres + redis
curl -fsS https://example.com/          # Caddy + certificate + frontend
nmap -Pn -p 5432,6379,8080,3000 <host-ip>   # must all be closed

deploy.sh refuses while deploy/Caddyfile.local is missing, still says example.com, or serves a different host than PUBLIC_BASE_URL. It also refuses an ACME deploy whose domain does not yet resolve to this host, because Let's Encrypt rate-limits failures; VIDRA_SKIP_DNS_PREFLIGHT=1 overrides.

POSTGRES_PASSWORD on an existing volume

Postgres reads that variable only at initdb. If the data volume already exists, put the database's current password in the env file, not a freshly generated one, and rotate with ALTER ROLE … PASSWORD inside the container first.

Every deploy after the first

./deploy/deploy.sh        # or: vidra deploy, or: make deploy

deploy.sh is the gated pipeline, and the ordering is the point:

Refuse below Compose 2.24. Silent failure mode: the !reset tags are ignored, the deploy reports success, and the database is on the public internet.
Pin nested checkouts to VIDRA_*_TAG, so the migrators run the matching schema versions.
Integrity-checked pre-deploy pg_dump, which aborts the deploy on failure. A deploy with no dump behind it is not a deploy you can undo.
Pull before stop. The old release keeps serving while images download.
Two exit-code-gated migrators, run as separate steps so you can see which one failed, with the final ledger state asserted against vidra-core/migrations including dirty=false.
up -d --no-build, then the Caddy reload.
Health probes — api, frontend and the edge.

It is mode-aware: VIDRA_TLS_MODE decides which of those still apply, and every skip is printed with its reason. If you did not read a skip line, the check ran.

There is no auto-rollback

Failed probes leave the broken release running. deploy.sh tells you; it does not undo. Rolling back is a deliberate operator action — see Upgrading and rollback.

The restart window

Step 4 recreates the api and frontend containers, so both are briefly gone. Caddy buffers that window rather than exposing it: lb_try_duration holds an arriving request and re-dials every 250 ms — 30 s for the api, 45 s for the frontend, which cannot start until the api is healthy. Callers get a slow response instead of a 502.

What that is not: it is not blue-green. There is one instance of each service and nothing serves while the replacement boots; the wait is merely spent inside Caddy. Requests already in flight are not covered — they are mid-response on the old container when it gets SIGTERM, so their fate is the graceful drain (HTTP_SHUTDOWN_TIMEOUT, 20 s by default, inside a 30 s stop_grace_period). Past the duration it is a 502 again: the buffer covers a normal recreate, not a boot-looping api.

TLS topologies

VIDRA_TLS_MODE is not only a certificate setting — it decides whether this host runs an edge at all.

Mode	Who terminates	What the deploy still checks
`acme` (default)	Managed Caddy, Let's Encrypt	Everything
`acme-staging`	Managed Caddy, LE staging	Everything
`internal`	Managed Caddy, its own local CA	Everything
`external`	Your proxy, load balancer or CDN	Caddy is not in the project at all; the Caddyfile check, the reload and the DNS preflight are skipped, and the edge probe becomes a warning
`plain-http`	Nobody	Caddy runs as a plain-HTTP site; `require_real_domain` and DNS preflight are skipped

With `external`, you own four things

Routing. Forward /api/*, /healthz, /readyz, /version, /sitemap.xml, /feeds/*, /nodeinfo/* and /.well-known/* to the api on 127.0.0.1:${HTTP_PORT} (default 8080); everything else to the frontend on 127.0.0.1:${FRONTEND_PORT} (default 3000). vidra setup writes deploy/nginx-external.conf.example mirroring the split exactly — start from it. A hand-written proxy that sends /feeds/* to the frontend 404s every feed link and nothing errors.
Headers. X-Forwarded-Proto: https and X-Forwarded-For. Without the first, the api believes it is serving plain HTTP and mints http:// links.
Upload limits and timeouts. Your proxy's body-size limit is now the upload limit. nginx's default is 1 MB.
TRUSTED_PROXY_CIDRS, but only when the terminator has a public IP. The api already trusts loopback, private and link-local sources. List only ranges you control — trusting a range you do not own lets anyone in it forge the header.

plain-http puts everything in the clear

PUBLIC_BASE_URL must be http://, and because the api applies production validation rules whatever VIDRA_ENV says, that origin is a hard refusal until you also set VIDRA_ALLOW_PLAIN_HTTP=true. That switch is the consent, and it is what turns off Secure cookies and HSTS. Every credential, cookie and upload crosses the network in the clear. Use it behind a VPN, on an isolated network, or through an SSH tunnel — never on anything the internet can reach. Federation and OAuth are https-only by design and will not work there.

Backups

./deploy/backup.sh, and the systemd timer that runs it, writes two files per run under the same UTC stamp:

File	Contents	Kept
`backups/vidra-<UTC>.dump.gz`	Custom-format `pg_dump`, verified with `pg_restore -l` before it is kept	14 daily + 8 weekly
`backups/vidra-config-<UTC>.tar.gz`	`env/production.env` + `deploy/Caddyfile.local`, repo-relative, mode 0600	14 daily + 8 weekly

plus a backups/last_success marker, whose format is a contract vidra doctor reads to judge backup age against a 26-hour window.

The config archive is why a dump is restorable at all. env/production.env holds MFA_KEY_KEK, which is generated once and is not derivable again — restore a database without it and every user's second factor is undecryptable. deploy/Caddyfile.local is gitignored and existed only on the host that died.

Off-site is opt-in. Set BACKUP_RCLONE_REMOTE, or BACKUP_S3_URI plus BACKUP_S3_ENDPOINT. Use a bucket in a different region from the media Space. Both files go to the same target, deliberately.
Alert on a missing backup, not just a failing one. Set HEALTHCHECKS_URL; the script pings /start, /fail on error, and success at the end, so a host that stops running the timer at all still pages you.
Redis needs no backup. It is a cache and rate-limit store and may be flushed at any time.
Media is in neither file. See Storage and media.
With VIDRA_EXTERNAL_POSTGRES=true, both backup.sh and restore.sh refuse outright. Use your provider's backups and PITR — and back up env/production.env and deploy/Caddyfile.local yourself, because such a host gets no config archive either.

Prove it before you need it:

sudo ./deploy/provision.sh --yes                     # installs AND verifies the timer
./deploy/backup.sh                                   # prove it works
RESTORE_CONFIRM=vidra ./deploy/restore.sh backups/<latest>.dump.gz   # on a SCRATCH stack
sudo reboot                                          # confirm every container returns

enable --now on a timer whose .service fails to parse can still exit 0, and that failure surfaces as "no backups have ever run" six weeks later, during a restore. provision.sh verifies with systemctl is-enabled / is-active for that reason, and vidra doctor checks the timer and the age of backups/last_success.

Run a restore drill quarterly.

Disaster recovery, in the non-obvious order

restore.sh refuses to run while deploy/Caddyfile.local is missing — the prod compose file mounts it, and a missing bind-mount source is created by Docker as an empty directory, which crash-loops Caddy with every app container perfectly healthy. So configuration lands before the database:

# 1. A host and a checkout.
sudo ./deploy/provision.sh --yes
git clone https://github.com/yegamble/vidra.git /opt/vidra && cd /opt/vidra
./bootstrap.sh

# 2. Fetch BOTH files for the SAME stamp from off-site.
rclone copy "$BACKUP_RCLONE_REMOTE/vidra-config-<stamp>.tar.gz" backups/
rclone copy "$BACKUP_RCLONE_REMOTE/vidra-<stamp>.dump.gz"        backups/

# 3. Configuration FIRST.
tar -xzf backups/vidra-config-<stamp>.tar.gz -C /opt/vidra
git check-ignore -v env/production.env               # MUST match

# 4. Bring up just enough to restore into, then restore.
./deploy/compose.sh up -d postgres
./deploy/restore.sh backups/vidra-<stamp>.dump.gz

# 5. Media, if STORAGE_BACKEND=local.
# 6. Point DNS at the new host, then ./deploy/deploy.sh.

The env file pins VIDRA_*_TAG, so the rebuilt host comes back on the release the dump was taken under — which is what you want, and is worth reading rather than assuming.

Migration failed mid-deploy

Both migrators drive golang-migrate as a library inside the service binary, with the SQL compiled into the image. That library marks its ledger dirty=true when a migration fails part-way and then refuses every subsequent up with an opaque error, and because the api gates on the migrator completing successfully, the site stays down.

There are two independent ledgers: public.schema_migrations (core) and public.vidra_search_migrations (search). Either can go dirty without the other.

# 1. Find out where it stopped. Note the REPEATED word: `run <service> <args>`
#    REPLACES the service's command, so the subcommand must be restated.
./deploy/compose.sh run --rm migrate        migrate version
./deploy/compose.sh run --rm search-migrate migrate version

# 2. Work out what that migration did before it failed and undo the partial
#    effect BY HAND. golang-migrate does not roll back for you.
./deploy/compose.sh exec postgres psql -U vidra -d vidra

# 3. Point the ledger at the last CLEAN version, N-1. `force` stamps the version
#    and clears dirty WITHOUT running any SQL — it is an assertion about the
#    schema you just repaired. --yes-i-know is checked before the database is
#    touched, so a refusal means nothing happened.
./deploy/compose.sh run --rm migrate        migrate force 41 --yes-i-know

# 4. Re-run the normal migrator.
./deploy/compose.sh run --rm migrate

Never point the ledger at N — that claims the broken migration succeeded, and the next deploy builds on a schema that does not exist. force -1 is the right target when it was the first migration that died.

A hung migration is not a crashed one

None of the up-migrations use CREATE INDEX CONCURRENTLY and none sets lock_timeout, so a data-dependent migration against a populated table stalls rather than fails, with api boot gated behind it. Look for a blocking lock in pg_stat_activity and pg_locks before assuming it died.

Schema compatibility is enforced, not just documented

Release N−1's code must run against release N's schema. That is what makes a rollback a 60-second tag flip that never touches the database, and it is why a true schema rollback is "restore the pre-deploy dump" rather than "migrate down".

make ci in both Go repos runs scripts/migrate-lint.sh, which rejects destructive forward DDL — DROP TABLE/COLUMN, RENAME, TRUNCATE, SET NOT NULL, ALTER … TYPE, DELETE FROM — in any *.up.sql. Down migrations are exempt, since they are the rollback path, and -- migrate-lint:allow is the escape hatch for an accepted break. On top of that, vidra-core's schema-compat workflow applies HEAD's migrations to a fresh database and runs the previous release tag's integration suite against it.

Email

DigitalOcean blocks outbound port 25 on new accounts. Use a relay (Postmark, SES, Mailgun, Resend, Fastmail) on port 587 with STARTTLS and AUTH, which is what SMTP_PORT=587 already defaults to.
SMTP_FROM must be a domain you control, with SPF and DKIM published for the relay, plus DMARC once those pass. Without them, password-reset mail lands in spam and users conclude the site is broken.
MAIL_ENABLED=true requires SMTP_HOST and SMTP_FROM; the api refuses to boot in production otherwise.

Never set DEV_MAIL_CAPTURE_ENABLED=true in production

It exposes GET /api/v1/dev/email-token, which returns a live password-reset token for any address. It is gated in config validation, gated at route registration, and 404'd at the edge. Do not go looking for a fourth way around it. The same goes for HTTP_IMPORT_ALLOW_PRIVATE_URLS=true: both are set in env/qa.env.example on purpose and must never be copied into a production env file.

Staging is production config with throwaway data

Promote by deploying the exact image tags staging validated: deploy.sh and rollback.sh both work from VIDRA_CORE_TAG, VIDRA_USER_TAG and VIDRA_SEARCH_TAG in the env file, so promotion is copying three lines.

The same image serves every environment. The frontend container reads PUBLIC_API_BASE_URL and serves it to the browser as /runtime-config.js, so the origin is a restart, not a rebuild.

All the compose-based Make targets honour PROD_ENV_FILE=env/staging.env, and every deploy/*.sh honours ENV_FILE=env/staging.env.

The env file​

Every deploy after the first​

The restart window​

TLS topologies​

With external, you own four things​

Backups​

Disaster recovery, in the non-obvious order​

Migration failed mid-deploy​

Schema compatibility is enforced, not just documented​

Email​

Staging is production config with throwaway data​