Production deployment

A real deployment never builds on the box. It pulls tagged images from GHCR and applies docker-compose.prod.yml, which is what binds the api and frontend to 127.0.0.1, removes the Postgres, Redis, search and optional-profile port publishes entirely, and adds restart policies, log caps, resource limits and TLS.

Images are ghcr.io/yegamble/{vidra-core,vidra-user,vidra-search}:vX.Y.Z, built only from tags. The newest release is on the vidra-core releases page, and each release's component pairing is a record under releases/.

The env file

env/production.env is generated by vidra setup and is docker-compose format, not shell.

Never source env/production.env

vidra setup generates VIDRA_COMPOSE_PROFILES=core frontend — unquoted, with a space. Compose's --env-file parser reads that as the literal string core frontend, which is correct. A shell that sources the same file sets VIDRA_COMPOSE_PROFILES=core and then tries to execute frontend.

Parse it as KEY=VALUE with the value taken literally, or hand it to docker compose --env-file. The same applies to any value with a space, such as a multi-word INSTANCE_NAME.

The failure is quiet, too: a set -e script that pipes its output to tail reports the pipeline's exit status, so a run that died on line 1 exits 0 and looks like a run that found nothing to do.

Set it up by hand — this is exactly what the installer automates:

git clone https://github.com/yegamble/vidra.git /opt/vidra && cd /opt/vidra
./bootstrap.sh                          # clones the three component repos

vidra setup --template env/production.env.example
                                        # asks for the domain, TLS mode, instance name,
                                        # release tag, storage, SMTP and registration;
                                        # writes env/production.env AND deploy/Caddyfile.local
git check-ignore -v env/production.env  # MUST match, or stop and fix .gitignore

./deploy/deploy.sh                      # or: vidra deploy

curl -fsS http://127.0.0.1:8080/readyz  # {"status":"ok"} incl. postgres + redis
curl -fsS https://example.com/          # Caddy + certificate + frontend
nmap -Pn -p 5432,6379,8080,3000 <host-ip>   # must all be closed

You do not hand-edit any secret. vidra setup mints JWT_SECRET, MFA_KEY_KEK, POSTGRES_PASSWORD, REDIS_PASSWORD and SEARCH_INTERNAL_SECRET and writes the file mode 0600. Scripted installs pass the same answers as flags; see Operator CLI.

Do not cp the template to env/production.env first

vidra setup reads env/production.env back before it writes it, and a copy of the template counts as an existing configuration. On anything but a first install it refuses to mint a blank key-encryption key, because on a live database a fresh one would orphan every sealed row, so it stops with MFA_KEY_KEK is blank in the configuration being merged. If you already copied the template and have never run a deploy, delete the copy and run vidra setup as above.

Run ./deploy/deploy.sh for the first deploy too, not hand-run migrate and up -d. On a fresh host it skips the pre-deploy dump because there is no database yet, and says so; everything else still runs. deploy/compose.sh is for reading and stopping — it gates nothing, so a run --rm migrate followed by up -d through it skips the Compose version check, the migrator tag floor, the Caddyfile and DNS checks, the independent ledger assertion against the expected migration version, and the health probes.

deploy.sh refuses while deploy/Caddyfile.local is missing, still says example.com, or serves a different host than PUBLIC_BASE_URL. It also refuses an ACME deploy whose domain does not yet resolve to this host, because Let's Encrypt rate-limits failures; VIDRA_SKIP_DNS_PREFLIGHT=1 overrides.

Federation is off after setup, and the interview does not ask about it. To turn it on, see Turn federation on after install.

POSTGRES_PASSWORD on an existing volume

Postgres reads that variable only at initdb. If the data volume already exists, put the database's current password in the env file, not a freshly generated one, and rotate with ALTER ROLE … PASSWORD inside the container first.

Every deploy after the first

./deploy/deploy.sh        # or: vidra deploy, or: make deploy

deploy.sh is the gated pipeline, and the ordering is the point:

1. Refuse below Compose 2.24. Silent failure mode: the !reset tags are ignored, the deploy reports success, and the database is on the public internet.
2. Pin nested checkouts to VIDRA_*_TAG, so the migrators run the matching schema versions.
3. Integrity-checked pre-deploy pg_dump, which aborts the deploy on failure. A deploy with no dump behind it is not a deploy you can undo.
4. Pull before stop. The old release keeps serving while images download.
5. Two exit-code-gated migrators, run as separate steps so you can see which one failed, with the final ledger state asserted against vidra-core/migrations including dirty=false.
6. up -d --no-build, then the Caddy reload.
7. Health probes — api, frontend and the edge.

It is mode-aware: VIDRA_TLS_MODE decides which of those still apply, and every skip is printed with its reason. If you did not read a skip line, the check ran.

There is no auto-rollback

Failed probes leave the broken release running. deploy.sh tells you; it does not undo. Rolling back is a deliberate operator action — see Upgrading and rollback.

The restart window

Step 4 recreates the api and frontend containers, so both are briefly gone. Caddy buffers that window rather than exposing it: lb_try_duration holds an arriving request and re-dials every 250 ms — 30 s for the api, 45 s for the frontend, which cannot start until the api is healthy. Callers get a slow response instead of a 502.

What that is not: it is not blue-green. There is one instance of each service and nothing serves while the replacement boots; the wait is merely spent inside Caddy. Requests already in flight are not covered — they are mid-response on the old container when it gets SIGTERM, so their fate is the graceful drain (HTTP_SHUTDOWN_TIMEOUT, 20 s by default, inside a 30 s stop_grace_period). Past the duration it is a 502 again: the buffer covers a normal recreate, not a boot-looping api.

TLS topologies

VIDRA_TLS_MODE is not only a certificate setting — it decides whether this host runs an edge at all.

Mode	Who terminates	What the deploy still checks
acme (default)	Managed Caddy, Let's Encrypt	Everything
acme-staging	Managed Caddy, LE staging	Everything
internal	Managed Caddy, its own local CA	Everything
external	Your proxy, load balancer or CDN	Caddy is not in the project at all; the Caddyfile check, the reload and the DNS preflight are skipped, and the edge probe becomes a warning
plain-http	Nobody	Caddy runs as a plain-HTTP site; require_real_domain and DNS preflight are skipped

With external, you own four things

1. Routing. Forward /api/*, /healthz, /readyz, /version, /sitemap.xml, /feeds/*, /nodeinfo/* and /.well-known/* to the api on 127.0.0.1:${HTTP_PORT} (default 8080); everything else to the frontend on 127.0.0.1:${FRONTEND_PORT} (default 3000). vidra setup writes deploy/nginx-external.conf.example mirroring the split exactly — start from it. A hand-written proxy that sends /feeds/* to the frontend 404s every feed link and nothing errors.
2. Headers. X-Forwarded-Proto: https and X-Forwarded-For. Without the first, the api believes it is serving plain HTTP and mints http:// links.
3. Upload limits and timeouts. Your proxy's body-size limit is now the upload limit. nginx's default is 1 MB.
4. TRUSTED_PROXY_CIDRS, but only when the terminator has a public IP. The api already trusts loopback, private and link-local sources. List only ranges you control — trusting a range you do not own lets anyone in it forge the header.

plain-http puts everything in the clear

PUBLIC_BASE_URL must be http://, and because the api applies production validation rules whatever VIDRA_ENV says, that origin is a hard refusal until you also set VIDRA_ALLOW_PLAIN_HTTP=true. That switch is the consent, and it is what turns off Secure cookies and HSTS. Every credential, cookie and upload crosses the network in the clear. Use it behind a VPN, on an isolated network, or through an SSH tunnel — never on anything the internet can reach. Federation and OAuth are https-only by design and will not work there.

Backups

./deploy/backup.sh, and the systemd timer that runs it, writes two files per run under the same UTC stamp:

File	Contents	Kept
backups/vidra-<UTC>.dump.gz	Custom-format pg_dump, verified with pg_restore -l before it is kept	14 daily + 8 weekly
backups/vidra-config-<UTC>.tar.gz	env/production.env + deploy/Caddyfile.local, repo-relative, mode 0600	14 daily + 8 weekly

plus a backups/last_success marker, whose format is a contract vidra doctor reads to judge backup age against a 26-hour window.

The config archive is why a dump is restorable at all. env/production.env holds MFA_KEY_KEK, which is generated once and is not derivable again — restore a database without it and every user's second factor is undecryptable. deploy/Caddyfile.local is gitignored and existed only on the host that died.

v0.7.5 ignores these keys in the env file

backup.sh in v0.7.5 and earlier reads BACKUP_RCLONE_REMOTE, BACKUP_S3_URI, BACKUP_S3_ENDPOINT and HEALTHCHECKS_URL only from the process environment, not from env/production.env. The shipped vidra-backup.service sets only ENV_FILE, so a host that put them in the env file has been taking local-only, unmonitored backups. The journal says so on every run: journalctl -u vidra-backup.service | grep 'LOCAL COPY ONLY'. BACKUP_AGE_RECIPIENTS does not exist in v0.7.5, so the only way to encrypt off-site copies there is an rclone crypt remote.

To get off-site backups on v0.7.5, put the values in a root-owned file and load it with a systemd drop-in:

sudo install -d -m 0700 /etc/vidra
sudo install -m 0600 /dev/null /etc/vidra/backup.env
sudoedit /etc/vidra/backup.env        # BACKUP_RCLONE_REMOTE=...  HEALTHCHECKS_URL=...
sudo systemctl edit vidra-backup.service
#   [Service]
#   EnvironmentFile=/etc/vidra/backup.env
sudo systemctl daemon-reload
sudo systemctl start vidra-backup.service && journalctl -u vidra-backup.service -n 30

From v0.7.7 the script reads these keys from the env file, so the upgrade starts uploading whatever the env file names. Set up encryption before you upgrade: see Upgrading from v0.7.5. A value in the process environment, including an empty one, still wins over the env file, so a drop-in you leave in place keeps working.

• Off-site is opt-in. Set BACKUP_RCLONE_REMOTE, or BACKUP_S3_URI plus BACKUP_S3_ENDPOINT, in env/production.env (v0.7.7 and later; see the warning below for v0.7.5). Use a bucket in a different region from the media Space. Both files go to the same target, deliberately.
• Encrypt the off-site copies. The config archive holds JWT_SECRET, every key-encryption key and the database and S3 passwords, so a plaintext upload hands all of it to anyone who can read the bucket. From v0.7.7, backup.sh refuses to upload the config archive in plaintext unless one of three things holds: BACKUP_AGE_RECIPIENTS is set, the rclone remote is type = crypt, or you have set BACKUP_OFFSITE_PLAINTEXT=true to accept the risk. When it refuses, the dump still uploads, the config archive does not, and the run exits non-zero, so systemd marks the unit failed and /fail is pinged if you set HEALTHCHECKS_URL. Before v0.7.7 the script uploaded plaintext and logged a WARNING on every run. Set the variable to one or more age public keys (age1..., space- or comma-separated) and both files are encrypted to them before upload and land off-site as <name>.age. Local copies stay plaintext, so restore.sh reads them unchanged. The host needs age installed: if it is missing, or an encryption fails, the run aborts (and pings /fail, if you set HEALTHCHECKS_URL) rather than uploading plaintext. Generate the pair with age-keygen -o key.txt, put the age1... public key in the env file, and keep key.txt off this server (a password manager or offline media). A private key stored on the host it protects defeats the point, and without it the off-site backups cannot be recovered. If your rclone remote is already a crypt remote you can skip this, and BACKUP_OFFSITE_PLAINTEXT is not needed.
• Local-only is a decision you record. vidra doctor warns under "off-site backup" while neither BACKUP_RCLONE_REMOTE nor BACKUP_S3_URI is set. If keeping backups on the host is deliberate, set BACKUP_OFFSITE_ACK=true to silence it. Only the doctor reads that flag; backup.sh ignores it. The check is a warning, never a failure, so it does not change the exit code, and with VIDRA_EXTERNAL_POSTGRES=true it is skipped.
• Alert on a missing backup, not just a failing one. Set HEALTHCHECKS_URL; the script pings /start, /fail on error, and success at the end, so a host that stops running the timer at all still pages you.
• Redis needs no backup. It is a cache and rate-limit store and may be flushed at any time.
• Media is in neither file. See Storage and media.
• With VIDRA_EXTERNAL_POSTGRES=true, both backup.sh and restore.sh refuse outright. Use your provider's backups and PITR — and back up env/production.env and deploy/Caddyfile.local yourself, because such a host gets no config archive either.

Prove it before you need it:

sudo ./deploy/provision.sh --yes                     # installs AND verifies the timer
sudo -u vidra ./deploy/backup.sh                     # prove it works
sudo reboot                                          # confirm every container returns

Then prove the dump restores, on a scratch host, never this one: copy the dump there and run ./deploy/restore.sh backups/<latest>.dump.gz on that host. restore.sh drops and replaces the database of the host it runs on, so run it bare first: it refuses, and prints which database it would drop and from which dump. Only after reading that, re-run it with --yes. Keep RESTORE_CONFIRM=<database name> for scripted drills on throwaway hosts; pasted onto a live host, it skips that refusal.

enable --now on a timer whose .service fails to parse can still exit 0, and that failure surfaces as "no backups have ever run" six weeks later, during a restore. provision.sh verifies with systemctl is-enabled / is-active for that reason, and vidra doctor checks the timer and the age of backups/last_success.

Run a restore drill quarterly.

Disaster recovery, in the non-obvious order

restore.sh refuses to run while deploy/Caddyfile.local is missing — the prod compose file mounts it, and a missing bind-mount source is created by Docker as an empty directory, which crash-loops Caddy with every app container perfectly healthy. So configuration lands before the database:

# 1. Fetch BOTH files for the SAME stamp from off-site, into a scratch directory
#    that is NOT /opt/vidra (the installer refuses a non-empty one).
mkdir -p ~/dr && cd ~/dr
rclone copy "$BACKUP_RCLONE_REMOTE/vidra-config-<stamp>.tar.gz" .
rclone copy "$BACKUP_RCLONE_REMOTE/vidra-<stamp>.dump.gz"        .
#    If the backups were encrypted (BACKUP_AGE_RECIPIENTS), the off-site names
#    end in .age. Fetch those instead, then decrypt with the private key you
#    kept off the old server. restore.sh and tar need the plaintext files:
#      age -d -i key.txt -o vidra-config-<stamp>.tar.gz vidra-config-<stamp>.tar.gz.age
#      age -d -i key.txt -o vidra-<stamp>.dump.gz       vidra-<stamp>.dump.gz.age

# 2. Find the release the dump was taken under. The archive's env file pins it;
#    read it without extracting anything.
tar -xzOf vidra-config-<stamp>.tar.gz env/production.env | grep '^VIDRA_.*_TAG='

# 3. The release tree and the CLI at THAT tag, WITHOUT the setup interview: it
#    would mint a fresh JWT secret and fresh *_KEK values, and over the restored
#    database that makes every sealed column unreadable.
curl -fsSL https://raw.githubusercontent.com/yegamble/vidra/main/install.sh \
  | sh -s -- --yes --no-setup --ref <VIDRA_CORE_TAG from step 2>
#    A git checkout install: add --git.

# 4. Then the host: AFTER the install, never before. install.sh refuses a
#    non-empty /opt/vidra. provision.sh from v0.7.7 creates /opt/vidra/backups;
#    the tree at an older tag does not, so create it (harmless if it exists).
cd /opt/vidra && sudo ./deploy/provision.sh --yes
sudo install -d -m 0700 -o vidra -g vidra /opt/vidra/backups
sudo install -m 0600 -o vidra -g vidra ~/dr/vidra-* backups/

# 5. Configuration FIRST, as the service user, so the 0600 env file is its.
sudo -u vidra tar -xzf backups/vidra-config-<stamp>.tar.gz -C /opt/vidra
#    A git checkout install only: git check-ignore -v env/production.env  (MUST match)

# 6. Bring up just enough to restore into, then restore. As the service user:
#    it can read the env file and it is the one in the docker group.
sudo -u vidra ./deploy/compose.sh up -d postgres
sudo -u vidra ./deploy/restore.sh backups/vidra-<stamp>.dump.gz   # refuses; read it, then add --yes

# 7. Media, if STORAGE_BACKEND=local.
# 8. Point DNS at the new host, then: sudo -u vidra ./deploy/deploy.sh

If provision.sh already ran on the new host (cloud-init, for example), the install in step 3 refuses because /opt/vidra/backups exists: move it aside, install, then move it back. install.sh --no-setup is fetched from the installer's main, so it is available whichever release you restore.

The env file pins VIDRA_*_TAG, so the rebuilt host comes back on the release the dump was taken under — which is what you want, and is worth reading rather than assuming.

Migration failed mid-deploy

Both migrators drive golang-migrate as a library inside the service binary, with the SQL compiled into the image. That library marks its ledger dirty=true when a migration fails part-way and then refuses every subsequent up with an opaque error, and because the api gates on the migrator completing successfully, the site stays down.

There are two independent ledgers: public.schema_migrations (core) and public.vidra_search_migrations (search). Either can go dirty without the other.

# 1. Find out where it stopped. Note the REPEATED word: `run <service> <args>`
#    REPLACES the service's command, so the subcommand must be restated.
./deploy/compose.sh run --rm migrate        migrate version
./deploy/compose.sh run --rm search-migrate migrate version

# 2. Work out what that migration did before it failed and undo the partial
#    effect BY HAND. golang-migrate does not roll back for you.
./deploy/compose.sh exec postgres psql -U vidra -d vidra

# 3. Point the ledger at the last CLEAN version, N-1. `force` stamps the version
#    and clears dirty WITHOUT running any SQL — it is an assertion about the
#    schema you just repaired. --yes-i-know is checked before the database is
#    touched, so a refusal means nothing happened.
./deploy/compose.sh run --rm migrate        migrate force 41 --yes-i-know

# 4. Re-run the deploy, not the bare migrator. It takes a fresh dump, runs both
#    migrators as separate gated steps, and asserts the ledger is clean and at
#    the expected version before it starts anything.
./deploy/deploy.sh

Never point the ledger at N — that claims the broken migration succeeded, and the next deploy builds on a schema that does not exist. force -1 is the right target when it was the first migration that died.

A hung migration is not a crashed one

None of the up-migrations use CREATE INDEX CONCURRENTLY and none sets lock_timeout, so a data-dependent migration against a populated table stalls rather than fails, with api boot gated behind it. Look for a blocking lock in pg_stat_activity and pg_locks before assuming it died.

Schema compatibility is enforced, not just documented

Release N−1's code must run against release N's schema. That is what makes a rollback a 60-second tag flip that never touches the database, and it is why a true schema rollback is "restore the pre-deploy dump" rather than "migrate down".

make ci in both Go repos runs scripts/migrate-lint.sh, which rejects destructive forward DDL — DROP TABLE/COLUMN, RENAME, TRUNCATE, SET NOT NULL, ALTER … TYPE, DELETE FROM — in any *.up.sql. Down migrations are exempt, since they are the rollback path, and -- migrate-lint:allow is the escape hatch for an accepted break. On top of that, vidra-core's schema-compat workflow applies HEAD's migrations to a fresh database and runs the previous release tag's integration suite against it.

Content-Security-Policy

The frontend enforces a Content-Security-Policy on every response, built per request in proxy.ts (lib/security-headers.ts in vidra-user). In brief:

• Scripts run only with the per-request nonce, plus 'self' and the API origin for older browsers. object-src is 'none', and base-uri and form-action are 'self'.
• Stylesheets come from this site and the API origin; inline style attributes are allowed.
• Fonts come from this site or data: URIs only.
• Images, media and network connections (connect-src) may still reach any http: or https: origin, so a third-party image or API call is not affected. Anything the policy does not name, such as an iframe, falls back to default-src 'self'.

Custom JavaScript and CSS

The admin-supplied custom.js and custom.css are served by the API (/api/v1/instance/custom.js and /api/v1/instance/custom.css), and the policy allows them from this site and the API origin. What they pull in from anywhere else is blocked: a webfont from a third-party host, an @import of a third-party stylesheet, or a <script> tag pointing at an outside host. Scripts are trusted by nonce rather than by host list ('strict-dynamic'), so a script that custom.js itself creates at runtime inherits its trust in current browsers. Do not rely on that: host fonts and stylesheets on this site or the API origin.

Finding violations

Each violation is logged by the frontend as a csp violation line carrying the violated directive, the blocked origin (an origin only, never a full URL) and the document path (no query string):

vidra logs frontend | grep 'csp violation'

The endpoint that receives browser reports (/csp-report) is unauthenticated, so it caps each report at 8 KiB and each process at 100 log lines a minute. A flood of violations is therefore truncated, not complete.

If the policy breaks your custom code

Set CSP_REPORT_ONLY=true in env/production.env, then run vidra deploy. No rebuild is needed. Do not use vidra restart: it is docker compose restart, which keeps the environment the container was created with, so the change would not apply. deploy.sh ends with up -d --no-build, which recreates any service whose configuration changed, and docker-compose.yml passes CSP_REPORT_ONLY through to the frontend.

The same policy is then sent as Content-Security-Policy-Report-Only, so nothing is blocked and violations are still logged. Fix the custom code, then remove the line from env/production.env and run vidra deploy again to enforce the policy once more.

Framing

Pages under /embed are meant to be embedded and are frameable by any site. Every other page sends frame-ancestors 'self', so other sites cannot frame the rest of the app.

Email

• DigitalOcean blocks outbound port 25 on new accounts. Use a relay (Postmark, SES, Mailgun, Resend, Fastmail) on port 587 with STARTTLS and AUTH, which is what SMTP_PORT=587 already defaults to.
• SMTP_FROM must be a domain you control, with SPF and DKIM published for the relay, plus DMARC once those pass. Without them, password-reset mail lands in spam and users conclude the site is broken.
• MAIL_ENABLED=true requires SMTP_HOST and SMTP_FROM; the api refuses to boot in production otherwise.

Never set DEV_MAIL_CAPTURE_ENABLED=true in production

It exposes GET /api/v1/dev/email-token, which returns a live password-reset token for any address. It is gated in config validation, gated at route registration, and 404'd at the edge. Do not go looking for a fourth way around it. The same goes for HTTP_IMPORT_ALLOW_PRIVATE_URLS=true: both are set in env/qa.env.example on purpose and must never be copied into a production env file.

Staging is production config with throwaway data

Promote by deploying the exact image tags staging validated: deploy.sh and rollback.sh both work from VIDRA_CORE_TAG, VIDRA_USER_TAG and VIDRA_SEARCH_TAG in the env file, so promotion is copying three lines.

The same image serves every environment. The frontend container reads PUBLIC_API_BASE_URL and serves it to the browser as /runtime-config.js, so the origin is a restart, not a rebuild.

All the compose-based Make targets honour PROD_ENV_FILE=env/staging.env, and every deploy/*.sh honours ENV_FILE=env/staging.env.