Backups & recovery

Issue #83 introduces one policy and generation format for both authoritative persistence modes. Install and approve the policy before updating a host: the pre-cycle gate now refuses an unconfigured backup instead of using implicit retention. Nothing in this PR installs timers or enables a host policy.

Policy and custody

Copy scripts/backup_policy.example.json to an absolute, owner-only file, assign the real custodian and paths, and set approved to true only after host review. The example values approved for this PR are:

Existing version-1 policies remain valid. The optional resume_published field defaults to false; only set it to true after the custodian approves scheduled continuation. With it enabled, a scheduled call continues a published generation only when every generation verifies, the newest is within the capture-age RPO, there are no staging/trash remnants, and the previous status.json is a valid ok receipt for the immediately preceding generation. It then uses the normal replication, capacity, retention, and completion path under the existing job lock. Any uncertainty remains a refusal. blocked_retry_seconds defaults to 3600 and accepts 60–604800 seconds; it limits repeated full content verification for an unchanged protected refusal. Its metadata fingerprint can only defer the same refusal: it never establishes integrity or authorizes capture, rotation, replication, or a healthy status. A metadata change or expiry triggers a fresh full verification. Manual finalize always bypasses this throttle and verifies the generations again.

The newest generation in each UTC epoch-aligned bucket is retained. Overlapping tiers share a generation. Always preserve the two newest valid generations. Rotation happens only after complete publication, fsync, checksum verification, and the local generation's replica attempt. Failed capture/verification never prunes prior generations. Before a new capture, generations outside the current retention tiers may be pruned, but the two newest valid generations are always preserved. Missed backups never trigger deletion of the last recovery points. This is a nominal eight-week recovery window with hourly and daily detail, conditional on successful jobs and sufficient capacity, not a promise that every historical hour is retained.

Budget for one complete new generation after retaining the required data. The controller may first remove generations outside the current retention tiers, but it refuses capture/publication when retained data plus the new generation still exceeds the budget; it does not sacrifice required generations to make space. Capacity failures, orphan staging directories, invalid generations, failed rotation, and an exceeded RPO are errors. Provision measured headroom before activation.

Only the custodian UID may own or access managed roots, generations, policy, and environment files (0700 directories, 0600 files). Declared forbidden live DB datadirs may belong to the database service UID. Parent directories must be root/custodian controlled; symlinks, hardlinked files, writable ancestors, and overlap with any live authority are rejected. Root-owned sticky temporary ancestors are allowed for disposable tests. The custodian is the trust boundary: another process running under that UID can alter its backups.

Set BACKUP_POLICY_FILE in the protected environment file. The launcher reads literal dotenv assignments only; it never evaluates shell commands, expansions, or substitutions. Quote values containing spaces. Explicitly configure all live authority roots, plus journal_roots.players and journal_roots.critical. A configured runtime journal environment path must agree with policy. An empty journal mapping is only appropriate when both pipelines are explicitly unused, such as a synthetic fixture.

MariaDB uses one full transactional dump, including schema, migration history, lifecycle tables, and all runtime tables. Nontransactional tables are rejected. Coordinate schema migrations/DDL with the backup window; a transactional data snapshot does not make concurrent DDL safe. Database capture checks runtime compatibility before and after the dump against the manifest frozen in the generation. Before an upgrade, set RUNTIME_COMPATIBILITY_MANIFEST to an absolute, owner-only copy of the deployed version's manifest when the checkout already requires a newer schema. Verify historical generations against their recorded manifest; do not relabel them with the checkout's current contract. Ownership validation requires the host UID identities of every path ancestor. A namespaced per-user systemd service can map root-owned ancestors to nobody; the guard correctly rejects that execution context. The root-managed production service template preserves the expected root mapping. Custom user services must qualify the backup under their actual service restrictions before cutover.

Flatfile capture preserves identity → critical authority → account locking, pending-transaction evidence, and the complete durable file tree.

Journal trees are copied before the authority snapshot and compared again after it. A changing/compacted WAL rejects the generation, preventing an old SQL snapshot from being paired with already-truncated replay evidence. Busy hosts must arrange a quiet snapshot window or a brief operator-controlled writer quiescence; repeated churn is an RPO incident, not a successful backup. Restores copy and replay these same journal directories. Historical generations are never edited.

Commands and scheduling

All commands below run as the custodian on a configured host. No command connects to a game session or promotes a candidate.

BACKUP_ENV_FILE=/etc/duris/backup.env scripts/backup_pfiles.sh
BACKUP_ENV_FILE=/etc/duris/backup.env scripts/backup_pfiles.sh status --require-drill
BACKUP_ENV_FILE=/etc/duris/backup.env scripts/backup_pfiles.sh schedule

The no-argument command remains the pre-cycle safety gate. A not-yet-created flatfile authority on its first boot reports authority_not_initialized and allows initial provisioning; scheduled jobs treat that state as an error, and no verified generation is claimed. The schedule command uses a separate persisted deadline, so pre-cycle runs do not defer independent snapshot. It is safe to invoke once per minute. All capture, rotation, restore, and status operations use an exclusive job lock with a bounded wait; a remaining busy condition is reported as a fixed error. The systemd backup, health, and drill units declare mutual conflicts, and the pre-cycle launcher retries a busy backup before refusing to boot.

The systemd backup timer's one-minute activity is not proof that a backup is due, completed, or healthy: schedule_seconds independently controls capture cadence, and failed runs do not advance schedule.json. The health command checks the verified generation age, capacity, completion/replication receipt, and (with --require-drill) drill qualification, but it does not inspect whether either timer is enabled or running. On the host, separately check systemctl is-enabled duris-backup-backup.timer duris-backup-health.timer, systemctl is-active duris-backup-backup.timer duris-backup-health.timer, and systemctl list-timers --all 'duris-backup-*'; also monitor each oneshot's exit status and alert path. The pre-cycle gate is separate again: inspect the actual cycle_mud.sh environment for SKIP_PREBOOT_BACKUP=1, which is printed explicitly when active. That opt-out does not disable the scheduled timer, and a scheduled timer does not compensate for the skipped pre-cycle check. Do not infer or change a host's opt-out setting from this repository policy.

Sample inactive systemd units are in deploy/systemd/duris-backup-*. Copy them, adapt User, WorkingDirectory, ReadWritePaths, paths, and permissions, and connect OnFailure to the custodian's existing alerting service before enabling the timers. The backup/health timers evaluate each minute; the drill timer evaluates hourly and runs only when the configured week has elapsed. Timer persistence catches missed invocations after host downtime. Test a deliberate invalid policy and verify that the failure reaches the responsible operator.

stdout/stderr contain JSON with fixed result/error codes, generation IDs, aggregate age/bytes/counts, and separate replica status. Recovery-point age starts before journal and authority capture, so dump duration cannot hide an RPO breach. No credentials, account names, hosts, or player values are telemetry. Alert on any nonzero job result, rpo_exceeded, capacity failures, interrupted work, or missing/overdue drill receipts. Monitor timer/unit availability too: a stopped scheduler cannot report its own failure.

A failed post-publication step leaves a complete generation and preserves prior ones. With resume_published=true, the next scheduled call may safely continue the narrow verified state above. Otherwise a mismatch is reported as published_generation_requires_finalize_command; this is protected refusal, not a healthy backup. Unchanged integrity/inspection refusals are re-reported without repeating full checksum reads until the configured retry window expires. Changed metadata invalidates that throttle and forces verification; corruption is never accepted from a cached result.

For manual recovery, first preserve and inspect the backup root's generation directories, status.json, and schedule.json. Compare the receipt's generation to the verified generation timestamps. Run finalize only when the newest generation is fully verifiable and there are no .staging-* or .trash-* remnants. If any remnant, checksum failure, uncertain replica, or capacity issue exists, stop and investigate; do not delete or rename evidence to clear the gate. Then run status (and status --require-drill for the health gate) and check the capture age. Finalization never changes the manifest's capture-start time and refuses a generation already outside its RPO; a successful receipt therefore cannot make an old capture appear fresh.

BACKUP_ENV_FILE=/etc/duris/backup.env scripts/backup_pfiles.sh status
BACKUP_ENV_FILE=/etc/duris/backup.env scripts/backup_pfiles.sh finalize
BACKUP_ENV_FILE=/etc/duris/backup.env scripts/backup_pfiles.sh status --require-drill

The manual command preserves every generation unless the normal, approved retention rotation applies; preserve any snapshots that rotation may remove before invoking it. It refuses with interrupted_job_requires_inspection when staging/trash remnants remain, and does not overwrite their evidence.

If a scheduled post-publication continuation returns replication_pending, replication is still retryable; the schedule deadline is not advanced. Keep the replica mount and capacity gates intact and let the next due invocation retry, or use the documented finalize command after inspection. Do not treat the local generation alone as a successful off-host backup.

Separate/off-host storage

replica_root optionally names an already mounted SSHFS destination with the same owner-only protections and a single custodian/writer. Other transports fail closed. Provision SSH host-key verification, a restricted remote account, protected keys, and mount permissions separately. Provision the remote storage with the same retention budget and sufficient publication headroom.

The controller copies an immutable generation, syncs it, renames it, re-reads every checksum, and applies the same retention policy remotely. Any remote failure is recorded in status.json as a retryable pending result; the verified local generation remains published, but local rotation waits until replication succeeds. transport_and_readback_verified means encrypted transport plus successful remote readback/fsync; it does not prove the remote operator's media replication, physical custody, or survival of host/storage loss. Verify those separately with an independent restore from remote media. A disconnected mount must fail; never replace it with an ordinary local folder.

Isolated restore and qualification

Install Python 3.11+, MariaDB 10.11 server/client tools, OpenSSL, iproute2, and util-linux. Build the native verifier and the matching server binaries:

make -C src
make -C src PERSISTENCE_BACKEND=flatfile DMS_BINARY="$PWD/bin/server/dms_restore_flatfile"
python3 scripts/build_restore_qualifier.py

The restore root must be a dedicated mounted filesystem, on a different device from every configured live authority and the backup root. Its entire filesystem size must not exceed max_bytes; maintain min_free_bytes there. A bounded tmpfs or a dedicated small recovery volume is suitable. A directory on the live filesystem is rejected, even if its pathname is different. This bounds compressed-dump expansion without risking the live filesystem. Mount provisioning belongs to the operator and is never performed by restore.

Supply independently current erasure evidence, held outside backups/candidates:

{"version":1,"captured_at":<UTC epoch seconds>,
 "policy_sha256":"<SHA256 of migrations/data_lifecycle_manifest.json>",
 "tombstones":[]}

The erasure custodian must establish this snapshot from the current authoritative ledger and approved policy, never from the selected historical generation. The ledger must be freshly captured (no more than five minutes old), and missing, stale, changed, or policy-mismatched evidence blocks restore. The current project has no durable source-wide erasure propagation adapter. Accordingly any nonempty tombstone ledger blocks the whole restore rather than risk resurrecting an erased identity or its indirect references. A historical tombstone row inside the database dump is not used as a substitute for that current external evidence. No erasure policy is enabled by the backup policy. Do not manufacture an empty ledger to pass.

BACKUP_ENV_FILE=/etc/duris/backup.env scripts/backup_pfiles.sh restore \
  --generation <generation-id> --tombstones /var/lib/duris/erasure-current.json
BACKUP_ENV_FILE=/etc/duris/backup.env scripts/backup_pfiles.sh drill \
  --tombstones /var/lib/duris/erasure-current.json

Omit generation to select the newest verified generation. Paths and existing databases cannot be supplied as restore targets: every candidate is newly created below the dedicated restore mount. The legacy flatfile wrapper now accepts a generation ID and current erasure evidence; its old arbitrary raw-copy target interface is intentionally closed.

Qualification requires the generation's exact runtime schema manifest. To restore an older schema, use its matching reviewed code/toolchain in a separate recovery workspace, then plan a separate migration on another candidate. Never bypass the compatibility check or edit backup metadata.

MariaDB restore initializes a new private datadir with networking disabled, imports through a schema-only account with no global/FILE privileges, validates the runtime schema, recomputes the complete migration history, and reconciles account/character, wallet, bank, and epic evidence. Runtime boot uses only the new socket. Flatfile restore verifies copied bytes before mutation, runs native authority replay, and validates existing account, snapshot, and world catalog bytes. Full player/domain loads run after WAL replay, allowing a durable first snapshot to materialize its missing projection. These loads reject lossy topology repair. Before boot, both journal types are scanned with the production codecs. Any corrupt/unsupported frame, quarantine evidence, or interrupted temporary journal blocks qualification. Both modes then boot the matching server against copied mini-world assets in a new user/network/PID namespace, exercise HTTP readiness, reject persistence startup failure messages, wait for both journals to drain, and require clean shutdown. Native postflight requires zero remaining records and no corruption/quarantine; authority reconciliation runs again after replay. Namespaces must be available; there is no fallback to a host-network boot. The server executable and qualification script are copied into the private candidate before entering the namespace, so recovery also works from a checkout under another user's private home. Include one server executable per retained candidate in the recovery filesystem's capacity budget.

QUALIFIED.json is written last and is evidence about this isolated candidate, not deployment approval or a full-world gameplay/balance certification. Manual candidates, including failed ones, remain private for diagnosis and consume the bounded recovery filesystem. Automated drills remove candidates, credentials, and detailed logs and keep only redacted drill.json evidence. Review the recorded generation/result and schedule at least weekly.

Incident selection, cutover, and rollback

1. Fence access and writes using the actual host supervisor; preserve incident logs and current authoritative state. Identify the last acceptable recovery point from verified generation timestamps, expected data loss, and erasure requirements. Record the selected ID and the custodian's decision.
2. Restore and qualify into isolation. Investigate all failures. Compare representative account, character, inventory, ledger and world state with independent incident evidence; a checksum alone proves neither correctness of the source nor the desired incident time.
3. Recheck the current erasure ledger immediately before any cutover. Prepare an explicit promotion plan covering the selected authority, journal paths, matching binary/schema, caches, listeners, and old-state rollback location. Stop writers, preserve a fresh full backup, and obtain host-owner cutover approval. These tools deliberately provide no production promotion command.
4. Publish one complete authority/candidate, rebuild caches only from that authority, perform load/value checks, then reopen access under observation. If validation fails, stop again and return the whole previous authority, journal set and matching binary/configuration; never mix generations.

Exceptional selective extraction is a private forensic operation on another qualified clone. Review identity/ownership/value dependencies and record any approved reconciliation separately. Do not inject an isolated character file or account row into production, create per-account archives, or treat selective recovery as a gameplay feature.

Verification

python3 tests/async/test_persistence_backup.py
python3 tests/async/test_backup_pfiles.py
python3 tests/async/test_flatfile_backup_manifest.py
python3 tests/async/test_flatfile_launcher.py
DURIS_RUN_BACKUP_INTEGRATION=1 python3 tests/async/test_persistence_backup_integration.py

The integration test uses synthetic identities, private MariaDB daemons, a bounded temporary restore mount, and isolated service namespaces. It requires mount/unshare privileges; run on a disposable Linux host/container (for Docker, CAP_SYS_ADMIN and an appropriate seccomp profile). No production environment, credentials, existing database, or live game connection is used.