Backup & restore¶

Almost all of an Etminan deployment is re-installable from the package and re-derivable by re-enrolling. Exactly one thing is not: the verifier trust store. This chapter is the disaster-recovery plan for that host.

Back up by blast radius, not by host count¶

Data	Where	If lost	Priority
Verifier trust store	verifier host — `baseline.db` (SQLite/WAL, which now also holds the UID→identity registry), its `.audit-head` anchor, `state.json`, `tls/`, `/etc/etminan-verifier`	Catastrophic — baselines, audit history, the identity registry, and every pinned host fingerprint are gone; re-enroll every host and rebuild every baseline.	Must back up
Daemon signing key	verifier host — `/var/lib/etminan-verifier/daemon-signing.key` (`0600`). The one key `etminan-verifierd` signs operator actions with. Operators hold no keys.	Past daemon signatures cannot be reproduced. It lives on the verifier by design, so back it up with the trust store, not separately.	Must back up (with the DB)
Agent state	each monitored host — `tls/`, `assigned_profile`, `/etc/etminan-agent`	Re-derivable: re-enroll or `rotate-tls` from the verifier.	Optional / nice
TPM AK/EK key material	inside each host's TPM	Not backup-able by design. A TPM clear forces re-enrollment of that host regardless.	Cannot back up

baseline.db is SQLite in WAL mode — never copy the live file

A naive cp of a live baseline.db yields a stale or corrupt restore, and its external audit-chain anchor can end up ahead of the copied DB — which the verifier reports as tampering. Always back up a consistent snapshot whose anchor is derived from the same generation.

The three responsibilities¶

Whatever backup tool you use — Bacula, restic, Borg, Veeam, ZFS snapshots — a correct Etminan DR backup must do all three of these. Port the intent, not any particular syntax:

Snapshot the WAL database before reading it. Run a pre-backup hook that produces a consistent copy of baseline.db and a matching audit anchor, then back up the snapshot. Exclude the live baseline.db, -wal, -shm, and live .audit-head from the backup set.
Capture the whole verifier trust-store set together. The snapshot + its .audit-head, state.json, tls/, the daemon-signing.key, and /etc/etminan-verifier (plaintext integration secrets and the plugin allowlist pins — the conf pins and the pinned binaries must restore together) form one consistent set. The identity registry needs no separate job: it lives inside baseline.db and is already captured by the DB snapshot.
Include the daemon signing key with the trust store. /var/lib/etminan-verifier/daemon-signing.key is the one key etminan-verifierd signs operator actions with; losing it means past daemon signatures can no longer be reproduced. It lives on the verifier by design (the daemon, not any operator, holds it), so it belongs in the trust-store backup — there is no off-host operator-key job to keep separate anymore, because operators hold no keys.

The WAL-safe snapshot pre-job¶

The load-bearing piece is a pre-backup hook that produces a point-in-time copy and derives the anchor from that copy, so the DB and its anchor are the same generation by construction — zero race window:

etminan-verifier-snapshot.sh (run before the backup reads the fileset)

#!/usr/bin/env bash
set -euo pipefail

DB="${ETMINAN_BASELINE_DB:-/var/lib/etminan-verifier/baseline.db}"
SNAP_DIR="${ETMINAN_BACKUP_DIR:-/var/backups/etminan}"
install -d -m 0700 "$SNAP_DIR"
[ -f "$DB" ] || { echo "no baseline.db at $DB — nothing to snapshot" >&2; exit 0; }

SNAP="$SNAP_DIR/baseline.db.snapshot"

# 1) Consistent point-in-time copy of the whole DB (commits the WAL internally).
sqlite3 "$DB" ".backup '$SNAP'"

# 2) Derive the audit anchor FROM the snapshot, so DB and anchor match exactly.
head_row="$(sqlite3 "$SNAP" \
  "SELECT seq || ' ' || entry_hash FROM audit_log ORDER BY seq DESC LIMIT 1;")"
if [ -n "$head_row" ]; then
  printf '%s\n' "$head_row" > "$SNAP.audit-head"
else
  rm -f "$SNAP.audit-head"   # empty audit_log (fresh verifier): no anchor yet
fi
chmod 0600 "$SNAP" "$SNAP.audit-head" 2>/dev/null || true

Why derive the anchor from the snapshot

verify_chain fails — and reports it as tampering — if the DB is behind its anchor. If you copied the live DB and the live anchor at two different instants, the anchor could be ahead of the DB snapshot, so the restore would look tampered. Deriving the anchor from the snapshot itself makes them the same generation.

state.json, the tls/ directory, daemon-signing.key, and /etc/etminan-verifier are written atomically (temp + fsync + rename) or are static key/config material, so a live copy of those is always consistent — no pre-job needed for them.

What to capture, concretely¶

Back up:

/var/backups/etminan/baseline.db.snapshot and .snapshot.audit-head (from the pre-job — the snapshot also carries the UID→identity registry)
/var/lib/etminan-verifier/daemon-signing.key (the daemon's one signing key, 0600 — custody-critical)
/var/lib/etminan-verifier/state.json
/var/lib/etminan-verifier/tls/
/etc/etminan-verifier/ (config, verifier.env, plugin allowlist pins and the pinned plugin binaries)
optionally /var/lib/etminan-verifier/identity.key if you want to preserve the integrity-summary signing identity

Exclude the live baseline.db, baseline.db-wal, baseline.db-shm, and live baseline.db.audit-head — the snapshot is the consistent copy.

There is no separate off-host operator-key job anymore: operators hold no key files, and the identity registry is inside baseline.db.

The Bacula example set¶

Etminan provides three heavily-commented Bacula example files you copy onto your own backup infrastructure and adapt. They are examples — not installed by the package and not part of the deliverable:

etminan-verifier-snapshot.sh — the WAL-safe snapshot pre-job (shown in full above); wire it into the verifier Job as Client Run Before Job.
etminan-filesets.conf — the FileSets: Etminan-Verifier-FileSet (the trust store — the only catastrophic-loss host, and the home of the daemon daemon-signing.key too) and Etminan-Agent-FileSet (one per monitored host, mostly re-derivable). There is no separate operator-keys FileSet: operators hold no keys and the identity registry rides along inside baseline.db.
etminan-jobs.conf — Schedules, separate pools per blast radius, JobDefs (the snapshot wired in as Client Run Before Job), Backup/Restore Jobs, and the Console-ACL roles — standard-operator (agent pool only) and dr-operator (verifier + agent — the whole trust store, which includes the daemon signing key). There is no off-host operator-key pool to fence off, because the one signing key lives on the verifier by design.

Copy both .conf files onto your Bacula Director (FileSets first), adapt the <-- site resource names and paths to your environment, and place the snapshot script on the verifier host.

etminan-filesets.conf — FileSets (what to back up)

# Bacula FileSets for Etminan disaster recovery.
#
# EXAMPLE / REFERENCE ONLY — this is not installed by the etminan package and is
# not part of the deliverable. Copy it onto your Bacula Director and adapt the
# paths and the "<-- site resource" names to your own deployment.
#
# A generic backup config for an Etminan deployment — Etminan has exactly two
# installable parts, so there are two installation FileSets:
#   * Etminan-Verifier-FileSet — the verifier install (the trust store; the only
#                                catastrophic-loss host). Includes the RBAC
#                                daemon's signing key — see the note below.
#   * Etminan-Agent-FileSet    — an agent install, per monitored host; mostly
#                                re-derivable. Cloned per host in the Jobs file.
#
# NO separate operator-keys FileSet. In the daemon (op) model, operators hold NO
# signing key at all: the etminan-verifierd daemon holds a single signing key on
# the verifier host and signs each authorised action on the operator's behalf,
# and operator identities are UID->role/scope rows inside baseline.db. So the
# only signing key to back up lives on the verifier and is captured by the
# Verifier FileSet below — there is no off-host key store to protect separately.
#
# NOTE: nothing here is created by any install script — an Etminan deployment is
# installed from the package; the operator hand-creates the real config, TLS
# identity, and (on the verifier) the DB/state. That operator-created state is
# exactly what these FileSets capture, since the package ships only *.example
# templates and empty directories.
#
# IMPORTANT — pair the verifier FileSet with the snapshot pre-job. baseline.db
# is SQLite/WAL; this FileSet backs up the *snapshot* produced by
# etminan-verifier-snapshot.sh, never the live DB. The Jobs, Schedules, Pools,
# and the ClientRunBeforeJob wiring live in the companion etminan-jobs.conf.
# Include both from your Director config, FileSets first:
#
#   @/etc/bacula/etminan/etminan-filesets.conf   # example path — put them
#   @/etc/bacula/etminan/etminan-jobs.conf        # wherever your Director reads

FileSet {
  Name = "Etminan-Verifier-FileSet"
  Include {
    Options {
      signature   = SHA256
      compression = GZIP9
      # Never back up the live SQLite DB or its sidecars — the snapshot below
      # is the consistent copy. Excluding these avoids a corrupt/stale restore.
      WildFile = "/var/lib/etminan-verifier/baseline.db"
      WildFile = "/var/lib/etminan-verifier/baseline.db-wal"
      WildFile = "/var/lib/etminan-verifier/baseline.db-shm"
      WildFile = "/var/lib/etminan-verifier/baseline.db.audit-head"
      Exclude  = yes
    }

    Options {
      signature   = SHA256
      compression = GZIP9
    }

    # --- The trust store (CATASTROPHIC if lost) ---------------------------
    # WAL-safe SQLite snapshot + its consistent audit-chain anchor, produced by
    # ClientRunBeforeJob = etminan-verifier-snapshot.sh. Restore: copy
    # baseline.db.snapshot -> baseline.db and baseline.db.snapshot.audit-head ->
    # baseline.db.audit-head (see the exclude block above — never the live DB).
    # The snapshot also carries the operator identity registry (UID->role/scope)
    # and all signed baseline batches, since both live inside baseline.db.
    File = /var/backups/etminan/baseline.db.snapshot
    File = /var/backups/etminan/baseline.db.snapshot.audit-head

    # The RBAC daemon's signing key (CROWN JEWEL). etminan-verifierd holds this
    # single Ed25519 seed and signs every authorised operator action with it;
    # operators hold no key of their own. It lives on the verifier host by
    # design, so it is part of this trust store — restore it together with the
    # DB snapshot so signed batches keep verifying. 0600, owned by the daemon
    # user: preserve those perms/owner on restore (see the Console ACL note in
    # etminan-jobs.conf — only the DR role can restore this pool).
    File = /var/lib/etminan-verifier/daemon-signing.key

    # Per-host enrollment + replay state: pinned AK/EK/TLS fingerprints, PCR10
    # cursor, signed enrollment/rotation accountability. Written atomically
    # (temp+fsync+rename) so a live copy is always consistent — safe as-is.
    # Losing it forces re-enrollment of every host from scratch.
    File = /var/lib/etminan-verifier/state.json

    # This verifier's mTLS identity. Its fingerprint is pinned into every agent
    # (ETMINAN_VERIFIER_CERT_FINGERPRINT). Losing it forces keygen-tls + re-pin
    # on every agent by hand.
    File = /var/lib/etminan-verifier/tls

    # --- Config + plugin allowlists (recoverable, but back up) ------------
    # verifier.env holds plaintext integration secrets (RT/FreeITSM/PagerDuty/
    # Slack/webhook). The two *-plugins.conf files carry the SHA256 pins that
    # authorize plugin execution; the *-plugins/ dirs hold the pinned binaries.
    # The conf pins and the binaries must restore together.
    File = /etc/etminan-verifier

    # --- Convenience: fast-restore binary (re-installable from package) ---
    WildFile = "/usr/bin/etminan-verifier"
    File = /usr/bin

    # --- NOT captured here (out of band): -------------------------------------
    #   * Plugin-catalog signing key (~/.etminan-catalog-signing.key) — offline,
    #     never on a verifier host (HANDBOOK §5). Only exists if you self-sign
    #     your own private plugin catalog; most deployments consume the vendor
    #     catalog, whose private half never touches a customer host at all.
    #   * The host TPM is the irreplaceable factor for AGENTS: a TPM clear/reset
    #     invalidates the AK/EK pinned in state.json and forces re-enroll. TPM
    #     key material cannot be backed up.
  }
}

FileSet {
  Name = "Etminan-Agent-FileSet"
  Include {
    Options {
      signature   = SHA256
      compression = GZIP9
    }
    # Agent mTLS identity — fingerprint is pinned in the verifier's state.json.
    # Loss requires re-enroll or op rotate-tls (verifier-side action), not fatal.
    File = /var/lib/etminan-agent/tls

    # Last profile pushed by op assign-profile; re-derivable (authoritative record
    # is in the verifier's state.json + audit_log), included for a clean restore.
    File = /var/lib/etminan-agent/assigned_profile

    # agent.env (holds ETMINAN_VERIFIER_CERT_FINGERPRINT, the pin) + profiles.conf
    File = /etc/etminan-agent

    WildFile = "/usr/bin/etminan-agent"
    File = /usr/bin
    # NOTE: TPM AK/EK are in-TPM only — nothing on disk to back up. A TPM clear
    # forces re-enrollment regardless of this backup.
  }
}

etminan-jobs.conf — Schedules, Pools, Jobs, Console ACLs (when/how)

# Bacula Jobs for Etminan disaster recovery.
#
# EXAMPLE / REFERENCE ONLY — not installed by the etminan package, not part of
# the deliverable. Copy onto your Bacula Director and adapt to your deployment.
#
# This is the companion to etminan-filesets.conf — that file defines *what* to
# back up (Etminan-Verifier-FileSet and Etminan-Agent-FileSet); this file
# defines *when* and *how* (Schedules, Pools, JobDefs, and the Backup/Restore
# Jobs that reference those FileSets). Without this file the FileSets are never
# exercised.
#
# Include both from your Director config, FileSets first:
#
#   @/etc/bacula/etminan/etminan-filesets.conf   # example path — put them
#   @/etc/bacula/etminan/etminan-jobs.conf        # wherever your Director reads
#
# The Director, Catalog, Messages, Client (FileDaemon), and Storage resources
# are site-specific and NOT defined here — supply them from your environment.
# The names referenced below and marked "<-- site resource" must match yours:
#
#   Client   = etminan-verifier-fd , etminan-agent-fd
#   Storage  = File
#   Messages = Standard
#   Catalog  = MyCatalog
#
# There is NO operator-workstation client. In the daemon (op) model, operators
# hold no signing key — the etminan-verifierd daemon holds the single signing
# key on the verifier host, so it is captured by the verifier backup, not a
# separate off-host job (see etminan-filesets.conf).
#
# See Bacula-Forgejo/docs/bacula-director.conf.example for a complete Director/
# Catalog/Storage/Messages scaffold to adapt.

# ── Schedules ─────────────────────────────────────────────────────────────────
# Weekly Full + daily Incremental. The verifier snapshot pre-job regenerates
# baseline.db.snapshot on every run, so Incrementals reliably capture DB changes.

Schedule {
  Name = "EtminanVerifierSchedule"
  Run = Full        1st sun at 01:00
  Run = Incremental mon-sat at 01:00
}

Schedule {
  Name = "EtminanAgentSchedule"
  # Agent state is small and mostly re-derivable — a nightly Full is cheap and
  # keeps restore dead-simple (no chain to replay).
  Run = Full daily at 02:00
}

# ── Pools ─────────────────────────────────────────────────────────────────────
# The verifier trust store is the only catastrophic-loss data AND contains
# plaintext secrets (verifier.env), this verifier's mTLS material, and the RBAC
# daemon's signing key. It gets its own restricted pool so restore access can be
# locked down via Console ACLs below — whoever can restore this pool can restore
# the daemon signing key, so keep that role tight.

Pool {
  Name             = EtminanVerifierPool
  Pool Type        = Backup
  Recycle          = yes
  AutoPrune        = yes
  Volume Retention = 365 days
  Maximum Volume Bytes = 20G
  Label Format     = "EtminanVerifierVol-"
}

Pool {
  Name             = EtminanAgentPool
  Pool Type        = Backup
  Recycle          = yes
  AutoPrune        = yes
  Volume Retention = 180 days
  Maximum Volume Bytes = 20G
  Label Format     = "EtminanAgentVol-"
}

# ── JobDefs ───────────────────────────────────────────────────────────────────

JobDefs {
  Name      = "EtminanVerifierDefs"
  Type      = Backup
  Level     = Incremental
  Client    = etminan-verifier-fd          # <-- site resource
  FileSet   = "Etminan-Verifier-FileSet"   # from etminan-filesets.conf
  Storage   = File                         # <-- site resource
  Messages  = Standard                     # <-- site resource
  Pool      = EtminanVerifierPool
  Schedule  = "EtminanVerifierSchedule"
  Priority  = 10
  # CRITICAL: take the WAL-safe snapshot BEFORE the FileSet is read, so the
  # backup captures baseline.db.snapshot (+ .audit-head) instead of the live
  # SQLite DB. The FileSet excludes the live DB; this produces the copy it wants.
  # Place etminan-verifier-snapshot.sh anywhere on the verifier host and point
  # this at it (example path shown — it runs on the client/FileDaemon).
  Client Run Before Job = "/usr/local/lib/etminan/etminan-verifier-snapshot.sh"
}

JobDefs {
  Name      = "EtminanAgentDefs"
  Type      = Backup
  Level     = Full
  Client    = etminan-agent-fd             # <-- site resource
  FileSet   = "Etminan-Agent-FileSet"      # from etminan-filesets.conf
  Storage   = File                         # <-- site resource
  Messages  = Standard                     # <-- site resource
  Pool      = EtminanAgentPool
  Schedule  = "EtminanAgentSchedule"
  Priority  = 15
  # No pre-job: agent state (tls/, assigned_profile, /etc/etminan-agent) is
  # written atomically (temp+fsync+rename), so a live copy is always consistent.
}

# ── Backup Jobs ───────────────────────────────────────────────────────────────

Job {
  Use  = "EtminanVerifierDefs"
  Name = "Etminan-Verifier"
}

# Define one agent Job per monitored host (each with its own Client). This is
# the template; clone it per host, overriding Name + Client.
Job {
  Use    = "EtminanAgentDefs"
  Name   = "Etminan-Agent"
  Client = etminan-agent-fd                # <-- site resource (one per host)
}

# ── Restore Jobs ──────────────────────────────────────────────────────────────
# NOTE on verifier restore: Bacula writes baseline.db.snapshot and
# baseline.db.snapshot.audit-head back to /var/backups/etminan. To bring the
# verifier live, copy them into place (see the Verifier FileSet's restore note):
#   cp /var/backups/etminan/baseline.db.snapshot \
#      /var/lib/etminan-verifier/baseline.db
#   cp /var/backups/etminan/baseline.db.snapshot.audit-head \
#      /var/lib/etminan-verifier/baseline.db.audit-head
# Never restore the live DB directly — the FileSet never captured it. The daemon
# signing key restores in place under /var/lib/etminan-verifier; re-assert its
# 0600 perms and daemon-user ownership, then restart etminan-verifierd.

Job {
  Name     = "Etminan-Verifier-Restore"
  Type     = Restore
  Client   = etminan-verifier-fd           # <-- site resource
  FileSet  = "Etminan-Verifier-FileSet"
  Storage  = File                          # <-- site resource
  Messages = Standard                      # <-- site resource
  Pool     = EtminanVerifierPool
  Where    = /                             # files carry absolute paths; restore in place
}

Job {
  Name     = "Etminan-Agent-Restore"
  Type     = Restore
  Client   = etminan-agent-fd              # <-- site resource
  FileSet  = "Etminan-Agent-FileSet"
  Storage  = File                          # <-- site resource
  Messages = Standard                      # <-- site resource
  Pool     = EtminanAgentPool
  Where    = /
}

# ── Console ACLs (restrict restores by blast radius) ──────────────────────────
# The verifier pool holds the trust store: plaintext secrets, mTLS material, and
# the RBAC daemon's signing key. Two roles keep day-to-day agent operators away
# from it — only the DR role can restore the verifier pool, and therefore the
# daemon signing key:
#   * standard-operator — agent pool only.
#   * dr-operator       — verifier + agent pools (the whole trust store,
#                         including the daemon signing key).

Console {
  Name       = etminan-standard-operator
  Password   = "standard-console-password"    # <-- CHANGE
  CommandACL = status, messages, show, run
  JobACL     = Etminan-Agent, Etminan-Agent-Restore
  PoolACL    = EtminanAgentPool
}

Console {
  Name       = etminan-dr-operator
  Password   = "dr-console-password"          # <-- CHANGE
  CommandACL = restore, run, status, messages, show, cancel
  JobACL     = Etminan-Verifier, Etminan-Verifier-Restore, Etminan-Agent, Etminan-Agent-Restore
  PoolACL    = EtminanVerifierPool, EtminanAgentPool
}

Restore¶

The whole restore is a copy of the trust store into place, then a health check — there is no re-import step.

Reinstall the package on the replacement verifier host (or reuse the existing one).
Restore the trust-store set. Your backup tool writes the snapshot files back (e.g. under /var/backups/etminan) alongside state.json, tls/, daemon-signing.key, and /etc/etminan-verifier. The identity registry comes back with baseline.db — it was inside the snapshot.

Bring the database live from the snapshot — never the live DB, which was never captured:

cp /var/backups/etminan/baseline.db.snapshot \
   /var/lib/etminan-verifier/baseline.db
cp /var/backups/etminan/baseline.db.snapshot.audit-head \
   /var/lib/etminan-verifier/baseline.db.audit-head
# (if the snapshot had an empty audit log, there is no .audit-head to copy)

Fix ownership/permissions so the verifier's unprivileged service user owns /var/lib/etminan-verifier, and keep baseline.db and daemon-signing.key non-world-readable (daemon-signing.key stays 0600).
Verify before trusting the box:
```
etminan-verifier op ping               # daemon up and answering
etminan-verifier op verify-signatures  # every signature + the full audit-log chain and its anchor
etminan-verifier op review             # enrolled hosts respond and pending items load
```
op verify-signatures re-checks every signature, the full audit-log hash chain, and its external anchor — the load-bearing confirmation that the restored DB and its .audit-head are the same generation. op ping confirms the daemon is up (and, being fail-closed, that trust-changing calls will be honoured); op review confirms enrolled hosts still respond. Watch journalctl -u etminan-verifier over the next cycle for any error.
Restart the timer/service and confirm the next run cycle succeeds and reaches every enrolled host.

Because state.json restored the pinned AK/EK/TLS fingerprints and PCR-10 cursors, the verifier resumes attesting the existing fleet with no re-enrollment.

What a restore cannot recover¶

The daemon signing key, if you did not capture it with the trust store — a lost daemon-signing.key means past daemon signatures cannot be reproduced (existing audited actions remain valid and verifiable; the daemon simply signs new actions with a fresh key once re-seeded). This is why it belongs in the trust-store backup, not a separate off-host job.
A host whose TPM was cleared. The AK/EK pinned in state.json no longer reside in that TPM, so it must be re-enrolled regardless of any backup — TPM key material cannot be backed up.