Skip to content

Troubleshooting

Common failure modes and how to diagnose them. There is no single self-check subcommand — diagnose from the read-only commands that already exist plus the service journals.

Start here: quick health checks

A handful of read-only commands and the systemd journals cover almost every misconfiguration before you have to dig further. None of them changes trust state.

On the verifier:

etminan-verifier op ping               # is the etminan-verifierd daemon up and answering?
etminan-verifier op whoami             # which identity / role / scope does your UID map to?
etminan-verifier op verify-signatures  # re-verify every signed row + the audit-log hash chain
etminan-verifier op review             # list pending items (and confirm enrolled hosts respond)
journalctl -u etminan-verifier         # the run timer's output and any alarm-path errors
systemctl status etminan-verifier.timer etminan-verifierd

If op ping fails, the daemon is down and — because the model is default-deny / fail-closed — every trust-changing op call is currently refused; start there. op whoami is the first thing to run when a call is unexpectedly refused: it prints the identity, role, and scope the daemon maps your kernel UID to.

On a monitored host:

systemctl status etminan-agent etminan-agent-ima-policy.service
journalctl -u etminan-agent            # the agent logs a specific error and refuses to start on a bad config
ls -l /dev/tpmrm0                       # the resource-managed TPM device the agent must use
cat /sys/kernel/security/integrity/ima/runtime_measurements_count   # must be > 0

The agent makes no trust decision and ships no self-check subcommand; it validates its own preconditions at startup and logs a specific error (then refuses to start) when the TPM device, IMA log, capability grant, mTLS identity, or verifier pinning is wrong. journalctl -u etminan-agent is the authoritative place to see which.

Enrollment: AK fingerprint mismatch

Symptom: enroll refuses, or a later check reports the host's AK fingerprint no longer matches the enrollment record.

  • The AK template must never change. The AK is created via TPM2_CreatePrimary from a fixed template; a different template derives a different key and breaks the fingerprint match. This is by design — it is what lets enrollment pin a stable identity with no persistent key storage.
  • A genuinely changed AK (TPM owner-clear or firmware change re-derived it) is not an error to force past — use rotate-ak, which re-binds the AK within the same TPM using the EK as the continuity anchor. A different EK means a different TPM and is refused as a re-enroll, not a rotation.
  • Re-enrolling an already-enrolled host is refused unless you pass --force, so a fat-fingered re-run cannot silently re-pin a possibly-compromised host via trust-on-first-use. Only --force after re-confirming the new AK fingerprint out of band.

PcrDigestMismatch on check / run

Symptom: check or run reports a finding: the replayed PCR 10 does not equal the value the TPM signed.

This is the verifier's core check — replay the IMA log, recompute PCR 10, and confirm it equals the signed quote. A mismatch is a real finding by default, but a few benign or environmental causes are worth ruling out:

  • The enroll → first-check cursor gap. Immediately after enrollment (and an agent restart), the first check can falsely report PcrDigestMismatch: the verifier replays from its stored cumulative value while the agent sends the full log from offset 0. The verifier resets to 0 on an offset regression, so this self-corrects; if you hit it once right after enrolling, re-run check.
  • Log growth between quote and read. The IMA log keeps growing from background system activity in the small window between the quote being taken and the log being read. Etminan handles this with prefix-matching — a valid quote's digest must match some prefix of what was received; anything after belongs to the next cycle. If you see transient mismatches that clear on the next cycle, this is the mechanism working.
  • Wrong IMA policy / PCR bank. The SHA-256 PCR bank must be read from ascii_runtime_measurements_sha256, not the generic ascii_runtime_measurements (which uses legacy SHA1-sized template hashes). A persistent mismatch across cycles on one host points at a genuine drift or a tampered log — treat it as a finding and review it.

Host stuck "catching up"

Symptom: a host reports CatchingUp cycle after cycle and never returns Genuine.

A host with a genuine backlog (a large log delta) clears in a finite number of cycles. state.json tracks consecutive_catching_up and resets it to 0 on every Genuine cycle. A host still catching up past ETMINAN_MAX_CATCHING_UP_CYCLES is a wedged agent — or a compromised host feeding a fabricated log_truncated delta to suppress the mismatch alarm forever — and Etminan alarms rather than staying silent. Investigate the agent process and the IMA log on that host.

TPM device: /dev/tpmrm0

Symptom: the agent logs a TPM-open error at startup, or otherwise cannot open the TPM.

  • The agent runs as an unprivileged tss-group user and must use /dev/tpmrm0 (the resource-managed device), never /dev/tpm0 (the raw device, group-owned by root). Never rely on the library's DeviceConfig::default(), which points at /dev/tpm0. Running the packaged agent as its real systemd user surfaces this; running manual tests as root masks it.
  • /dev/tpmrm0 does not exist → the TPM is absent or the kernel resource manager is not enabled.
  • exists but cannot be opened → check the device is group tss, mode 0660, and the agent user is in the tss group.
  • A generic TSS TCTI environment variable set in the agent's environment points it at a lab / emulated TPM; a production host must not set one.

IMA log permission denied

Symptom: the agent logs that it cannot read the IMA measurement log.

The IMA measurement log is root:root 0440 by kernel design — no group membership grants an unprivileged user read access. The fix is the systemd unit granting AmbientCapabilities=CAP_DAC_READ_SEARCH (the narrowest capability that bypasses the read-permission check), not running the agent as root. A permission error reading the log means the unit is missing that grant. Run by hand as root it reads OK because root holds every capability — which is exactly why the packaged, unprivileged run is the honest test.

IMA policy did not load / measurement count is 0

Symptom: runtime_measurements_count reads 0 — IMA is not measuring.

  • IMA accepts exactly one successful policy write per boot (failed writes do not consume the slot). Write the complete policy in one shot, early at boot, via etminan-agent-ima-policy.service — not incrementally.
  • Once loaded, the policy write node can disappear from securityfs entirely (ENOENT), not just start rejecting writes with EACCES. write-ima-policy treats either failure mode as "already loaded" when runtime_measurements_count is already nonzero, so a second run is a no-op, not an error.

Agent won't start: verifier pinning

Symptom: the agent refuses to start with a verifier-pinning / TLS-mode error in its log.

  • Set ETMINAN_VERIFIER_CERT_FINGERPRINT to the verifier's fingerprint (exactly 32 bytes / 64 hex chars). A wrong length or non-hex value is rejected at startup.
  • The only alternative is ETMINAN_ALLOW_PERMISSIVE_TLS=1, which accepts a client certificate from any caller and is intended only for the one-time enrollment bootstrap. The agent logs a warning while it is set; pin the fingerprint and drop the flag once enrolled.

Certificate expiry

Symptom: a TLS handshake between agent and verifier fails, or the logs show a certificate has expired (or is close to it).

mTLS certificates do expire. Rotate a host's agent certificate by regenerating it on the agent (etminan-agent keygen-tls), then re-pinning it on the verifier with etminan-verifier op rotate-tls <host> --reason <why>. For the verifier's own certificate there is no rotate subcommand — run etminan-verifier keygen-tls, update ETMINAN_VERIFIER_CERT_FINGERPRINT in every agent's agent.env, then restart each agent — in that order (restarting an agent before its env is updated locks it out until fixed).

Audit-log chain broken

Symptom: run or op verify-signatures reports the audit-log chain is broken.

The error text names the exact failure — a seq gap (a row deleted or reordered), a prev_hash mismatch (chain broken), a stored-hash mismatch (a row edited after being written), a high-water above the surviving head (tail truncation), or a missing/altered externally-anchored head row (rollback / DB restore). See The audit log. On a verifier you administer, the most common innocent cause is an incorrect backup/restore — see Backup & restore for restoring a snapshot whose anchor matches the DB generation. A break you cannot explain from your own operations is a finding: treat the host as potentially compromised.

A finding won't leave the box

Symptom: you expect alerts but none arrive.

The alarm path is the only place a finding leaves the verifier, so test it directly rather than waiting for a real finding:

etminan-verifier notify-preview            # render templates against samples (sends nothing)
etminan-verifier notify-test --channel <name>   # send one synthetic finding through the REAL plugin

plugins verify confirms a channel is allowlisted, root-owned, and hash-matched but never executes it; notify-test closes that gap by actually attempting delivery and reporting the plugin's exit status. run also re-verifies every plugin each cycle, so a broken channel alarms on its own rather than being discovered only when needed.

See also