Notification channels¶

How a finding actually reaches a human: the shipped PagerDuty, Slack, and generic-webhook channels, the JSON contract every channel receives, per-channel formatting and layout, and how to add a channel of your own with no Rust code.

etminan-verifier already prints every finding from a run/check cycle and, if ETMINAN_NOTIFY_EMAIL is set, emails it. Notification channels layer additional destinations on top of that — PagerDuty, Slack, and a generic webhook ship as reference plugins, and any other target (Microsoft Teams, SMS, an internal on-call system) is a shell script away, never a code change.

The one design fact that explains everything else

The verifier has no in-process HTTP client — ureq was removed entirely in 0.5.0 to shed that dependency's own trust and timeout surface. Every channel beyond email is therefore an external, allowlisted, SHA-256-pinned script running under the shared Etminan Plugin API — the same verify-then-exec model that change-source correlation uses, and for the same reason. Email is the one exception: it is a local sendmail call in alarm.rs, not a remote API, and predates this feature.

What can be a channel¶

Channel	Shipped script	Needs	Behaviour
PagerDuty	`deploy/notify-plugins/pagerduty.sh`	`curl`, `jq`, `ETMINAN_PAGERDUTY_ROUTING_KEY`	One Events API v2 event per finding; `dedup_key = attest-<host_id>` folds repeat findings for a host into one incident
Slack	`deploy/notify-plugins/slack.sh`	`curl`, `jq`, `ETMINAN_SLACK_WEBHOOK_URL`	One digest message for the whole batch, one line per finding
Generic webhook	`deploy/notify-plugins/webhook.sh`	`curl`, `ETMINAN_WEBHOOK_URL`, optional `ETMINAN_WEBHOOK_AUTH_HEADER`	POSTs the findings array through unmodified — the template to copy for a custom integration
Email	(not a plugin)	system `sendmail`, `ETMINAN_NOTIFY_EMAIL`	Local `sendmail -t`, bounded to a 20 s timeout so a dead relay can never wedge the alarm path

Every channel is independently enabled and independently filterable by severity, so — for example — critical findings can page PagerDuty while warning-tier SLA breaches only reach Slack.

Prerequisites¶

The shipped reference scripts need curl; pagerduty.sh and slack.sh also need jq.
Whatever credential or URL the channel requires:
- PagerDuty — an Events API v2 integration routing key (Service → Integrations → Add Integration → Events API v2 in PagerDuty).
- Slack — an incoming webhook URL (Slack app config → Incoming Webhooks).
- Generic webhook — any HTTPS endpoint that accepts a JSON POST, plus an optional auth header.

Configuring a channel¶

Four steps, all outside the verifier binary. Nothing here requires a rebuild or a restart — etminan-verifier is invoked fresh per check/run, so the next scheduled cycle picks up config changes immediately.

1. Install the plugin script, root-owned¶

Install the script somewhere root-owned and not group- or world-writable:

sudo install -D -m 0755 -o root -g root \
  deploy/notify-plugins/pagerduty.sh \
  /etc/etminan-verifier/notify-plugins/pagerduty.sh

Repeat for slack.sh / webhook.sh — install only the channels you are actually enabling.

2. Allowlist it, with its exact content hash¶

Copy deploy/notify-plugins.conf.example to /etc/etminan-verifier/notify-plugins.conf and add one name path sha256 line per channel:

pagerduty  /etc/etminan-verifier/notify-plugins/pagerduty.sh  <sha256sum output>
slack      /etc/etminan-verifier/notify-plugins/slack.sh      <sha256sum output>

sha256sum /etc/etminan-verifier/notify-plugins/pagerduty.sh

Recompute the hash every time you touch a plugin file

A stale or wrong hash means the verifier refuses to run the plugin at all — a clear error, never a silent bypass. The pin is re-verified fresh before every single execution, not once at load. Override the allowlist file's location with ETMINAN_NOTIFY_PLUGINS_CONF if you need it elsewhere.

3. Enable, filter, and (optionally) reformat¶

These live in /etc/etminan-verifier/verifier.env (see deploy/verifier.env.example for the shipped, commented template). They are read by the plugin, not the verifier binary:

ETMINAN_NOTIFY_CHANNELS=pagerduty,slack

ETMINAN_PAGERDUTY_ROUTING_KEY=...
ETMINAN_SLACK_WEBHOOK_URL=https://hooks.slack.com/services/...

# Optional: page only on critical findings, let Slack see everything.
ETMINAN_NOTIFY_PAGERDUTY_SEVERITIES=critical

# Optional: reshape this channel's per-finding line (see "What a plugin
# receives" below). Unset falls back to "[{kind}] {host_id}: {text}".
ETMINAN_NOTIFY_SLACK_FORMAT="{severity} on {host_id}: {text} (type: {kind})"

ETMINAN_NOTIFY_CHANNELS is the master switch: unset or empty, this feature never executes any plugin at all. Every other setting only matters once a channel's name is both listed here and allowlisted in step 2.

A mistyped severity filter can't silently mute a channel

ETMINAN_NOTIFY_<CHANNEL>_SEVERITIES is validated against the real severity set (critical, warning). An unknown value (e.g. critcal) or an effectively-empty one is caught by plugins verify and by every scheduled run, and surfaced as a plugin-failure finding — rather than quietly filtering out every finding so the channel never fires. There is also a cross-channel safety net: if every configured channel's _SEVERITIES together exclude critical, verification reports a (severity routing) failure, because a critical must always reach at least one notify channel.

4. Verify before trusting it¶

etminan-verifier plugins verify

This confirms every channel in ETMINAN_NOTIFY_CHANNELS is allowlisted, root-owned, and its pinned hash matches — do it right after setup, don't wait for a real finding to discover a typo'd hash or a missing routing key. The same check runs automatically at the start of every run cycle; a failure there becomes an ordinary warning-severity finding, so a broken notify channel can never fail silently.

plugins install — a catalog path that is shipped but not yet live

etminan-verifier plugins install <name> is designed to do steps 1 and 2 at once: it verifies a catalog manifest's Ed25519 signature against a compiled-in trust anchor before trusting anything it lists, then downloads, hashes, and allowlists the script. The command and its verification exist today, but nothing publishes a real signed catalog at its default URL yet, so the manual four-step path above remains the only route in practice.

The alarm dispatch path¶

Findings from a run/check cycle fan out to email and every enabled channel independently. One channel's failure never blocks another, and never fails the run — the whole point of the alarm path is that a finding it was called to surface can't be suppressed by a broken destination.

flowchart TD
    A[run/check cycle produces findings] --> C[per-channel fan-out]
    C --> D{ETMINAN_NOTIFY_EMAIL set?}
    D -->|yes| E[sendmail -t, 20s timeout]
    C --> F[for each channel in<br/>ETMINAN_NOTIFY_CHANNELS]
    F --> G{allowlisted &<br/>hash matches?}
    G -->|no| H[log + skip]
    G -->|yes| I[filter to channel severities]
    I --> J[render 'formatted' per _FORMAT]
    J --> K[verified_exec: JSON array on stdin]
    K -->|exit 0| L[delivered — clears plugin-failure health]
    K -->|non-zero / error| M[log + record plugin-failure work item]
    C --> N[emit_siem — CEF/LEEF, Enterprise]

What a plugin receives¶

Every enabled, allowlisted channel gets the findings from that cycle — filtered to its own configured severities first — as one JSON array on stdin (never argv: finding text is free-form and possibly attacker-influenced, and shell-escaping it onto a command line is exactly the injection surface this avoids). Nothing on stdout is read; the exit code is the only signal.

[
  {
    "host_id": "web-01",
    "text": "quote signature failed to verify",
    "severity": "critical",
    "kind": "signature-invalid",
    "formatted": "[signature-invalid] web-01: quote signature failed to verify"
  },
  {
    "host_id": "web-02",
    "text": "3 pending item(s) aged past the 24h review SLA",
    "severity": "warning",
    "kind": "sla-exceeded",
    "formatted": "[sla-exceeded] web-02: 3 pending item(s) aged past the 24h review SLA"
  }
]

Five fields, all additive as the feature grew — a plugin reads only what it uses, and one written against just the first three keeps working forever:

Field	Meaning
`host_id`	The monitored host this finding is about, or the literal `verifier` for a finding about the verifier's own config (e.g. a broken plugin).
`text`	Free-form, human-readable prose describing what happened.
`severity`	`critical` (a genuine verification failure or operator rejection) or `warning` (escalated, but not the same tier — e.g. a review past its SLA).
`kind`	A stable, machine-matchable category — distinct from the prose `text` — so a plugin can route/format without ever parsing prose.
`formatted`	This channel's pre-rendered display line (see below). Use it instead of hand-assembling the fields yourself.

Every `kind` value¶

Each maps to a real code path in alarm.rs — nothing here is speculative:

not-enrolled · unreachable · ak-mismatch · signature-invalid · nonce-mismatch · pcr-mismatch · unexpected-pcr-selection · template-hash-mismatch · stale-attestation · pending-overflow · check-error · sla-exceeded · plugin-failure · audit-chain-invalid · operator-rejected · test

Enterprise edition

The Enterprise build adds one further kind, dual-control-pending, emitted when a four-eyes approval request is created so the whole approver pool is notified through these same channels.

Per-channel formatting¶

ETMINAN_NOTIFY_<CHANNEL>_FORMAT (channel name uppercased) overrides one channel's formatted field with a {kind} / {host_id} / {text} / {severity} template. Purely a config change — no plugin edit, no re-hashing. Unset falls back to [{kind}] {host_id}: {text}. Two channels can render the same finding differently, and the same knob exists for email as ETMINAN_NOTIFY_EMAIL_FORMAT. An unrecognized {placeholder} is left as literal text rather than dropping the alert.

Configuring the overall layout¶

_FORMAT reshapes one finding's line. To change the whole message around those lines — email's subject/greeting/footer, Slack's digest header, PagerDuty's payload shape, whether webhook wraps the array — every channel has an optional _TEMPLATE env var pointing at a plain file you write:

ETMINAN_NOTIFY_EMAIL_TEMPLATE=/etc/etminan-verifier/templates/email.txt
ETMINAN_SLACK_TEMPLATE=/etc/etminan-verifier/templates/slack.txt
ETMINAN_PAGERDUTY_TEMPLATE=/etc/etminan-verifier/templates/pagerduty.jq
ETMINAN_WEBHOOK_TEMPLATE=/etc/etminan-verifier/templates/webhook.jq

Unset — or unreadable, which prints a warning and falls back rather than failing the cycle — means the exact built-in layout. This is purely opt-in.

Email and Slack use plain-text templates with two placeholders: {count} (how many findings) and {{findings}} (replaced with the joined, already-formatted lines). Email's first line is Subject: ...; everything after is the body:

Subject: [ALERT] {count} etminan finding(s) need attention

Hi team,

The following {count} issue(s) were detected during the latest check cycle:

{{findings}}

Please review these in `baseline review`. Thanks,
The Etminan Verifier

If a template omits {{findings}}, the lines are appended at the end rather than silently dropped — a typo must never make findings vanish.

PagerDuty and webhook take a real jq filter, not text substitution. This is deliberate: finding text can come straight off an IMA log on a possibly-compromised host and may legally contain quotes or newlines, and naive substitution into a JSON template risks broken or injected JSON. jq's own object/string construction escapes correctly for any content. PagerDuty's filter sees . as one finding object plus $routing_key / $dedup_key as jq variables (a copyable starting point is below); webhook's filter sees . as the whole array:

{source: "etminan-verifier", count: length, findings: .}

PagerDuty jq template¶

Copy this, edit it, and point ETMINAN_PAGERDUTY_TEMPLATE at your copy (e.g. /etc/etminan-verifier/pagerduty-template.jq). It reproduces pagerduty.sh's built-in default exactly — change only what you want different:

# Example ETMINAN_PAGERDUTY_TEMPLATE — a jq FILTER, not a plain-text
# template. Copy this file, edit it, and point ETMINAN_PAGERDUTY_TEMPLATE
# at your copy (e.g. /etc/etminan-verifier/pagerduty-template.jq).
#
# `.` is one finding object: {host_id, text, severity, kind, formatted}.
# $routing_key and $dedup_key ("attest-<host_id>") are supplied as jq
# variables by pagerduty.sh — reference them as $routing_key/$dedup_key,
# not as fields of `.`.
#
# This is a real jq filter (not {placeholder} text substitution) on
# purpose: jq's own object/string construction escapes JSON correctly for
# any finding content, including a path pulled straight from an IMA log on
# a possibly-compromised host, which can legally contain quotes or
# newlines. Raw text substitution into a JSON template would risk broken
# or injected JSON on exactly that content.
#
# The filter below reproduces pagerduty.sh's own built-in default exactly
# — start here and change only what you actually want different (e.g. add
# a custom_details field, change what becomes the summary).

{
  routing_key: $routing_key,
  event_action: "trigger",
  dedup_key: $dedup_key,
  payload: {
    summary: (.formatted // .text),
    severity: .severity,
    source: .host_id,
    custom_details: {
      kind: .kind
    }
  }
}

Previewing before you trust it¶

etminan-verifier notify-preview
etminan-verifier notify-preview --channel pagerduty

notify-preview renders four representative fake findings (one signature-invalid, one pcr-mismatch, one sla-exceeded, one plugin-failure) through the current configuration — every _TEMPLATE / _FORMAT / _SEVERITIES you have set — and prints the full email (subject + body, if email is configured) plus the raw JSON payload each channel would receive on stdin.

Preview never sends anything

It never executes a plugin and never calls sendmail, so it is safe to run any time — including against a channel that isn't fully allowlisted yet. It previews the verifier's side of the contract only; whatever a Slack/PagerDuty/webhook _TEMPLATE does with that payload happens inside the plugin script, which preview deliberately never runs. Pipe the printed payload into the script by hand if you want to see its transformed output — that performs the real send.

Testing a channel for real¶

etminan-verifier notify-test --channel pagerduty

plugins verify confirms a channel is allowlisted, root-owned, and correctly hashed — an integrity check, not a functional one. A channel can pass it and still fail the first time it is needed: a wrong routing key, an expired webhook URL, an outbound firewall rule. notify-test closes that gap. It builds one synthetic finding (kind: "test", severity: "warning", host_id notify-test — distinct from any real host so it can't collide with a real PagerDuty dedup_key) and pipes it through the plugin's real invocation path — the exact verified_exec call a live run uses. Unlike notify-preview, it performs a real send and reports the plugin's actual exit status and any stderr. It deliberately ignores the channel's _SEVERITIES filter — you're asking to test this channel now, not to have the attempt skipped because a test finding's severity doesn't match.

Adding a new channel¶

No Rust code, no verifier rebuild — see the Plugin API for the full, category-agnostic walkthrough. The whole contract for this category:

#!/bin/sh
set -eu
: "${MY_CHANNEL_URL:?not set}"

findings=$(cat)                 # the JSON array, on stdin
text=$(echo "$findings" | jq -r 'map(.formatted) | join("\n")')

curl -fsS --max-time 20 -H "Content-Type: application/json" \
  -X POST "$MY_CHANNEL_URL" -d "$(jq -n --arg t "$text" '{text:$t}')"
                                # exit 0 = fired, non-zero = "skipped this cycle"

Then:

Install it to /etc/etminan-verifier/notify-plugins/, root-owned.
Allowlist it (name, path, sha256sum) in notify-plugins.conf, and add its name to ETMINAN_NOTIFY_CHANNELS.
Run plugins verify. Nothing else in the verifier changes — dispatch, severity filtering, and per-channel formatting are already channel-agnostic.

Plugin API — the shared verify-then-exec security model these channels run under.
Change-source correlation — the other plugin category, same mechanism.
SIEM output — structured CEF/LEEF for a SOC, alongside (not instead of) these channels (Enterprise).
Baseline review — where many findings originate.