Xodiac — Feature Inventory

Two repos, one product:

1 · Monitoring checks

Platform-runner probes (poller daemon xodiac:poll, batch 25 / 4 workers):

Agent-runner checks — the same types executed inside the LAN by an enrolled device (private IPs are allowed there; platform targets go through the SafeTarget SSRF guard instead).

Per-check knobs: interval_s (≥30 s), timeout_s, confirm_fails (transient suppression — degraded state never pages), region, slo_target (0–99.999 %), enable/disable, per-check history.

2 · Metrics pipeline

• Agent host metrics: cpu_pct, mem_pct, mem_used_mb, disk_pct (+ per-mount disk_pct.<mp>), net_rx_bytes, net_tx_bytes, procs, uptime_s (Linux + Windows + BSD/Darwin variants).
• Raw samples → 5-minute buckets → 1-hour rollups (xodiac:rollup), per-metric latest/last_ts index; samples expire after sampleRetentionDays (14 d) — buckets carry history beyond that. Bucket uniqueness is enforced by generated COALESCE(device_id,0)/COALESCE(check_id,0) columns (migration 2026-11-14-000001) — plain nullable columns in a unique key never deduplicated under MySQL/MariaDB.
• Per-device graph explorer: pick any reported metric name, 1–168 h window.
• metrics_scrape plugin turns any URL into graphed metrics — Prometheus text format or flat/nested JSON; per-endpoint prefix and conf.d-only auth (token / user+pass).
• custom_metrics plugin — every executable/*.prom file in conf.d/metrics.d/ emits graphed metrics (name value / name=value / name: value / Prometheus text), filenames prefix the metric name. The "monitor anything" escape hatch: UPS charge, weather APIs, certificate counts — if a script can print a number, it's a graph.
• snmptrap plugin — inbound SNMPv1/v2c UDP trap receiver → per-OID snmptrap.<oid> recency gauges + listener entity (conf.d listen, recent_s).
• snmp plugin also emits per-interface traffic metrics — with walk: "interfaces" the agent reads ifTable ifHCInOctets/ifHCOutOctets (falling back to 32-bit counters) each cycle and reports link.<ifName>.bytes_in/bytes_out rates (bytes/sec, wrap/reset safe for 32- and 64-bit counters, in-memory deltas — no first-sample spike). Works for both v2c and v3 targets; graphable + alertable via device_metric rules like any built-in.
• ipmi-sensors plugin — ipmitool sensor readings → meta, degraded on non-ok sensors (local BMC or remote lanplus); also collects FRU inventory (part/product/serial/asset via fru field) into entity meta.
• custom_http plugin — declarative JSON→entity probe for APIs without a dedicated spec: params.url/path (dotted, e.g. data.items)/ name_field/id_field/state_field/parent_field/meta_fields extract virt entities from any JSON response (≤200/call). Auth only via conf.d/custom_http[.<cred_key>].json — a conf.d url pins the fetch target so tenant params can't redirect credentials; cred_key gives each authed endpoint its own creds file.

3 · Devices & the agent

Identity & transport

• Enrolment: one-time tokens (max_uses, expiry, per-tenant device cap, atomic claim); Ed25519 keypair generated on-device — platform pins the pubkey TOFU; every request signed (X-Agent-Sig over ts + sha256(body)).
• 25 s long-poll for config changes/actions + report cycle (default 30 s, poll_interval per device); early returns can never tight-loop.
• Offline spool (bounded, drop-oldest, check-results excluded — stale transitions would corrupt consec_fails).

Fleet ops

• Inventory: OS/packages/services refreshed daily → feeds plugin detection + CVE advisories.
• Remote actions (plan-gated actions, audited): service_restart, reboot (0–300 s delay), plugin_run (execute one configured plugin on demand — params always come from the device's platform config, never the payload) — strict payload allowlist, no arbitrary exec. Updates are not an action: they arrive via the signed poll update directive.
• Self-update: Ed25519-signed releases, per-device auto_update, update channels (stable/beta), version pinning, staged rollouts by device-id percentage; versioned binary store + atomic swap + service-manager restart.
• Device groups, per-device poll interval, suspend/delete.

Deploy surface (scripts/install-agent.{sh,ps1}, Dockerfile, ipk builder):

• Linux {amd64, arm64, armv7, 386} · Windows · FreeBSD {amd64, arm64} · OpenBSD/amd64 · NetBSD/amd64 · macOS {amd64, arm64}
• Init detection: systemd, OpenRC, rc.d (BSD), procd (OpenWrt), nohup fallback, Windows service, Docker
• OpenWrt .ipk package via scripts/build-ipk.sh
• /etc/xodiac-agent/conf.d/*.json = plugin credentials (local-only, never uploaded); SIGHUP = reload creds + re-pull config without restart
• Flags: -once (single cycle), -enrol-only, -test-plugins (run every conf.d plugin locally and dump entities/metrics — cred verification), -config/-server/-token/-state/-version, -key (32-byte hex/base64 sealing key via flag/XODIAC_CONFIG_KEY/<configdir>/config.key), -encrypt-config (seal agent.json in place — AES-256-GCM; sealed configs load with the same -key)

4 · Plugin platform (232 collectors)

Per-device enablement from the UI (Settings → device → Plugins); candidates auto-detected from the device's own service/package inventory; credentials live only on the host (conf.d/<name>.json — the platform can never redirect them: creds.url wins over params.url, auth attaches only to creds-origin URLs; a creds file with secrets but no url sends nothing to params.url-controlled targets — tightened 2026-11-07). Plugin enablement is plan-gated (plugins).

Hypervisors & containers proxmox (nodes/QEMU/LXC/storage via API) · esxi (govmomi) · libvirt (KVM domains via virsh) · incus/lxd · docker · podman (both emit per-container cpu_pct/mem_bytes/mem_pct/net_*_bytes metrics — graphable + alertable per container) · openwrt (ubus board + interfaces)

Databases & caches mysql/mariadb · postgres · redis/valkey/keydb/dragonfly · mongodb · cassandra · scylladb · mssql · couchdb · clickhouse · neo4j · cockroach · elasticsearch · opensearch · solr · arangodb · dgraph · surrealdb · tidb · hazelcast · couchbase · cratedb · druid · pinot · etcd · influxdb · rqlite · qdrant · weaviate · milvus · chromadb · meilisearch · typesense · memcached · zookeeper · beanstalkd

Web & app infra nginx/Apache status (auto-detect) · haproxy (CSV stats) · varnish · squid · caddy · traefik · envoy · kong · nginx-unit · tomcat · php-fpm · rabbitmq · nats · redpanda · nexus · artifactory · harbor · portainer · rancher · jenkins · drone · woodpecker · argocd · netbox · jira · confluence · bitbucket · bugzilla · gitea · gitlab · forgejo

Observability stack prometheus · grafana · alertmanager · loki · mimir · tempo · thanos · victoriametrics · victorialogs · influxdb · netdata · graylog · kibana · logstash · beats · sentry · posthog · plausible · crowdsec

DNS / network gear bind9 · unbound · powerdns · coredns · nsd · dnsmasq · blocky · pihole · adguard · technitium · opnsense/pfsense (REST) · mikrotik (RouterOS REST) · unifi · ddwrt · openwrt · snmp (sysDescr/uptime + custom OIDs + ifTable walk → per-interface link entities, label=oid syntax) · nut/apcupsd (UPS vars via upsc/apcaccess)

Security, mail & comms fail2ban · clamav · rspamd · postfix · mailpit · listmonk · ejabberd · synapse · jitsi-videobridge · rocketchat · mattermost · zulip · mastodon · peertube · pixelfed · discourse · nodebb · flarum · ntfy · gotify · step-ca · dex · hydra · kratos · authelia · authentik · keycloak · zitadel · vault · consul · nomad · infisical · vaultwarden

Homelab / media / IoT sonarr · radarr · lidarr · readarr · prowlarr · bazarr · sabnzbd · tautulli · overseerr/jellyseerr · qbittorrent · jellyfin · emby · plex · navidrome · audiobookshelf · kavita · photoprism · immich · miniflux · nextcloud · owncloud · seafile · syncthing · homeassistant · openhab · domoticz · frigate · zoneminder · octoprint · moonraker (Klipper) · duet (RepRap) · grocy · mealie · homebox · wikijs · joplin · readeck · wordpress · pmg (Proxmox Mail Gateway) · synology · pbs · fronius solar · evcc · go-e wallbox · tasmota · shelly · wled

System & hardware smartmon (SMART temp/hours/pass-fail) · nvme · mdadm · lvm · drbd · zfs · ipmi · sensors (lm-sensors) · nut-ups · apcups · pf (BSD) · speedtest (Ookla) · systemd-failed units · pending updates (apt/dnf/apk) · reboot-required · timesyncd · chrony · supervisor · pm2 · borg · k8s · k3s

VPN & overlay wireguard · openvpn · tailscale · zerotier · netbird · headscale · ipfs

Escape hatches

• script — drop any executable in conf.d/scripts/; JSON-array output → entities, k: v lines → metrics (dir is conf.d-local, never remote-settable)
• gameserver — A2S query (CS2/Rust/ARK/Palworld/Valheim/TF2/DayZ…), Minecraft SLP, FiveM /info.json

Every collector runs sandboxed: per-plugin recover(), 30 s timeout, failures logged-and-skipped — one bad target never blocks the report.

Writing a collector — docs/PLUGIN-SDK.md in xodiac-agent: the spec struct fields, the entity/metric contracts, the conf.d credentials boundary, and the registration checklist (agent run() + platform Plugins::KNOWN + FEATURES). Most JSON APIs need zero Go — custom_http params or a ~4-line specs row covers them.

5 · Alerting & incidents

Incident lifecycle — deduped per subject, open → ack → resolve; degraded never pages (confirm_fails); degraded→down upgrades severity; recovery resolves. Public ack links (/ack/{id}/{token}) for paging emails. Mute/silence — POST /incidents/{id}/silence {minutes} suppresses new incidents for that subject until expiry (any-status row carries silenced_until); 0 clears. UI: "mute" button per incident row.

Device-offline alerting — xodiac:sweep flips agents silent >5 min to offline and opens a device incident ("agent offline — last seen …"); the next poll/report auto-resolves it (detail marked "offline":true so a coexisting metric-rule incident is never cleared by a check-in).

Channels — email (tenant owners implicit when nothing configured) · signed webhook (HMAC) · Slack · Discord · Telegram · ntfy · Gotify · MS Teams · Mattermost · Rocket.Chat · Twilio SMS · PagerDuty · Opsgenie · VictorOps. Per-channel test-fire; delivery log per incident + replay of failed deliveries.

Alert rules — down/degraded triggers and metric thresholds (check latency_ms/value, device cpu_pct/mem_pct/disk_pct/any metric): {kind: metric|device_metric, metric, op: >,<,>=,<=,==,!=, threshold}. Per-rule channel routing (a rule's channels list wins — no more blasting every channel), severity, group scoping, digest-only rules (incidents open + appear in the daily digest but never page), anomaly rules (z-score vs rolling metric_samples baseline — catches drift, not just thresholds), escalation stages, on-call rotations.

Escalations — multi-stage {after_min → channel_ids | rotation_id} chains snapshotted onto the incident at open; xodiac:sweep fires due stages.

On-call rotations — member round-robin over a configurable period_s (min 1 h, default weekly); escalation stages page whoever is on duty by email.

Maintenance windows — suppress alerting during windows (shown on the public status page too). Scopes nest correctly: tenant covers everything, group covers member devices and their checks + virt entities, device covers the device plus its checks/entities, check covers just that check. Optional end-of-window digest: pick notify channels at creation; when the window ends, xodiac:sweep sends each a one-time summary ("N failed check runs across M checks") built from check_samples — exactly-once via digest_sent_at (migration 000006). Bulk assign: Devices page multi-select → "maintenance…" creates one device-scoped window per selected device (POST /maintenance/bulk, ≤100/call).

Daily digest — per-tenant email summary (xodiac:digest).

6 · Heartbeats (dead-man's switch)

Cron/job monitoring: GET/POST /ping/{token} endpoints, interval_s + grace_s; a missed interval opens an incident, the next ping resolves it. Subject-linked to incidents like checks.

7 · Ingest endpoints

POST /ingest/{token} — any external tool (Grafana, scripts, other monitors) pushes {title, detail, severity, dedup_key} → incident. dedup_key idempotency — a repeat won't reopen; auto-resolve on recovery keys.

8 · Public status pages

/status/{slug} — per-tenant slug (configurable in Settings): live check states, open incidents, scheduled maintenance, escaped-output safe. Machine-readable twin at /api/status/{slug}. Custom domain — set status_domain in Settings and point status.example.com at the host: the status page serves at / (Host-header match in Site::home) and JSON at /api/status-host; slug path keeps working as fallback.

9 · Virtualization & infra inventory

/api/v1/virt — unified entity view across all device plugins: VMs, containers, zpools, disks, links, services, clusters — with parent edges (parent_ext) for hypervisor → guest trees and per-entity meta/state. Backup collectors (restic/borg/kopia) report kind:"backup" entities — snapshot count + newest age (state=stale past params.max_age_h); opt-in params.stats=true adds repo_size_bytes/ repo_files meta (one extra CLI call per cycle — off by default since stats can be slow on large repos).

10 · Security advisories & blocklists

• xodiac:advise — scans device package inventories against OSV.dev + CISA KEV (nightly cron); hits surface per device with mute controls.
• RBL scanning — tenant IPs/domains vs DNSBL zones (xodiac:sweep queues, rbl check type + /blacklist/scan on demand).

11 · Accounts, RBAC & security

• Roles owner → admin → member → viewer + granular capabilities (Caps::effective); TenantScope enforces per-route caps (incl. index.php/ bypass guard, devices/{id}/action→actions, devices/{id}/plugins→plugins sub-path rules); viewers are read-only.
• TOTP 2FA — login challenge (5-min/5-try pending window + time-step replay guard) and full self-service management in Settings (setup → confirm → disable with password).
• API keys — read / read,write scopes, one-time token display, masked in lists; key auth never mints a session.
• Audit log on every mutation — /app/audit page + GET /api/v1/audit (action/actor/text/date filters, paginated); login throttles (auth 10/min, signup 5/5min, agent enrol 20/min, public endpoints 20–120/min); CSRF hardening (JSON-only API, X-Requested-With required on session mutations).
• SSRF guard (SafeTarget) on all platform-originated checks/notifications: loopback/RFC1918/link-local/CGNAT/multicast blocked, DNS-rebinding safe, HTTP(S)-only protocols, no response-body echo in errors.
• Secrets at rest: enc:-prefixed base64 ciphertext via the app encrypter.
• SSO/OIDC (business plan) — per-tenant generic OIDC relying party: discovery doc (/.well-known/openid-configuration, 10-min cache, https-only, same-host/subdomain endpoint validation), PKCE S256, state + 10-min pending window, firebase/php-jwt JWKS verification with iss/aud/azp checks. Config in Settings → Single sign-on (issuer/client_id/secret — secret enc:-stored and masked on read, allowed_domain restriction, auto_provision JIT at role member). Login page "Sign in with SSO" → workspace slug → IdP redirect; failures return to /app?oidc_error=<reason>.
• Per-user UI prefs — tenant_users.prefs JSON (whitelisted top-level keys, 8 KB cap) via GET/PUT /api/v1/auth/prefs; powers dashboard widget visibility + ordering (customize mode on /app), incl. a fleet-health 24h sparkline widget.
• PWA — installable app shell: web manifest + maskable icons, service worker caches hashed /assets/* only (API/auth/pages always network), root-scope /sw.js route.

12 · Multi-tenancy & platform admin

• Tenants isolated at the row level; signup → trial (configurable days, 3/IP/day cap, unique slug w/ suffix).
• Team management: invite users, roles/caps, activate/deactivate, last-login.
• Tenant export/import commands (migration tooling).
• Platform admin API (api/admin, separate PlatformAuth): tenant list, impersonate (audited, unimpersonate exit), suspend/resume (402 wall).

13 · Billing, plans & licensing

Plan gates (Xodiac::PLAN_GATES): free → starter → pro → business; trial = business rank. Gate map: checks/metrics/RBL/webhooks = free · chat channels/advisories/status page = starter · SMS/PagerDuty/plugins/actions/ API keys = pro · multi-region = business.

Free limits: 5 devices · 20 checks · 300 s min interval · 30 d retention.

• Stripe Checkout + PayPal subscription webhooks (/webhooks/{stripe,paypal})
• Plan catalog in DB (plans w/ price ids), billing events log, checkout redirect builder
• Self-hosted: same image on-prem, community cap of 10 devices; offline-signed license keys (xodiac:licensegen, sodium signatures), optional phone-home (/api/license/ping)

14 · Background machinery

15 · Operator cookbook

Custom metrics — drop files in /etc/xodiac-agent/conf.d/metrics.d/:

# executable — stdout parsed as "name value" / "name=value" / "name: value"
cat > metrics.d/ups.sh <<'EOF'
#!/bin/sh
echo "ups.charge_pct $(apcaccess status | grep -o '[0-9.]*' | head -1)"
echo "ups.runtime_min $(apcaccess status | grep -o 'Runtime: [0-9]*' | grep -o '[0-9]*')"
EOF
chmod +x metrics.d/ups.sh

# or a node_exporter textfile — read verbatim, no exec
cat > metrics.d/hw.prom <<'EOF'
# TYPE hw_temp_celsius gauge
hw_temp_celsius 41.2
hw_fan_rpm 980
EOF

Filenames prefix metric names (ups.sh → ups.charge_pct); cap 512 pts/cycle, 10 s per executable. Metrics are graphable per-device and usable in device_metric alert rules like any built-in.

SNMP interfaces + custom OIDs — creds conf.d/snmp.json = {"community":"…"} (v2c; default public), enable snmp on the device, then params:

{ "target": "192.168.1.1:161",   // host:port, default 127.0.0.1:161
  "walk": "interfaces",          // GETNEXT ifTable walk → one link entity per NIC
  "walk_oid": "1.3.6.1.4.1.9",    // optional arbitrary subtree → entity meta
  "oids": ["1.3.6.1.2.1.1.3.0", "label=1.3.6.1.4.1.9.9.48.1.1.1.5"] }

SNMPv3 (USM) — set params.version: "v3" (or just put "user" in creds) and extend conf.d/snmp.json — secrets stay local, params never carry them:

{ "user": "monitor", "level": "authPriv",
  "auth_proto": "SHA256", "auth_password": "…",
  "priv_proto": "AES256",  "priv_password": "…",
  "context_name": "optional-v3-context" }

level: noAuthNoPriv | authNoPriv (default needs auth_password) | authPriv (needs both). Protos: auth MD5/SHA/SHA224/256/384/512, priv DES/AES/AES192/AES256 (+C Reeder variants). EngineID is auto-discovered per target; optional params.engine_id (hex) pins it. Implemented via gosnmp — USM localization/auth/priv are library-handled; validate against a live v3 device before relying on it in prod.

oids accepts a comma string, JSON array, "label=oid" pairs, or an {"oid": "label"} map (max 16). walk also accepts iftable/true. Walks are GETNEXT-based with proper BER varbind parsing — 64-row cap. With walk: "interfaces" the agent additionally emits per-interface traffic metrics: link.<ifName>.bytes_in / link.<ifName>.bytes_out (bytes/sec rates from ifHCIn/OutOctets with 32-bit fallback; counter wrap/reset safe, both v2c and v3) — graphable under the device's metric picker and usable in device_metric alert rules.

Self-hosted push channels — ntfy and Gotify sit beside Slack/Discord:

• ntfy: channel type ntfy, config {url, token?} — url is the full topic URL (https://ntfy.sh/mytopic or self-hosted); posts the message body with Title/Priority/Tags headers (severity maps to priority; resolves send low priority). Optional token → Authorization: Bearer.
• Gotify: type gotify, config {url, token} — url is the server base (https://gotify.local), token the app token sent as X-Gotify-Key; posts to /message with extras.ack_url pointing back at the incident's public ack link.

Both go through the same SafeTarget URL check, 10 s timeout, and response-body suppression as every other channel.

plugin_run action — Device → Plugins → "run now" (or POST /devices/{id}/action {kind:"plugin_run", plugin:"docker", op:"run"}). Runs one configured plugin immediately; op:"metrics" forces the metrics path. Result (entity/metric counts + sample names) lands in action history. Params always come from the device's stored plugin config — the payload can only name the plugin.

16 · Surface map