feat(fixtures): curated demo targets + seed script + golden PLC baselines (#187)
CI / Check (push) Skipped
CI / Check (pull_request) Failing after 3m1s
CI / Detect Changes (pull_request) Skipped
CI / Deploy Agent (pull_request) Skipped
CI / Deploy Dashboard (pull_request) Skipped
CI / Deploy Docs (pull_request) Skipped
CI / Deploy MCP (pull_request) Skipped

Versioned, reproducible set of representative targets so every scan path can
be exercised repeatably and the nightly regression (#188) has a baseline.

- fixtures/demo-targets/targets.json: 5 targets — PlcSps composite
  (pump_station.st + pump_fbd.xml + optional Modbus live URL + optional
  firmware image), PlcSps pure (conveyor.xml + traffic_light.st), plain git
  SAST (sharang/cra-vuln-demo, pinned), WebApp (juice-shop v19.2.1 + live
  URL), FirmwareRtos (zephyr example-application, pinned). Each carries an
  `expect` golden baseline (min_findings, sast_rule_ids, cwes, control_refs,
  min_sbom_components, scans_offered, pentest_supported, detected_facts).
- compliance-agent::fixtures: typed loader (DemoTargets/DemoTarget/
  DemoArtifact/Expect) + lib tests that keep the manifest well-formed and
  assert the PLC baselines offline by running analyze_tree over the checked-in
  fixtures (runs in the normal --lib CI job).
- scripts/seed-demo-targets.sh: curl+jq seeder over the public onboarding API
  (create → upload → detect → optional --scan), --only, --reset (deletes only
  the "Demo · " prefix), env overrides for infra-dependent artifacts.
- docs/guide/demo-targets.md + sidebar entry.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EgxGHn22YEfQz5fLHSHkLv
This commit is contained in:
Sharang Parnerkar
2026-08-31 14:33:01 +02:00
co-authored by Claude Fable 5
parent 0834547a74
commit 3ac4b34c29
6 changed files with 732 additions and 0 deletions
+84
View File
@@ -0,0 +1,84 @@
# Demo Targets
Certifai ships a small, versioned set of **demo targets** — representative
inputs that exercise every scan path repeatably. They double as the fixture
set for the nightly regression and as a ready-made walkthrough for demos.
The set lives in [`fixtures/demo-targets/targets.json`](https://git.breakpilot.com/sharang/compliance-scanner-agent/src/branch/main/fixtures/demo-targets/targets.json).
## What is in the set
| Key | Target type | Artifacts | What it exercises |
|-----|-------------|-----------|-------------------|
| `plc-pump-station` | PLC / SPS (composite) | `pump_station.st`, `pump_fbd.xml`, optional Modbus live URL, optional firmware image | PLC control-logic SAST (ST **and** FBD-as-XML), ICS probe, semantic CRA/master-control mapping |
| `plc-conveyor-line` | PLC / SPS | `conveyor.xml`, `traffic_light.st` | Pure control-logic SAST on a PLCopen-XML program plus a realistic OpenPLC-style sample with three planted defects |
| `git-cra-vuln-demo` | Backend service | git `sharang/cra-vuln-demo` | Plain git SAST: Semgrep → CWE → CRA control refs (`cra-ai-8/13/20`) |
| `web-juice-shop` | Web app | git `juice-shop/juice-shop` @ v19.2.1 + live URL | SAST + SBOM/CVE on a large Node app, DAST + pentest against the running instance |
| `firmware-zephyr-example` | Firmware (RTOS) | git `zephyrproject-rtos/example-application` | Tramiton detect handoff (`build_system = zephyr`), firmware source SBOM |
The PLC files are the same ones under `examples/plc-demo/` that the PLC rule
tests already run against, so their expected rule ids are enforced offline on
every CI run (`compliance-agent::fixtures` tests).
## Reproducibility
* Git artifacts carry a `pin` (commit SHA, plus `pin_tag` where a release tag
exists). The agent clones `branch`; the pin records **which commit the
baseline was recorded against**. When a baseline drifts, re-pin and update
`expect` in the same change.
* Uploaded artifacts are checked into this repository.
* Anything that needs infrastructure is **optional** and env-driven, so the
set seeds cleanly on a laptop and gains the dynamic pieces on `comp-dev`:
| Env var | Used by | Meaning |
|---------|---------|---------|
| `DEMO_WEB_URL` | `web-juice-shop` | Live URL for DAST/pentest. Default `http://localhost:3000` — run `docker run -d -p 3000:3000 bkimminich/juice-shop:v19.2.1`. |
| `DEMO_PLC_MODBUS_URL` | `plc-pump-station` | Modbus endpoint for the ICS probe (in-cluster default `modbus://plc-sim:502`). Skipped when unset. |
| `DEMO_PLC_FIRMWARE_IMAGE` | `plc-pump-station` | Absolute path to a device firmware image to attach. Skipped when unset. |
## Seeding the targets
```bash
# against a local dev agent (no Keycloak → dev tenant)
scripts/seed-demo-targets.sh
# seed and trigger the first scan of each
scripts/seed-demo-targets.sh --scan
# only some targets
scripts/seed-demo-targets.sh --only web-juice-shop,git-cra-vuln-demo
# wipe every "Demo · " target and reseed
scripts/seed-demo-targets.sh --reset --scan
# a deployed agent
AGENT_URL=https://comp-dev.breakpilot.com AGENT_TOKEN=$TOKEN \
DEMO_PLC_MODBUS_URL=modbus://plc-sim:502 scripts/seed-demo-targets.sh --scan
```
The script uses only `curl` + `jq` and the public onboarding API:
`POST /api/v1/targets`, `POST /api/v1/targets/{id}/artifacts/upload`,
`POST /api/v1/targets/{id}/detect`, `POST /api/v1/targets/{id}/scan`.
Every seeded target is named `Demo · <name>`; `--reset` deletes exactly that
prefix and nothing else.
## Manual (re)onboarding
Each target can also be created through the onboarding wizard:
1. **PLC targets** — pick *PLC / SPS*, upload the `.st` / `.xml` files from
`examples/plc-demo/`, optionally add a `modbus://` live URL. See
[PLC / SPS (CODESYS)](/guide/plc).
2. **Git targets** — pick the type, add the git URL and branch from the
manifest. Public repos need no credentials.
3. **Web app** — add the git repo *and* the live URL; enable DAST and, if
wanted, pentest on the scan-selection step.
## Golden baselines
Each target's `expect` block states what a healthy scan must produce
(`min_findings`, required `sast_rule_ids`, `cwes`, `control_refs`,
`min_sbom_components`, `scans_offered`, `pentest_supported`,
`detected_facts`). The PLC baselines are asserted in unit tests today; the
nightly regression story (#188) runs the full set against a live agent and
alerts on drift.