What unit tests can't do

This is where DART earns its place in CI. Every example below asserts something a unit test — or even a normal integration test — structurally cannot.

Survive a real reboot

Mocks can’t power-cycle a machine. Capture state, reboot for real, and prove the service came back.

tests:
  - name: record the boot id
    node: vm
    type: execute
    options:
      command: cat /proc/sys/kernel/random/boot_id
      capture: boot_before

  - name: reboot the machine
    node: vm
    type: reboot
    options:
      mode: graceful        # 'force' models a power cut
      timeout: 300

  - name: the machine really rebooted
    node: vm
    type: execute
    options:
      command: '[ "$(cat /proc/sys/kernel/random/boot_id)" != "{{capture.boot_before}}" ]'
      evaluate: { exit_code: 0 }

  - name: the service came back by itself
    node: vm
    type: service_status
    options: { service: myservice }

Wait for a distributed system to converge

Elections settle, DNS propagates, replicas catch up. retry reruns the test until reality agrees or the deadline passes — no sleep 30 guesses.

  - name: the cluster elects exactly one leader
    node: [db-1, db-2, db-3]
    type: consistency
    retry: { timeout: 90, interval: 5 }
    options:
      command: cluster-role
      evaluate:
        matching: { pattern: "^leader$", count: 1 }   # split brain fails

Prove firewall and network policy

Reachability is a property of where you stand. from: node — the default, spelled out here for clarity — probes from the machine that matters, and asserting closed proves a rule works.

  - name: the app server can reach the database
    node: app
    type: port_check
    options: { host: db.internal, port: 5432, from: node }

  - name: the app server cannot reach admin SSH
    node: app
    type: port_check
    options:
      host: admin.internal
      port: 22
      from: node
      evaluate: { status: closed }

Catch configuration drift across a fleet

Per-node tests can’t compare nodes to each other.

  - name: every node runs identical config
    node: [web-1, web-2, web-3]
    type: consistency
    options:
      command: sha256sum /etc/app.conf

A failure names who disagrees: web-1,web-2 => "3f2a…" | web-3 => "9c1b…".

Gate on performance regressions

Extract measured numbers and assert on them with tolerances.

  - name: throughput has not regressed
    node: testbed
    type: execute
    options:
      command: loom run --json tcp-100g.yaml
      extract:
        throughput_mbps: { jsonpath: "$.summary.throughput_mbps" }
        p99_us:          { regex: "p99=([0-9.]+)us" }
      evaluate:
        throughput_mbps: { within: 12476, tolerance_pct: 5 }
        p99_us:          { lte: 49 }

Test destructively, then roll back

Snapshot before you break things; restore in teardown. Far faster than rebuilding the environment.

setup:
  - name: capture clean state
    node: vm
    step:
      type: snapshot
      options: { name: clean }
teardown:
  - name: roll back
    node: vm
    step:
      type: snapshot
      options: { name: clean, action: restore }

Warning: teardown steps do not run when a run aborts. An unhandled setup step failure, a test that errors, and --stop-on-error on a failing test all skip straight to node and platform teardown, so a rollback placed in teardown: does not execute. This is harmless for ephemeral nodes that are deleted at teardown, but a long-lived node — an SSH host, or a container kept across runs — stays in its broken state. Suites that depend on a teardown rollback are best run without --stop-on-error. A test that merely fails is not an abort: without --stop-on-error the run continues and teardown steps do run.

Know your certificates before your users do

  - name: the gateway certificate is not about to expire
    node: local
    type: tls_cert
    options:
      host: gateway.internal
      evaluate:
        min_days_remaining: 30
        chain_valid: true