Skip to content
Install

volant shadow

Replay a run that ansible-playbook recorded, without contacting a host, and see what Volant would have done differently.

volant shadow replay reads a recording of the volant.compat.record callback (see Run recordings). It runs the same playbooks with the same options through Volant and answers every task that would reach a host with the result ansible-core recorded. Then it compares what Volant decided with what ansible-core did. No host is contacted and no agent starts. Switch without risk walks through the whole path, CI included.

Terminal window
volant shadow replay RECORDING [--json] [--ledger PATH]
volant shadow status [--json] [--require-ready] [--ledger PATH]

Write the report as JSON Lines: one object per divergence, then a summary object.

The ledger to append to or read, instead of .volant/shadow.jsonl beside the ansible.cfg in effect (in the working directory when there is none).

Exit 1 unless every playbook in the ledger is ready.

The recording names the files the run read. Each is looked up from the recording’s working directory, so run the replay at the same place in your checkout. The replay refuses to start, with exit code 4, when:

  • the recording’s format is newer than this Volant reads; the message names both versions;
  • a playbook or an inventory file differs from the recorded one (site.yml changed since the recording);
  • a file an -e @file argument names has changed.

A recording of an ansible-core outside the 2.19 series replays with an out-of-profile warning. A recording cut short by a run that was killed replays up to its last complete line, and the hosts stop where it stops.

The run’s options come from the recording: inventory, extra variables, tags, limit, check and diff modes, escalation, vault ids and forks. You do not give them again.

Tasks that reach a host get the recorded result instead. Volant then applies changed_when, failed_when, register, loops and until itself. An until gets the recorded failed attempts in order. A pause takes the answer its prompt got and does not wait. A host recorded as unreachable is unreachable. Tasks that run on the controller, such as debug, set_fact, includes and meta, are computed by Volant as usual.

A playbook with Python modules or fact gathering still needs ansible-core on the controller to prepare them, through VOLANT_PYTHON or the managed runtime.

Some values are different on every run: the pipe, env, password, lines and url lookups, any lookup with random in its name, a file lookup outside the directory the replay runs in, now(), and the random and shuffle filters. In a replay they give an unknown value instead. So does any key read from a recorded no_log result, since the recording keeps only its status.

An expression that reads an unknown value is unknown as a whole, whatever it does with it: text around it, a filter, a comparison, an attribute or an item. A task argument that is unknown is not compared, and the report counts it under unknown: and the reason, such as unknown: lookup 'pipe'. A when, a loop, a changed_when, a failed_when or an until that reads an unknown value follows the recording: the task runs or is skipped as it was, loops over the recorded items, keeps the recorded status, and stops at the attempt the recording stopped at.

The replayed run prints on stderr, exactly as volant playbook would. The report goes to stdout. It starts with a summary: how many tasks were compared, on how many hosts, and the divergences by kind. Then it lists what was not compared and why, and each divergence with its host, its file:line, its task name, what ansible-core did, what Volant did and the difference:

Shadow replay: 23 tasks compared on 2 hosts, 1 divergence: judgment 1.
Arguments not compared: no_log 2, not in the recording 8.
judgment local site.yml:30 fails
ansible: {"status":"ok"}
volant: {"status":"failed"}
diff: status: "ok" -> "failed"

A divergence is one of five kinds, from the most serious:

Kind What it catches
inventory A host is missing, or is in other groups than in the recording.
flow A task one engine ran and the other did not, a different order, or a handler that ran on one side only.
loop A different number of items, or a different item value.
args An argument Volant sends to a module with another value than the one the module ran with. Arguments only in the recording are the module’s defaults and are not compared.
judgment A different status (ok, changed, failed, skipped), a different condition that skipped the task, a different number of attempts, or a different recap.

After the first flow divergence on a host, the rest of that host is not compared, because every task after it is shifted. The other hosts are still compared. A no_log task is compared on its status alone, since the recording keeps nothing else of it.

With --json, each divergence is one line:

{"host":"local","task":{"path":"site.yml:30","name":"fails"},"kind":"judgment","ansible":{"status":"ok"},"volant":{"status":"failed"},"diff":"status: \"ok\" -> \"failed\""}

The last line is the summary:

{"summary":{"compared":23,"hosts":2,"divergences":{"judgment":1},"not_compared":{"no_log":2},"arguments_not_compared":{"no_log":2,"not_recorded":8},"truncated":false}}

The reasons in not_compared are after_flow (after a flow divergence on that host), no_log (compared on the status alone) and unknown: <reason> (a loop item that is unknown). The reasons in arguments_not_compared are no_log, not_recorded (ansible-core passed no invocation to the callback, as for debug, or as ansible-core 2.21 does for every task) controller (Volant ran the task on the controller and sent nothing) and unknown: <reason>, counted per argument.

One clean replay proves little. Each replay that is not refused appends one line to the ledger, .volant/shadow.jsonl beside the ansible.cfg in effect: the date, the playbooks and their hashes, the Volant version, the recording’s ansible-core and hash, the hash of volant.lock when there is one, the --limit and --tags of the run, the number of hosts, the tasks compared and not compared, the file:line of each task compared and of each task the replay compiled, the divergences by kind, and clean, dirty or out-of-profile. Volant does not decide where the ledger lives between runs: commit it, or keep it in your CI cache.

volant shadow status reads it and gives one row per playbook:

  • The streak is the number of clean replays in a row. A dirty replay resets it, and so does a new Volant version. A changed playbook or collection does not, because both engines run the same files. out-of-profile replays do not count either way.
  • The coverage is the share of the playbook’s tasks, every branch included, that the replays of the streak compared at least once. The tasks never compared are listed by name.
  • The verdict is ready when the streak reaches shadow_runs (5 by default) and the coverage reaches shadow_coverage (0.9 by default), both read from the [volant] section of ansible.cfg. It is diverged when the last replay was dirty, and not-ready otherwise, with the reason.
site.yml not-ready streak 2/5, coverage 96% of 25 tasks (1 never compared), no divergence
2 clean replays in a row, 5 wanted

With --json, each row is one object with playbook, verdict, streak, coverage, tasks, never_compared, last_divergence and reasons. A ledger line that cannot be read is left out with a warning that names its line number.

Code Meaning
0 No divergence.
1 At least one divergence.
4 The recording cannot be replayed against this checkout: a newer format, or a file that changed.

volant shadow status exits 0, or 1 under --require-ready when a playbook is not ready or the ledger is empty.