The Runbook That Had Never Worked

An automation agent watches for alerts and runs named remediations from an allowlist. One of them restarts a container on a host. It had passed its guardrail test suite on every run for weeks.

The first time I watched it execute end to end, it failed. It had never worked, on any host, since the day it was written.

The error

no configuration file provided: not found

That is the message docker compose gives when it cannot find a compose file in the working directory. The runbook built what looked like a perfectly reasonable remote command:

argv = ["ssh", host, "bash", "-lc",
        "cd /opt/stack && docker compose restart blackbox"]

Why it failed

ssh does not preserve argument boundaries. It joins everything after the hostname with spaces into a single string and hands that to the remote login shell, which parses it again from scratch. The quoting that grouped those words into one argument locally does not survive the trip.

# what the remote shell actually received
bash -lc cd /opt/stack && docker compose restart blackbox

# which it read as two separate commands
bash -lc cd /opt/stack
docker compose restart blackbox

The first command changed directory inside a shell that immediately exited. The second ran in the login shell's home directory, where there is no compose file. I confirmed it by replacing the payload with pwd, which printed the home directory rather than the stack directory.

Why the tests never caught it

The guardrail suite existed to answer a security question: can this action be tricked into touching a host or a service outside its allowlist? It built the argument list, asserted the host and service were permitted, and asserted that disallowed input was rejected. Every one of those assertions was correct, and none of them ran the command.

The suite validated arguments. The bug lived in what happens to those arguments in transit. No amount of argument checking can see a transport that reinterprets them.

The fix, and the fix that broke something else

My first correction quoted the directory path before interpolating it. That is the textbook answer, and it broke every host whose stack directory is written relative to the home directory, because quoting a leading tilde turns it into a literal directory name that does not exist.

The guardrail suite caught that regression with two failures, which was the first genuinely useful thing it had done. The working form passes the whole payload as a single argument and leaves tilde expansion to the remote shell. After the fix: fifteen of fifteen guardrails passing, plus a real container restart with probes green afterwards.

What I took from it

A test suite that has never executed the thing it guards is measuring your intentions, not your system. The security assertions were worth keeping, but they created a false impression of coverage: the action was verified in the sense that mattered least. Any remediation that can be run unattended now has at least one test that actually runs it against something real and checks the result, not just the arguments.