~/tutorials/061-when-to-automate-and-when-it-lies-to-you.md

061: When to automate, and when it lies to you

You now own the whole automation bench: scanner, baseline, fuzzer, templates. This lesson is the operator's manual for the bench, and it has two chapters with one root. When to automate is a decision about cost and coverage. When automation lies is a fact about assumptions, because every tool this tier taught you carries an assumption about how targets behave, and the moment a target breaks the assumption, the tool's output inverts from evidence to fiction, fluently, at machine speed.

What this lesson is for

The tier saved this for second-to-last on purpose. You needed the tools before their failure modes could be shown on them, and you needed 054 through 056's reading discipline before a lying report could be caught by anything other than luck. This lesson builds a target designed to break the tools' shared assumption, shows all of them lying on it in the same hour, and then hands you the decision framework for when the machine earns its keep and when it spends your credibility.

What the reader meets

The tarpit, built on the web lab page: a server that answers 200, with an identical friendly body, to every request for every path. Nothing is wrong with any page, because every page is the same page, and that one page is fine. It is not an attack tool. It is the shape of several real things: the catch-all route a framework ships by default, the WAF that answers for the application, the marketing redirect that swallows every path. And it breaks the assumption every discovery tool carries, that a 200 means the name exists.

The fuzzer's lie first, because it is the most total. Seven guesses, including nothing-here and zzz, which mean nothing:

ffuf -u http://172.31.21.2:5007/FUZZ -w seven-words.txt

FOUND: admin        [200] 54 bytes
FOUND: login        [200] 54 bytes
FOUND: backup       [200] 54 bytes
FOUND: nothing-here [200] 54 bytes
FOUND: zzz          [200] 54 bytes

7 hits, 7 guesses, 100 percent hit rate

Every word found. The fuzzer asked seven questions and received seven identical yes answers, and its honest, faithful, machine-speed output is that everything on this server exists, including the words that cannot. The lesson of 058 said it in advance, and here is the receipt: the wordlist is the question, the status code is the meaning, and the tarpit makes the meaning a lie. A tester who reports those seven rows has reported seven fictions with tool-generated confidence, and the fix was in the lesson all along, calibration, -fs 54, the friendly page's size, and all seven fictions vanish in one flag. The tool did not know to add it. You had to.

And the scanner's lie, same hour, same target. nikto, whose checklist interprets a 200 on a known path as the path existing, walks its list and finds, among other things, a Java monitoring console:

+ [007303] /JAMonAdmin.jsp: JAMon - Java Application Monitor Admin interface
  identified. Versions 2.7 and earlier contain XSS vulnerabilities.
  See: https://nvd.nist.gov/vuln/detail/CVE-2013-6235

There is no Java on this server. There is no JAMon. There is no page at that path, only the tarpit's one answer, and the scanner has reported a named, versioned, CVE-linked vulnerability on a Python process serving one friendly page. Every element of that row is fabricated by the collision of a checklist with a target that broke its assumption, and the row is formatted exactly like a true one. This is the deepest lie on the bench: not a wrong version, 056's lie, not a missed meaning, 055's miss, but a positive finding of a thing that does not exist anywhere, delivered in the report's voice.

The instrument

The assumption table, the root of every lie, one row per tool. The scanner assumes 200 means exists and error pages mean what they say. The baseline assumes the noise floor is stable. The fuzzer assumes different names answer differently. The template assumes its matcher's yes means its finding's cause. The proxy assumes it is seeing the whole conversation. Every one of those assumptions is true on a well-behaved target, every one is breakable by design or accident, and the whole craft of when-to-automate is knowing which assumption your next run depends on and whether this target honors it.

The honest test for that, run before trusting any run on a new target: the nonsense probe. Ask for a path that cannot exist, zzz-definitely-not-here-987654321, and read the answer. A 404 honors the assumption, and the discovery tools may be trusted, with 054's kinds and 056's card as always. A 200 honors nothing, and every discovery finding on that target is unverified until the calibration flag or the behavior channel separates real from friendly. One request, before the run, and the entire afternoon's output changes grade from evidence to maybe. The nonsense probe is the cheapest tool on the bench, and no lesson before this one needed it, because no lesson before this one had a target that lied this way.

When to automate, then, falls out as three honest cases. Automate the census: breadth work, known-unknown questions, the eight thousand requests of 054, where the machine's patience is the entire value and your reading discipline is the quality control. Automate the repetition: baselines, regression, the policy templates of 060, where the machine's memory and consistency beat your memory, every time, forever. Automate the boring, and only the boring, with the boring defined precisely: any question where the cost of a wrong answer is caught downstream by a human read. Do not automate the meaning, 055's law, unchanged. Do not automate the trust, 056's card, unchanged. And do not automate the first contact with a new target, because the first contact is where the assumptions live or die, and the nonsense probe is yours, not the tool's.

Try it

Build the tarpit from the web lab page, run the fuzzer with no filter, and watch your own seven fictions arrive. Then add -fs 54 and watch them go. Then run nikto and find its Java console on your Python server. You have now personally receipted every lie this lesson names, on your own machine, in under ten minutes.
The nonsense probe, institutionalized. Add it to your session's opening moves, before any discovery tool: one request for a path that cannot exist, the answer recorded. Do it on every target for a month. You are building the habit that would have caught every lie in this lesson before any tool reported one, and habits beat flags, because flags are per-run and habits are per-career.
The assumption inventory. Take the tier's five tools, scanner, baseline, fuzzer, templates, proxy, and write each one's assumption in one sentence, then one sentence on how you would personally break it if you were the target. This is red-team thinking applied to your own bench, and the ten sentences you produce are worth more than the ten tools.

Reveal the answer

The automation contract, the one page to keep with the bench, the three cards of 054-056 folded in, and the two new laws of this lesson:

BEFORE THE RUN
  nonsense probe        zzz-987654321 -> 404? assumptions hold.
                        200? everything downstream is maybe.
  subset with intent    060's habit. All templates is 057's noise,
                        weaponized.

DURING THE RUN
  calibrate             -fs the friendly page, -fc the friendly count.
                        The flag is yours to add. The tool will not.

AFTER THE RUN
  read with 054         every row: which kind of yes?
  trust with 056        version rows: second source or capability probe
  mean with 055         the report's center is yours, not the tool's
  verify 100% hits      a hit rate near 100 is not success. It is the
                        tarpit signature, and this lesson's receipt.

THE TWO NEW LAWS
  The lie is formatted exactly like the truth. There is no font for
  fiction. The nonsense probe is the only cheap separator.
  The cost of automation is credibility: every row you forward from a
  tool is a row you signed. 044's report discipline has no tool
  exception.

The contract's last line is the tier's whole answer to its own title, and it is the sentence to keep: the question was never whether to automate, it was which of your judgments you are willing to run at machine speed, and which you refuse to delegate even once. The bench runs your patience and your memory at machine speed now. The meaning, the trust, and the signature stay in the chair, and the final lesson of the tier puts the chair where it was always pointing: in front of a finding that matters, with no tool left between you and the decision.

What this lesson does not claim

Not every 200-everywhere target is hostile, and the tarpit shape appears for innocent reasons constantly, catch-all routes, SPA frameworks, misconfigured error pages, which is precisely why the nonsense probe belongs in your opening moves rather than your incident response. The lies this lesson receipted were designed, but the same lies arrive by accident, and the profession's scars are mostly from the accidental ones. What the lesson claims is the shape: assumptions break, output inverts, the inversion is formatted as truth, and the discipline that catches it is one cheap probe and a signed report. The bench is good. The chair is better. The last lesson of the tier is about the chair.