~/tutorials/052-your-first-pass-the-spider.md

052: Your first pass, the spider

You have a target, a scope, and an engine that holds conversations. The first real pass over a web application is a crawl: walk the site the way a fast, tireless visitor would, following every link, filling every form with something, and record everything you meet. The spider is that visitor, and this lesson teaches what it is, what it found, and, just as important, what it walked past without ever knowing.

What this lesson is for

A map of the application. You cannot test what you have not found, and applications are routinely larger than their owners believe, with pages linked from other pages, parameter routes nobody remembers building, and documentation folders shipped by accident. The spider's job is to make the surface area visible: one list of everything that answers, which becomes the input queue for every later lesson, the scanner's target list, the fuzzer's wordlist seed, the manual review's map.

What the reader meets

The lab from the last two lessons, the web lab apparatus page builds it in full, with one deliberate change made for this lesson. The address in the command below, 172.31.12.2, is that page's docker-network address for the target, your own run prints its own number. Before crawling, the site was planted with two things nobody links to: a hidden/note.txt and a backup.old, both live, both answering 200 to anyone who asks by name. They exist. Nothing in any page points at them. Hold that fact while the spider runs.

One action call from last lesson, aimed at the root:

$ curl -s "http://127.0.0.1:8090/JSON/spider/action/scan/?url=http://172.31.12.2/&maxChildren=50"

{"scan":"0"}

The status climbs to 100 in about twenty seconds, and the history count says thirty requests. Thirty requests is the pace of the walk, and the pace matters: roughly one and a half requests a second, polite by any standard, and configurable if your worksheet from lesson 048 wrote a rate line. The spider did not hammer anything. It strolled.

What it found, read from the history, is the map. Twenty eight distinct paths, and the shape of the list teaches what a crawl is:

/robots.txt
/login.php
/setup.php
/about.php
/instructions.php
/instructions.php?doc=PDF
/instructions.php?doc=changelog
/instructions.php?doc=readme
/dvwa/css/main.css
/dvwa/js/dvwaPage.js
/docs/DVWA_v1.3.pdf
/var/www/html/config/config.inc.php

Read the list as three strata. Pages a human would click, the logins and abouts. Assets the browser would load, the css and js, which the spider collects because they are linked and because 049's census applies to crawlers too. And the third stratum, the interesting one: paths no menu shows, a changelog reader with a parameter, a pdf under docs, a config path leaked from somewhere in the page source. That third stratum is why the spider exists. Every one of those was invisible until something followed every link mechanically.

The instrument

What the spider actually does, so its limits are laws instead of surprises. It fetches a page, reads it as text, extracts every URL-shaped string, hrefs, srcs, actions, and puts them on a queue. It fills forms with placeholder values and submits them, because forms are doors too. It does not execute the page's scripts by default, so anything the page builds client-side, menus rendered by javascript, links computed at runtime, does not exist for it. And it follows what it is given without judgment, which cuts both ways: it will find the config path, and it will also fire the logout link, submit the delete form with placeholder text, and touch the setup page, which this crawl did, a GET of setup.php sitting right there in the history.

That last fact is the one to slow down on. A spider is not a reader. It is a doer. Every form it meets, it submits. Every link it meets, it requests. On a lab box that is the point. On a real target, the crawl is your first attack traffic, before any scanner fires, and the worksheet's excluded list is not decoration: the admin pages, the delete endpoints, the untested payment flow. ZAP's spider has an exclude list for exactly this, set it before the crawl, not after the history shows you what you did.

And now the two planted files, and the lesson's center. The spider found twenty eight paths. It did not find hidden/note.txt. It did not find backup.old. Both answer 200 when asked directly, receipted after the crawl through the proxy lane. The spider did not miss them because it was lazy. It missed them because nothing links to them, and the spider is a follower of links. Discovery by crawl is bounded by the link graph, and the link graph is the site's story about itself, written by its developers. Anything the developers did not write a link for does not exist, as far as the crawl is ever concerned.

That boundary is not a flaw to fix. It is the division of labor for the whole tier. The spider answers: what does the site say it has. The fuzzer of a later lesson answers: what else answers when asked by name. Between those two questions lives most of web application testing, and the tester who knows which question their tool is asking is the tester who knows what their results mean.

Try it

Crawl your lab box from the root and read the history as a map. Count the distinct paths, then sort them into the three strata: clicked, loaded, leaked. The third stratum is your findings queue for every later lesson, and writing it down now is the map-making habit.
Plant your own invisible file, then crawl. Watch it not appear. Then fetch it by name through the proxy and watch it answer. That ten-minute exercise, watching a thing exist outside the crawl's world, teaches the boundary faster than any paragraph can.
Crawl again with one exclusion configured, the setup page or any other state-changing path. Then read the history and confirm the exclusion held. You have now configured a spider the way it must be configured before any crawl of anything that matters, and lesson 048's worksheet is where the exclude list comes from.

Reveal the answer

The crawl review card, the five questions to answer after every spider run, with this lesson's receipts as the worked answers:

1. How many requests, over how long?       30 in ~20s, 1.5/s, polite
2. What strata did the paths sort into?    clicked / loaded / leaked
3. What did it touch that changes state?   GET /setup.php, seen in history
4. What did the exclude list hold?         whatever the worksheet says
5. What does NOT appear that should?       planted: hidden/note.txt, backup.old

Question five is the one nobody asks, and it is the reason this card exists. The history shows you what the spider found. Nothing shows you what it walked past. The only way to know the crawl's blind side is to plant something you know is there and confirm the crawl cannot see it, which is why the lab ships with the two files, and why Try it number two has you plant your own. A crawl you have never tested for false absence is a crawl you trust for no reason.

The card's deeper use, save the output per target. The map of stratum three, the leaked paths, is the input queue for the scanner lesson. The exclusion receipts are part of your evidence of care, in lesson 044's shape, when somebody asks what your crawl touched. And the request count over time is your answer when the client asks what load you put on their site, asked with a number because you counted. Everything on the card is a receipt you already own, because the engine held all of it from lesson 050, and reading the crawl is just asking the history the right five questions.

What this lesson does not claim

The map is the site's own story about itself, and applications lie in both directions. Pages exist that no link reaches, the spider's blind side, proven with the planted files. And links exist that lead nowhere real, error pages, dead routes, the spider records them all with the same flat existence. The map is a starting inventory, not a truth. The next lesson puts a human hand on the same instrument, and the one after that asks the fuzzers' question: what answers when you guess. The crawl is the first word of the enumeration, not the last.