~/tutorials/047-the-conversation-the-browser-hid.md

047: The conversation the browser hid

Every web application is one conversation: a request and a response. The browser renders the response body and quietly disposes of everything else. The request you sent, the headers both sides exchanged, the status codes it followed or swallowed, and the entire half of the conversation that only ever happens as POST. Web application testing starts by getting that conversation back, and the first instrument for it is already on your box.

What this lesson is for

This is the door of the tier. Everything after this, proxies, spiders, scanners, fuzzers, is machinery for holding more of the conversation at once. None of it will make sense as anything but magic until you have read one exchange raw, end to end, and know exactly which half the browser was showing you all along.

The instrument is curl. Not because it is powerful, but because it is honest. It prints the conversation with nothing rendered, nothing followed, nothing swallowed, and it is installed everywhere by default. You will keep it for the rest of your career, long after the fancier tools become background noise.

What the reader meets

A page you have looked at a hundred times, this site, and its hidden half. The receipts in this lesson were all fired against l1ackers.com while writing it, which is the right way to learn this: aim the instrument at a page you already understand as a reader, and watch it become a different object.

What the browser shows you when you open the tutorials index:

$ curl -s -o /dev/null -w "%{http_code} %{size_download} bytes\n" https://l1ackers.com/tutorials

200 24689 bytes

Two hundred means the conversation succeeded, and the body is the page. That is the entire deal the browser offers you: one number it mostly hides, and the body it renders.

What the browser never shows you, the response headers, the rest of the server's half of the conversation:

$ curl -s -I https://l1ackers.com/tutorials

HTTP/1.1 200 OK
Server: nginx
Content-Type: text/html; charset=utf-8
ETag: W/"6071-WpsTG50dmt1RHCjgwosZi5+sFDs"
Strict-Transport-Security: max-age=31536000
X-Content-Type-Options: nosniff
X-Frame-Options: DENY

Read those lines as sentences. The server names itself and its version policy. The ETag is the body's fingerprint, how caching knows the page changed. The last three are promises about security behavior, and in a later lesson you will learn to read them as claims that can be true, false, or aspirational. All of this was present every time you visited in a browser. None of it was ever shown.

The instrument

curl's flags for this lesson are four.

-s silences the progress meter, because you are reading, not watching. -I asks for headers only, the server's half alone. -v is the whole conversation, both halves, with the request side prefixed > and the response side prefixed <. -o /dev/null with -w throws the body away and prints only what you ask for, status, size, timing.

The -v view is the one to sit with, because it shows your own half of the conversation, the request the browser sends on your behalf and never shows you:

$ curl -sv -o /dev/null https://l1ackers.com/robots.txt

> GET /robots.txt HTTP/1.1
> Host: l1ackers.com
> User-Agent: curl/8.22.0
> Accept: */*
>
< HTTP/1.1 200 OK

Four lines with > are everything you said. When a browser visits that same address it says more, much more, cookies, referrer, a dozen Accept headers, cache conditions. You have never seen a single one of those lines, and they are all yours, sent as you.

Then two shapes the browser actively disguises. The redirect, which the browser follows silently and renders as if nothing happened, and the error, which the browser renders as a designed page while the number that explains it sits unseen. Fire the miss:

$ curl -s -o /dev/null -w "%{http_code}\n" https://l1ackers.com/tutorials/047-nope

The browser would show you a friendly 404 page. curl shows you the fact: four hundred four, not found, no rendering, no diplomacy. Learning to read statuses as facts rather than pages is half of what this instrument is for.

And the verb the browser hides entirely. Browsing is GET. Changing state, logging in, posting, submitting, is POST, and the browser only speaks it when a form tells it to. You can speak it directly:

$ curl -s -X POST https://l1ackers.com/api/login \
    -H "Content-Type: application/json" \
    -d '{"handle":"nosuchuser","password":"x"}'

{"error":"Wrong handle or password."}

That is the login endpoint of this site, answering one wrong guess in the raw. Lesson 040 sprayed this exact endpoint. The spray was five lines around one curl line exactly like this one, and if you understand this exchange, you already understand how that attack was built, because you can see the material it was built from.

Try it

Run the four receipts above against this site yourself. Then run the same four against one page you use every day, and write down one header you did not expect to exist. Reading other people's headers honestly is the first enumeration habit of this tier.
Find the redirect. Some page you know follows a redirect before it renders. Use -w "%{http_code} %{redirect_url}\n" on your candidates until one answers 301 or 302, then follow it on purpose with -L and watch -v show you both hops, the conversation the browser swallows.
Send one POST on purpose. The login shape above against this site is fair game, one guess, wrong on purpose. Read the error as a fact about the endpoint's design: it answers the same whether the handle exists or not, and that sentence is a finding about care, you will learn why in the spraying lesson's quieter moments.

Reveal the answer

The whole conversation, annotated, the answer key to what the browser hid. One command, the site's own tutorials page, both halves:

$ curl -sv -o /dev/null https://l1ackers.com/tutorials

> GET /tutorials HTTP/1.1
> Host: l1ackers.com
> User-Agent: curl/8.22.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: nginx
< Content-Type: text/html; charset=utf-8
< Content-Length: 24689
< ETag: W/"6071-WpsTG50dmt1RHCjgwosZi5+sFDs"
< Strict-Transport-Security: max-age=31536000
< X-Content-Type-Options: nosniff
< Referrer-Policy: strict-origin-when-cross-origin
< X-Frame-Options: DENY

The reading, line by line. Your half is the four lines above the blank: the verb and path, the host that routes it, who you claim to be, what you will accept. The blank line is the end of your half, and everything below it is the server answering. First the verdict, 200, then the server's identity, then the body's type and exact length, then the fingerprint, then four security promises. Sixteen lines, and the browser had been showing you a rendering of one of them all along.

Two moves to keep from this receipt. Match the Content-Length against the -w "%{size_download}" answer from the top of the lesson: 24689 both places, the two halves of the instrument agreeing with each other, which is your sanity check that you are reading the same conversation. And notice what is absent: no Set-Cookie line, because this site serves reading without sessions. In a later lesson you will watch a login response hand you a Set-Cookie, and that line, which the browser silently stores and silently replays on every later request, is the single most consequential thing the browser ever hides. The session cookie is a bearer token, lesson 038 proved it on this site, and the browser keeps it invisible while carrying it everywhere.

What this lesson does not claim

curl shows you the conversation, and that is all. It does not render, so it cannot tell you what the page does to a user. It does not execute scripts, so it cannot see anything the page builds after loading. It does not intercept, modify, or repeat on your behalf, which is the next lesson's instrument and the reason it exists. The browser hides the conversation. curl hides the page. You will spend this whole tier learning to hold both at once.