Optimizing Write-Intensive Database Performance (Sponsored)
Free masterclass: Learn practical strategies for predictable low-latency writes at scale
Free Masterclass: Optimizing Write-Intensive Database Performance
Database writes – at scale – are one of the hardest problems in distributed systems. This masterclass will teach you how to understand and avoid latency spikes in real-time, write-heavy database workloads. Our panel of experts will share a practical framework for diagnosing write bottlenecks and knowing which strategies to apply in different scenarios.
After this free 2-hour masterclass designed for developers, engineers, architects, and database practitioners, you will know how to:
Identify which factors (database internals, database configurations, data modeling…) are impacting your write performance
Avoid mistakes that have caused 40x write amplification in production
Sustain low P99 latencies even during sustained growth and volatile spikes
All attendees will get the complete Database Performance at Scale book by the masterclass instructor Felipe Mendes.
How does a website charge a visitor who arrives anonymously, skips every advertisement, and leaves within a second?
For most of the web’s history, the question rarely came up, because the visitor was a person whose attention a site could sell through an ad or a subscription. If the website had good content, the website owner could count on multiple such visits by the same user. But now the visitor being a person is not always true. More than half of the traffic online now comes from software that acts on a person’s behalf, requests a page, and leaves without engaging with any ads or subscriptions [2].
Cloudflare seeks to change this.
As you might be aware, Cloudflare sits between a large share of the world’s websites and everything requesting them. It works as a reverse proxy that each request passes through before it reaches the origin server [1]. This lets Cloudflare read a request and act on it early. Over the past year, the company has used this position to sort automated traffic by what it does, to verify the identity behind a request, and, most recently, to collect payment for a request through an open protocol named x402 [1].
In this article, we will go through Cloudflare’s solution in the following five steps:
The web’s usual way of earning value leans on human attention, and agent traffic changes it fundamentally.
A look at Cloudflare’s initial solution around blocking automated traffic and then charging for each crawl.
Settling identity, permission, and payment inside a single request.
A look at how x402 exchange completes this settlement.
The x402 exchange completes this settlement through a short back-and-forth over ordinary HTTP
The costs and open questions of this approach
Disclaimer: This post is based on publicly shared details from Cloudflare. References at the end. Please comment if you notice any inaccuracies.
The Attention Model
For most of its history, websites made money after a request rather than during it. A browser asked for a page, and the server returned it at no charge. The value arrived later, once a person saw an advertisement, bought a subscription, or came back for another visit [1]. The request itself stayed free, and this setup funded a large part of the Internet.
A growth in agent traffic is changing this dramatically. An agent is software that acts on a person’s behalf, which in practice means it requests a page or a data feed once, takes what it needs, and finishes in a single pass. It moves past advertisements, operates outside any subscription, and completes its task before a site has a chance to earn from that attention [1]. Each of the three old settlement points depended on a person staying long enough to be counted, so software traffic leaves those points idle.
The strain here is that this kind of traffic is now fast becoming the majority. More than half of the requests reaching websites come from software rather than people [2]. This means that request volume climbs while revenue stays flat.
To make things clear, parts of the Internet were already charging by usage before any of this. Cloud services and APIs have been sold by the call and by the hour for years, though only to a buyer the seller already knew, who signed up and received an API key [1]. Charging an anonymous caller a fraction of a cent for a single request stayed impractical, because collecting such a small payment once cost more than the payment returned [1].
If the value used to settle downstream of the request, the natural question is which party is positioned to move that settlement back onto the request. The answer starts with where Cloudflare sits.
The Proxy Layer
A reverse proxy is a server that stands in front of other servers and receives requests on their behalf. Since Cloudflare operates as a reverse proxy for a large portion of the web, a request headed for one of those sites reaches Cloudflare’s network first and passes through it on the way to the origin [1]. The origin is the site’s own server, the machine that ultimately holds the page or runs the API.
On a side note, the position of a proxy is more general-purpose than caching. Caching stores a copy of a response so it can be served quickly the next time, and it is just one of the many useful things a proxy can do. From the same middle position, a request can also be read, classified, checked, and acted on before it continues to the origin. This broader capability is quite consequential at Cloudflare’s scale. [2].
However, acting on a request early depends on first knowing what the request is for. This is a classification problem. Let us see how Cloudflare handles this.
Traffic Classification
Before any rule can apply to an agent, the traffic has to be sorted by what it does. Cloudflare’s taxonomy groups automated traffic by behavior rather than by the single label “AI”. Three behaviors impact the key policy decisions [4].
Search covers behavior that builds an index of a site so an engine can answer questions about it later. This behavior is responsible for sending referral visitors back.
An agent covers behavior that acts in real time on a person’s behalf, usually with a human waiting for the result. Think of it like an assistant fetching a page during a conversation.
Training covers behavior that takes content to train or fine-tune a model, where the content is absorbed into the model rather than pointing a visitor back.
These three appear identical in a raw request log, yet they carry very different consequences for a site’s business.
A single crawler can also perform more than one of these behaviors, but this separation lets them record all of them[4]. A crawler that both builds a search index and gathers training data is tracked as doing both, which lets a site owner reason about the full set of things that the crawler does on their pages.
Cloudflare classifies further behaviors as well, including checkout actions and data collection, and it also lets a site express what a bot may store and reshare after accessing a page [4].
With a way to see the proxy position and a way to classify what arrives, let us now look at Cloudflare’s first attempts at implementing a policy.
Blocking and Charging
Cloudflare’s first solution was a simple one. A single control lets a site block automated AI traffic outright. This protected the content and left the earning model untouched. Blocking answers the question of access, yet it does not rebuild the revenue that the old bargain provided.
The second solution added a middle path.
Pay Per Crawl lets a website allow a crawler, block it, or charge it a flat per-request price, with Cloudflare handling the billing as the merchant of record [5]. A site owner could keep a crawler out, let it through, or attach a price to its access, all from one setting. This turned the binary of block-or-allow into three options and gave content owners a way to earn from crawler access at network scale [5].
A year on, Cloudflare made an adjustment to its own model. The company argued that a crawl is a weak measure of value, because a single page might be crawled once and then cited in thousands of AI answers, or crawled repeatedly and cited in none [3]. It backed the argument with a figure from its own network, that more than half of the crawl traffic from well-behaved bots goes to re-fetching pages that have stayed the same since the last visit [3]. Counting crawls, then, counts something that only loosely tracks the value delivered.
So the unit of payment began to move from the crawl toward the use, an approach Cloudflare describes as Pay Per Use. It is candidly framed as an experiment at this point [3]. This is because while pricing the outcome aligns payment with value more closely than pricing the fetch, it is also harder to measure.
The Request Layer
The design goal for recent changes is to resolve identity, permission, and payment inside a single request, at the edge, before the origin responds.
Three main concerns are considered over here:
Identity: The traditional identifier, the User-Agent string, can be set to any value by the caller, so it offers weak assurance. Cloudflare’s answer is Web Bot Auth, an authentication method that uses cryptographic signatures in HTTP messages to verify that a request comes from a particular automated source [7]. In practice, the operator signs its request with a private key and publishes the matching public key at a known location, and Cloudflare validates the signature at the edge [8]. A valid signature stands in for a reliable identity, which replaces a guess with a verifiable claim.
Permission: This is expressed through the behavior classification already covered and through the preferences a site sets about how its content may be used [4].
Payment: This comes last and is attached to the request through x402. We will cover this in detail in the next section.
All three concerns resolve at the edge, so the origin receives a request only once identity, permission, and payment have been settled [1]. The metering and settlement are taken away from the website’s own servers. What stays with the site owner is the part that matters to them, which is their rules, their prices, and their revenue [1].
To summarize:
Identity answers who is making the request, through a signed and verified claim.
Permission answers whether this behavior is allowed on these pages, through the classification and content preferences.
Payment answers whether the caller has paid the stated price through the x402 exchange.
These pieces sit at different stages of maturity. Identity verification through Web Bot Auth is available today at the edge [7], while the Monetization Gateway that brings the payment piece together is open as a waitlist rather than a shipped product [1].
The key takeaway is that when one component sits in the middle of a flow, the concerns that are common across every request are collected at that point. Authentication, authorization, and billing consolidating at a gateway is the same pattern that appears in service meshes and middleware.
The x402 Exchange
The x402 protocol makes it possible to pay over HTTP. It takes its name from a status code that has been part of the HTTP standard for a long time [1]. The code is 402, and it means Payment Required. Sites behind Cloudflare already send more than a billion of these responses on an average day, which shows how often a machine requests something priced and receives a message that a payment is due [6].
See the diagram below that shows the overall setup:
The exchange runs through a short sequence that can be compared to a small state machine [1].
A client requests a resource that sits behind a price.
Rather than returning the resource, the server responds with 402 and a small payload stating the price, the accepted asset, and where to pay.
The client re-sends the same request with proof of payment attached.
A facilitator verifies the payment, and the server returns the resource.
Two properties make this suitable for machine traffic. The payment amounts can be very small because the protocol adds almost no overhead to the request. And the payment itself serves as the credential, so a buyer with no prior relationship can access the content by showing the proof of payment [1]. This property is the one that matches an anonymous agent passing through once, since it removes the signup step that per-seat licensing and API keys always required.
The negotiation process is handled inside ordinary requests and responses, with a redirect to a checkout page absent and a separate payment API absent [1]. Nothing was added at the protocol level, since the 402 code has been part of HTTP for decades. What changed is that a settlement angle now exists that makes collecting a fraction of a cent practical.
Costs and Limits
Resolving identity, permission, and payment at the edge means one proxy performs those functions for a large share of the web at once [1]. Cloudflare presents that responsibility as an advantage for settling everything inside one request. But it can also be seen as a potential risk when so much runs through a single provider.
Cloudflare also states several limits plainly:
Trust that travels with a request may reach only the traffic that can afford to be identifiable, and small or privacy-sensitive sources of traffic need other building blocks, such as private rate limiting [4].
Usage-based payment helps a site that already has demand, and does little for a small site whose real difficulty is discoverability rather than monetization, which leaves that site weighing visibility against giving content away [4].
Collecting a payment through this exchange depends on callers built to recognize and honor the 402 response. Therefore, the revenue depends on adoption within the ecosystem. [6].
Pricing an outcome rather than a crawl aligns payment with value, but it is also harder to measure and verify. This is the reason Cloudflare frames the shift to Pay-Per-Use as an experiment [3].
Lastly, Cloudflare argues that crawlers combining several purposes under one identity reduce transparency for a site. This is because the site cannot tell why it is being accessed [4]. While the argument is sound based on technical merits, it also aligns with Cloudflare’s commercial interest in separated, verifiable traffic.
None of these costs undoes the shift. They simply mark the current trade-offs that website owners should consider before adopting it.
Conclusion
The web is moving value settlement from after the request to inside it. For most of the web’s history, a request was served free, and value was settled later through human attention. However, agent traffic, which now comprises the majority of requests, leaves that later settlement with nowhere to land [2].
Cloudflare’s response is to resolve four things from its position as a reverse proxy: seeing what a request is through classification, verifying who sent it through Web Bot Auth, enforcing the site’s rules, and settling payment through the x402 exchange, all before the origin responds [1][4][7].
Identity verification lives at the edge today, and the payment gateway opens as a waitlist [1][7]. The open questions around concentration, reach, adoption, and how to price an outcome are worth tracking as the model develops.
References:
Announcing the Monetization Gateway: charge for any resource behind Cloudflare via x402
Content Independence Day, one year on: building the business model for the agentic Internet
Your site, your rules: new AI traffic options for all customers
Introducing pay per crawl: Enabling content owners to charge AI crawlers for access
Launching the x402 Foundation with Coinbase, and support for x402 transactions
Forget IPs: using cryptography to verify bot and agent traffic
it's good to learn new ways of thinking
So you’re telling me attention is not all you need? 🤣