What happened when crawlers found Nerdulator

The traffic that was not readership

Nerdulator can render pages for a very large range of inputs. The sitemap is deliberately smaller: it lists pages the engines nominate as useful, rather than every URL the application can resolve. That was not enough on its own. A crawler could start at a seeded page, follow a link to an unseeded value, and continue through more generated pages.

In September 2026, two AI crawlers requested more than 200,000 distinct URLs in four days. Many were arbitrary numbers, timestamps, and hex colours. Those pages were valid responses, but most were not useful search results and were not pages I meant to put in front of crawlers.

The first fix was about crawl boundaries. Pages that are not deliberately seeded can still be opened by a person, but their links do not invite crawlers to continue indefinitely. Indexing and link following are separate decisions. Some unseeded pages may still be useful if someone searches for them, but following every link from them creates an unbounded crawl.

That cut the crawl substantially. It did not solve every load problem.

The outage

On 4 October, the site started returning 502s and timing out. Search impressions dropped from roughly 20,000 a day to about 240 over the same period. The app was running as one copy, and rendering pages was effectively serial. Under crawler load it sat around 100 to 140 percent CPU, while requests queued behind the render work.

We looked at caching first. The access logs showed 171,166 page requests for 150,620 distinct URLs. Only 267 URLs, about 0.2 percent, were requested by more than one kind of client. Even a perfect shared cache would have served about 12 percent of those requests. Most were cold visits to different generated URLs. A bigger cache would not address the cause.

The useful fix was to put a boundary around crawler traffic and give real requests room to complete. I added an Nginx gate in front of three app copies. It limits self-declared crawlers by bot family and in aggregate, returning 429 with Retry-After when they exceed the limit. Static assets are not limited, and ordinary browser and API requests do not get classified as crawlers by that rule.

I checked the behavior directly: a test bot was allowed through its burst and then received 429; Googlebot and curl completed their test requests normally. Each visitor is routed consistently to one app copy so its local page cache and API limits remain meaningful.

Measure people, not requests

The harder part was deciding which bots to let through. A large request count can look like interest when it is just automated fetching. I had to distinguish search crawlers from search referrals that brought an actual reader to the site.

One example changed the policy. Over ten hours, OAI-SearchBot fetched about 2,300 pages, around a third of the sitemap, and referrals tagged utm_source=chatgpt.com brought three readers. PerplexityBot fetched 84,000 pages in the measured period and brought no readers. I exempted the reader-facing ChatGPT fetchers from the gate and refused PerplexityBot. GPTBot and ClaudeBot were allowed under the crawl rate limit, since training access might help the site get named later, but that is a different outcome from sending a visitor now.

The visitor measure is intentionally conservative. I count visits referred by search engines such as Google, Bing, and DuckDuckGo. Counting asset loads or page requests as humans was misleading because a headless scraper can load a page exactly like a browser. At the time of the measurement, the reliable figure was about 47 Google referrals a day. It was much smaller than the raw request totals suggested.

What changed in the product

The incident exposed a product decision as much as an infrastructure one. A page being computable does not mean it should be linked from every other page, or submitted to a search engine.

• Can the input be resolved?
• Is the resulting page useful and substantial enough to index?
• Should crawlers follow links from it?
• Is it one of the pages I deliberately seed and stand behind?

The sitemap contains nominated pages, not the entire input space. Unseeded pages remain available when a person has the URL, but they do not open a crawl path into every possible value. The crawler gate handles request rates; it is not a substitute for keeping the URL space bounded.

What did not help

The main dead end was assuming caching would absorb crawler traffic. It helps when the same pages are requested repeatedly. Here, crawlers were asking for different URLs, so cache hit rates were almost zero.

Another tempting approach was to block broad classes of user agents. That would have been simpler, but it risks blocking the fetchers that actually bring people. The policy is based on measured referrals where possible, and rate limits where I still want to allow a crawler without letting it dominate the machine.

The other lesson is operational: the site shares a small home server with many other services. More app copies helped spread rendering across available CPU, but they do not make the host unlimited. The gate and crawl boundaries matter because they keep an unbounded input space from consuming the machine just because it can be computed.

Where things stand

Nerdulator serves three app copies behind the gate. Google and the reader-facing ChatGPT fetchers can crawl without the general bot limit. Other self-declared crawlers are limited, and some are refused based on observed behavior. The sitemap still describes the finite set of pages the site wants discovered.

This is not a claim that the crawler problem is permanently solved. Bot behavior changes, and the measurements need to be revisited. The useful outcome was a clearer model: publish a finite set of pages, keep arbitrary inputs available to people, and spend server capacity on traffic that can plausibly lead to a reader.