An MCP tool for checking the HTML a crawler receives

3 days ago by jonas_tmb to c/aicoding

A coding assistant can explain SEO and still miss what the server actually sends back. The useful starting point is the response, before interpreting why a page might be hard to discover.

I’m the founder of Prerender Buddy. Our open-source MCP exposes public crawler diagnostics to an assistant: returned HTML, metadata, readable-text counts and discovery-file checks. Those public tools run without a PB account. The source and setup are here: https://github.com/kopachlager/prerenderbuddy-mcp

An example question to try on a site you control: “Compare the standard and GPTBot HTTP responses for my pricing page, and show the evidence behind any differences.”

There’s an important limit: both requests are HTTP fetches. Neither executes JavaScript, and changing a user agent does not reproduce a verified crawler’s network origin. A difference is something to inspect, not proof of cloaking or lost rankings.

For ongoing monitoring and account evidence, the separate PB platform is at https://prerenderbuddy.com/. The public diagnostics remain usable on their own.

What would make these findings useful in your development workflow: CI failures, a short evidence summary, or something else?

aicoding
aicoding

@programming.dev

login for more options
132
297
152

A community to discuss agentic and AI assisted coding.

All users are expected to follow the code of conduct, particularly in regards to voting:

1.3. Vote for quality: When downvoting content in communities the downvotes should reflect the quality of the content rather than whether you personally agree with it.

go to feed...