Conversation
This was created with the Cloudflare AI skill for robots.txt files. See this AI readiness evaluation of the guide for more: https://isitagentready.com/bitcoin.design?profile=content
Per the Cloudflare AI readiness evaluation: https://isitagentready.com/bitcoin.design?profile=content It took the search.json file that indexes the site content as an API catalog to serve to robots. Let's try this out. My suspicions is that the file may be too big to be ingested easily.
Allows agents to request markdown instead of HTML.
All set to yes, so AI can freely use the content for training, search, and input.
Added content signal to netlify.
✅ Deploy Preview for bitcoin-design-site ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
There was a problem hiding this comment.
Pull request overview
This PR aims to improve “agent readiness” for bitcoin.design by adding standard discovery/config signals for crawlers/agents and enabling Markdown content negotiation.
Changes:
- Adds a
robots.txtwith allow/deny rules, sitemap pointer, and an AI-related content signal. - Adds Netlify response headers (
Content-Signal, discoveryLinkheaders) and registers an Edge Function. - Introduces
.well-knowndiscovery resources (service-desc.jsonand anapi-cataloglinkset), plus an Edge Function that converts HTML to Markdown when requested.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| robots.txt | Adds crawler rules, sitemap reference, and a content-signal directive. |
| netlify/edge-functions/markdown-negotiation.ts | Adds an Edge Function to return Markdown when Accept: text/markdown is present. |
| netlify.toml | Configures headers (Content-Signal, Link discovery, api-catalog Content-Type) and enables the Edge Function for all paths. |
| .well-known/service-desc.json | Adds a service descriptor for machine clients. |
| .well-known/api-catalog | Adds an API catalog linkset document for discovery. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
When run on Netlify preview, the function to serve the markdown content threw an error because it couldn't find a DOM parsing library it expected to exist.
…/Guide into feature/robots-txt
The deployed Edge function still crashes. Probably related to this dependency.
|
I’m not entirely sure about this, but I am just thinking out loud about whether an MCP server could make sense for us in the long run.Right now, it feels like overkill since we don’t have enough dynamic data or actionable workflows, most of what we have is still static content.However, with the recent SOB project, if we start structuring things as skill files and design code samples, this could evolve into something more interesting. In that setup, an LLM could take a task along with a reference link to the website , retrieve the relevant design knowledge, and reason over it,which is fairly standard (and something that already happens ).Where this could become more powerful is if we also expose functions that generate components (like buttons, headers, layouts) in a consistent way. |
|
@Aditya-Sarna that seems like a better discussion for the Discord channel than this PR. This PR only focuses on some basic instructions for web-scraping by agents. MCP, skills, etc are a totally different story. If you look at the options the Cloudflare tester provides, you can see how they split that up. 
|
|
@GBKS Sure ! I will move the conversation to discord . |
|
nice one christoph! thanks for doing that. |
sbddesign
left a comment
sbddesign
left a comment
There was a problem hiding this comment.
tl;dr - Approved because it's an improvement over what was there before. It satisfies the original problem of "make this Cloudflare dial become green instead of red", and my OpenClaw says it's better.
Future improvements
- Update the
markdown-negotiation.tsfile to deliver cleaner output. Runcurl -H 'Accept: text/markdown' https://deploy-preview-1228--bitcoin-design-site.netlify.appand you will see exactly what an agent sees when it visits now. Has a bunch of messy stuff like header nav stuff that's irrelevant for the agent. Could also provide better overall description of "this is what you are looking at and why it might be useful for you or the user". - llms.txt is a nice convention to have, but not strictly required
How I tested
I went back and forth with BOLTy, my OpenClaw at ATL BitLab. I ask for his blunt assessment of his experience using the live site vs. the deploy preview.
BOLTy's Review
Blunt truth: the updated site is noticeably better for me as an agent.
The biggest improvement is that the updated version exposes a much richer, more machine-readable content map. I can immediately see site hierarchy, guide sections, deep links, and content organization without guessing.
Why the updated version feels better
1. Much better navigational clarity
The updated version exposes a large portion of the site's structure directly in the extracted output.
That makes it easier for an agent to understand:
- what content exists
- how it is organized
- where to go next
2. Better deep-link discoverability
The preview exposes many specific links into the guide, including detailed topics like:
- privacy
- coin selection
- silent payments
- sign in with bitcoin
- custom spending conditions
- wallet selector
- accessibility
That is much better than just seeing a short homepage summary.
3. Absolute links are helpful
The preview returns many fully-qualified URLs, which is useful for tooling and agent navigation.
4. More content, less guessing
The original homepage mostly came through as concise human-facing marketing copy.
The updated preview came through as something closer to a combined homepage + sitemap + guide index. For an agent, that is usually more useful.
Important caveat
Part of the improvement may be due to how the preview is rendered to fetch tools, not only because the site itself is inherently better.
The original page was extracted more like a traditional readability view.
The preview came back as markdown with much more visible structure.
So the real conclusion is:
- Yes, the updated version is more agent-friendly
- but some of that may come from how the content pipeline exposes structure to extraction tools
What looked noisy or weak in the updated extraction
These items stood out in the extracted representation:
LinkSearchMenuExpandDocumentVideo- large blank spacing / repeated UI artifacts
Header design by Author
This feedback is specifically about the processed output from the fetch/extraction tool, not a claim about what a human sees visually in the browser.
Clarification on that point
I was not judging those items from a raw HTML source dump.
I was judging them from the processed output of OpenClaw's web_fetch tool, which returns a readable markdown/text extraction of the page.
So when I say those labels looked noisy, I mean:
- they appeared prominently in the extracted representation an agent would consume
- they may be UI chrome leaking into the content layer
- they can dilute the signal for agent-style reading
Honest overall assessment
The updated version is a real improvement for agents, mainly because it gives me:
- better structure
- better discoverability
- better direct access to deep content
But it is not perfectly clean yet.
The main remaining issue is that the extracted output still mixes valuable content with interface scaffolding and repeated navigation noise.
Recommendations
If the goal is to make the site genuinely more agent-friendly, these would help:
- Add a dedicated
/llms.txt - Provide an explicit agent-oriented site index or content map
- Keep core content prioritized above nav/UI chrome in extracted output
- Reduce repeated interface labels that appear in text extraction
- Replace weak placeholders like
Authorwith meaningful metadata - Consider whether the full nav tree needs to appear so prominently in page extraction
|
Thanks for the thorough review, Stephen and BOLTy. Seems like we could do with a small follow-up PR to tweak the HTML-to-markdown parsing. Basically, agents can request a markdown version of each page. In that case, a small script runs to take the HTML and convert it to markdown. And it creates slightly messy output still. Easiest is probably to take some of that output and ask AI to improve it. |
|
Yeah that's what I had in mind. A good next PR might be: clean up the markdown output to remove unneeded information such as navigational elements. A PR after that might be: make each page more relevant. For example, when an agent lands on the homepage, it mostly sees walls of links. It might be better to surface content at the top of the document like: "This is a collection of UX guidance for bitcoin applications, curated by a community of people interested in bitcoin and UX. It includes breakdown of bitcoin product designs for different use cases that are open sourced for your use, such as a daily spending wallet and a multi-sig vault. There are also articles describing different technologies within bitcoin from a UX perspective..." Or whatever you want the agent to hone in on. Similar to how every page has an SEO meta description, each page might have an agent guidance blurb in the content files that gets injected into the markdown. |
|
Didn't Google always penalize serving different content to different types of visitors? |
Maybe? But this is a little bit different. If I make an HTTP request to your web server, the person in the U.S. and the person in Canada should get the same content. But this is a case where one client is requesting something in HTML and the other one is requesting something in Markdown. So that implies to me that the client wants slightly different content. What I'm talking about isn't vastly different content. I'm just talking about having text that's more helpful and contextual. But perhaps what I'm talking about should live in an On a related note, I think the whole field of SEO is kind of in disarray right now. Digital marketers that are trying to grow brands are faced with the challenge that website traffic that used to be coming from Google five years ago is now coming predominantly from platforms like Claude and ChatGPT. Unlike with SEO, where there were a lot of commonly understood patterns, the LLMs seem to be kind of a black box. Armies of digital marketers are performing rain dances, giving burnt offerings, and chanting arcane incantations that they hope will appease the LLMs into promoting their content. |
|
@sbddesign I created #1229 to improve the markdown. How does BOLTy like it? |
Cloudflare posted a tool to evaluate if your site is set up so AI agents can interact with it smoothly. This requires various instruction files/tags/header/settings that agents look for. I can bitcoin.design and it performs pretty poorly with a 17/100.
Good thing Cloudflare also created a bunch of skills to fix things, along with prompts to use them. So I just went into Visual Studio Code and had Copilot work on those. For example:
I ran five of these, each in a separate commit:
Those are the ones the test pointed out that should be done. I am sure this is just a first step and that there will be more detailed feedback once the changes are live. We should test these changes both in the Cloudflare tool again, but also reference the guide in our own project-related prompts to see if the agents can access and use the content with ease.
I am not a pro on these agent settings, so I'd appreciate reviews. Let's start with having Copilot evaluate this PR and see what it comes up with.