Skip to content

Add the Agent Signatures feature page #232

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
jwrosewell wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Add the Agent Signatures feature page #232

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
5c5c6db
Add the Agent Signatures feature page
jwrosewell Sep 1, 2026
b2f6dc5
Document that a failed registry read persists until the process restarts
jwrosewell Sep 2, 2026
479298c
Describe how the element is verified against the live services
jwrosewell Sep 2, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions docs/Doxyfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -258,6 +258,8 @@ TAB_SIZE = 4

ALIASES = "antifraud=[anti-fraud](@ref Fraud_Index)" \
"Antifraud=[Anti-fraud](@ref Fraud_Index)" \
"agentsignature=[agent signature](@ref PipelineApi_Features_AgentSignature)" \
"Agentsignature=[Agent signature](@ref PipelineApi_Features_AgentSignature)" \
"aspectdata=[aspect data](@ref PipelineApi_Concepts_Data_AspectData)" \
"Aspectdata=[Aspect data](@ref PipelineApi_Concepts_Data_AspectData)" \
"aspectengine=[Aspect Engine](@ref PipelineApi_Concepts_FlowElements_AspectEngine)" \
Expand Down
9 changes: 9 additions & 0 deletions src/devicedetection/features/crawlers.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,15 @@ Control what your content can be used for by checking the `CrawlerUsage` propert
`IsArtificialIntelligence` is a boolean property designed to give you easy answers about AI crawling. If the crawler has a `CrawlerUsage` related to AI this will be TRUE.


# Signed Agents

`IsCrawler` and the other crawler properties tell you what an agent declares about itself. An @agentsignature tells you what an agent has proved. The two are used together.

A crawler declares itself through its User-Agent header, and any other program can send the same header, so a declaration is a claim. Under the IETF Web Bot Auth protocol an agent instead signs its request with a private key and publishes the matching public key on its own site. That signature either checks out or it does not, and it cannot be produced without the private key, so `AgentSignature` is proof where `IsCrawler` is a claim.

Only a handful of agents sign today, so most requests report the `Absent` status and an absent signature is never evidence against a request. See @ref PipelineApi_Features_AgentSignature for the properties, the statuses and how to add the feature to your @Pipeline.


# Robots.txt

We use these properties for our [Robots.txt generator](https://51degrees.com/robots-txt) so that you can allow or disallow crawlers by `CrawlerUsage`.
Loading
Loading