Skip to content

Say which SDKs provide the robots.txt engine #234

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
jwrosewell wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Say which SDKs provide the robots.txt engine #234

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions src/devicedetection/features/robotstxt.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,25 @@ With the explosion of AI training crawlers, search bots, analytics agents, archi

The `RobotsTxtEngine` shipped in the [device-detection-dotnet](https://github.com/51Degrees/device-detection-dotnet/tree/main/FiftyOne.DeviceDetection.RobotsTxt) package solves this problem by generating a `robots.txt` document on demand from the up-to-date 51Degrees crawler dataset, driven by a small set of allow/disallow choices expressed in terms of [crawler usages](@ref DeviceDetection_Features_Crawlers) rather than individual user-agents.

# Availability

The on-premise `RobotsTxtEngine` is available in .NET only. It ships in
the [device-detection-dotnet](https://github.com/51Degrees/device-detection-dotnet/tree/main/FiftyOne.DeviceDetection.RobotsTxt)
package, and there is no equivalent engine in the Java, C, C++, Go,
Rust, PHP, Node or Python SDKs.

Robots.txt generation is available from every language through
51Degrees Cloud, which is an HTTP call and needs no SDK at all. See
[Cloud Usage](#cloud-usage) below. There is also a hosted generator at
[51degrees.com/robots-txt](https://51degrees.com/robots-txt) that
produces a file to host, with no code involved.

| How you use it | Available in |
| --- | --- |
| On-premise engine | .NET only |
| 51Degrees Cloud | Any language, over HTTP |
| Hosted generator | No code required |

# Output Properties

The engine produces two synthetic properties — they are not stored in any data file, they are computed at request time from the crawler dataset and the evidence supplied to the engine.
Expand Down
Loading