OdenBot

1 What OdenBot does

OdenBot fetches publicly reachable web pages for ODEN, a web search and synthesis API. It reads the page, extracts the readable text, and stores it so ODEN can answer a question with a citation pointing back to your page.

Two separate components send requests with an OdenBot user agent:

Both honor the rules described below. Blocking OdenBot in robots.txt stops both.

2 How to identify it

Every request carries an OdenBot user agent containing a link to this page. We do not disguise our traffic: OdenBot never impersonates a browser, never rotates user agents to avoid detection, and never attempts to work around bot protection.

3 Rules it follows

• robots.txt is fetched per host, cached for 24 hours, and obeyed. A matching Disallow means we do not fetch the page.
• Crawl-delay is respected when set.
• TDM opt-out is respected. We check machine-readable rights reservations under EU DSM Article 4: the tdm-reservation header, X-Robots-Tag: noai / notdm / noimageai, and the equivalent <meta> and <link rel="tdm-reservation"> signals in the page itself.
• Blocking AI crawlers counts as an opt-out. If your robots.txt disallows known AI crawlers such as GPTBot or ClaudeBot, we read that as a statement about AI use and stay out too, even if you never named OdenBot.
• Login walls and paywalls are never passed. OdenBot sends no cookies, holds no accounts, and does not try to reach content behind authentication.
• Conditional requests. We send If-None-Match and If-Modified-Since, so an unchanged page costs you a 304 instead of a full response.
• Backoff. A 429 or 503 triggers exponential backoff and we honor Retry-After. A 401, 403 or 451 pauses your entire host for 24 hours.
• Pages that profile individuals are skipped rather than stored.

4 How to block it

Add this to robots.txt at the root of your host. A rule naming OdenBot takes precedence over your * rules.

User-agent: OdenBot
Disallow: /

To allow most of the site but keep OdenBot out of one area:

User-agent: OdenBot
Disallow: /members/
Disallow: /internal/

To slow it down instead of blocking it:

User-agent: OdenBot
Crawl-delay: 10

You can also reserve text and data mining rights without touching robots.txt, by returning a header on your pages:

X-Robots-Tag: noai
tdm-reservation: 1

Changes take effect within 24 hours, which is how long we cache robots.txt. If you need a change to apply immediately, contact us and we will drop the cached copy.

5 Request rate

OdenBot makes at most one request per second to a single host, and never runs requests in parallel against the same host. If your robots.txt sets a longer Crawl-delay, that value is used instead.

We fetch URLs from your sitemap rather than following links indiscriminately, so the number of requests is bounded by what you publish there.

6 Removal requests

If we have stored a page you want removed, email [email protected] with the URL. We delete the stored copy and add the path to an exclusion list so it is not fetched again. You do not need to explain why.

Copyright notices and TDM reservation notices are handled the same way and take priority over everything else in our queue.