About / QuarryBot

QuarryBot, the OppIntell web crawler

QuarryBot fetches publicly available web pages, starting with official election and government sites, to build the search index behind OppIntell’s public-records research. If you found it in your server logs, this page explains what it is, how it behaves, and how to slow it down or keep it out.

Obeys robots.txt1 request per second per site, at mostOne published addressNo CAPTCHA or bot-protection evasion

How to recognize it

Every QuarryBot request carries exactly this User-Agent header:

QuarryBot/1.0 (+https://oppintell.com/quarrybot)

Its robots.txt token is QuarryBot. QuarryBot does not change or disguise its user agent, rotate addresses, or route requests through proxies.

The address it crawls from

Address pending allocation. QuarryBot will send every request from one dedicated, static IPv4 address, and that address will be published here before QuarryBot makes its first request. Until it appears on this page, QuarryBot is not crawling, and any request that claims to be QuarryBot is not ours.

How it behaves

  • It reads robots.txt before crawling a site and checks it again for every redirect. If robots.txt cannot be fetched (a server error, a timeout, or a 401, 403 or 429 response), QuarryBot treats the whole site as off limits.
  • It waits at least 1 second between requests to the same host, and longer when your robots.txt sets a Crawl-delay (honored up to 30 seconds).
  • It keeps a daily request limit for every site it visits.
  • When it revisits a page it sends If-None-Match and If-Modified-Since (when it has them), so an unchanged page costs you a 304.
  • It does not log in, submit forms, keep cookies or run JavaScript, and it skips very large files.

When a site says no

QuarryBot never tries to get past bot protection. It does not solve CAPTCHAs, replay clearance cookies, imitate a browser or switch to another address or client. A challenge page or a 401, 403 or 429 response is treated as a refusal: QuarryBot stops, does not retry that request, and backs off the whole site for an increasing period.

Pages marked noindex are not added to the index as text. Pages marked noai or carrying a text-and-data-mining reservation can still appear as search results, but their content is kept out of AI-generated answers.

Blocking or slowing QuarryBot with robots.txt

Add a group for QuarryBot to the robots.txt at the root of your site. To keep it out entirely:

User-agent: QuarryBot
Disallow: /

To keep it out of part of your site:

User-agent: QuarryBot
Disallow: /private/

To slow it down (here, to one request every 10 seconds):

User-agent: QuarryBot
Crawl-delay: 10

QuarryBot reuses a robots.txt it has already fetched for up to 24 hours, so a change takes effect within a day. Rules written for all crawlers (User-agent: *) apply to QuarryBot too when your file has no QuarryBot group.

Opting out or reporting a problem

To have your site removed from QuarryBot’s crawl list, or to report a request that looks wrong, email [email protected] or use the contact form. Include your site’s domain and, if you can, the time and path of the requests in question. A site we remove stays excluded even if its robots.txt changes later.

Research methodology

How OppIntell sources, validates and publishes research from public records.

Contact OppIntell

Questions about QuarryBot, opt-out requests and anything else.