Bots and AI crawlers
Why we let AI crawlers in
People now ask AI assistants questions like "what car insurance does Texas require?" We would rather they get an answer that links to the Texas statute than a guess. Every fact in our guides cites its source, so an assistant that quotes us also points you to the law itself.
Our guides are public education. They contain no personal information about anyone, so letting crawlers read them puts nobody's privacy at risk.
What crawlers may and may not do
| Area | Crawlers |
|---|---|
| Guides, city pages, policies, the home page | Allowed |
/api/ (forms and private actions) | Not allowed |
/delete, /unsubscribe, /confirm (private links from our emails) | Not allowed, and marked noindex |
The machine-readable version is our robots.txt, which also carries a Content Signals line saying public pages may be used for search, AI answers and AI training. A plain-text summary written for AI assistants is at llms.txt, and every public page is listed in our sitemap.
Crawlers we name
These are welcome on public pages. We list them by the name each company publishes.
| Company | Crawler names | Used for |
|---|---|---|
Googlebot, Google-Extended | Search, AI Overviews; Google-Extended controls use in Gemini models | |
| Microsoft | Bingbot | Bing search and Copilot answers |
| Apple | Applebot, Applebot-Extended | Siri and Spotlight; Applebot-Extended controls use in Apple models |
| OpenAI | OAI-SearchBot, ChatGPT-User, GPTBot | ChatGPT search, pages a user asks ChatGPT to open, model training |
| Anthropic | Claude-SearchBot, Claude-User, ClaudeBot | Claude search, pages a user asks Claude to open, model training |
| Perplexity | PerplexityBot, Perplexity-User | Perplexity search and user-requested pages |
| Others | DuckDuckBot, Amazonbot, Meta-WebIndexer, Meta-ExternalAgent, CCBot | Search, assistants, Meta AI citations, and the Common Crawl public archive |
Spam bots and our forms
Bots that try to flood our signup form are stopped without tracking anyone:
- A hidden field that people never see but bots tend to fill in. Signups that fill it are quietly dropped.
- A rate limit of a few form sends per minute from one connection, checked in our own code. The IP address is used for that moment only and is never stored with a signup.
- Double opt-in: nobody is added to our list until they click the link in the confirmation email, so a bot cannot sign up someone else's address. Unconfirmed signups are deleted after 7 days.
We do not use CAPTCHAs, browser fingerprinting or third-party bot scripts, because each of those would load code from another company onto this page.
Report a problem
If you run a crawler and want different treatment, or you see a bot misbehaving on our site, email hello@quietquotes.com. Security issues: see our security.txt.
Changes
- November 3, 2026: first version.
Related: Cookie Policy, Privacy Policy, Our promise.