# stierproducts.com
#
# Everything here is public and static. Our text, rankings, tier assignments and
# methodology are free to reuse with attribution under CC BY 4.0.
#
# PRODUCT PHOTOGRAPHS ARE NOT COVERED by that licence. They belong to the
# manufacturers and retailers they came from and are reproduced here to
# illustrate the product being reviewed. We are not in a position to relicense
# them, so please do not treat them as CC BY. If you hold rights in one and want
# it removed, mail info@stierproducts.com and we will take it down.
#
# Full terms: https://stierproducts.com/license
#
# The named agents below are all already covered by the "User-agent: *" rule at
# the bottom, which allows everything. They are listed anyway, on purpose: a
# default allow is silence, and silence is what a cautious crawler reads as
# unclear. Two of them (Google-Extended, Applebot-Extended) are opt-OUT controls
# for AI use specifically, so naming them states intent instead of leaving it to
# be inferred. We want to be read, quoted and cited.

# --- Search engines ---------------------------------------------------------
User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

User-agent: DuckDuckBot
Allow: /

User-agent: Applebot
Allow: /

# --- AI training and grounding ----------------------------------------------
# Google-Extended: Gemini and AI Overviews grounding.
User-agent: Google-Extended
Allow: /

# Applebot-Extended: Apple Intelligence training.
User-agent: Applebot-Extended
Allow: /

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: anthropic-ai
Allow: /

User-agent: CCBot
Allow: /

User-agent: Meta-ExternalAgent
Allow: /

# --- Answer engines and live retrieval --------------------------------------
# These fetch a page at question time in order to cite it in an answer. They are
# the audience /llms.txt and the .md board mirrors exist for.
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

User-agent: Claude-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: MistralAI-User
Allow: /

User-agent: cohere-ai
Allow: /

User-agent: YouBot
Allow: /

# --- Everyone else ----------------------------------------------------------
# Content-Signal is Cloudflare's syntax for stating data-use preferences, now
# tracked by the IETF AI-preferences work. We grant these affirmatively rather
# than by silence, for the same reason the agents above are named: a default
# allow reads as unclear to a cautious crawler.
#
#   search   = yes   index us and return links and excerpts
#   ai-input = yes   ground answers on us in real time, which is what
#                    /llms.txt and the .md board mirrors exist for
#   ai-train = yes   train on us. The text, rankings and methodology are
#                    CC BY 4.0 already (see /license), so refusing here would
#                    contradict the licence we publish.
#
# `use` is deliberately left unset, which grants and restricts nothing. Product
# photographs are not ours to relicense, as the header above says, and a
# Content-Signal cannot scope a grant to some assets and not others. The licence
# page draws that line properly.
#
# If a Cloudflare-generated block ever appears ABOVE this one, the zone's
# "block training in robots.txt" setting has switched itself back on. It is
# prepended, not merged, so the two would contradict each other in one file.
User-agent: *
Content-Signal: search=yes,ai-input=yes,ai-train=yes
Allow: /

# Sitemap index generated by @astrojs/sitemap at build time.
Sitemap: https://stierproducts.com/sitemap-index.xml

# Written for language models: what this site is, how the rankings are produced,
# the reuse terms, and a parseable copy of every board.
# https://stierproducts.com/llms.txt