# AI Crawler Policy Notes

Use this as a policy checklist before changing `robots.txt` or `llms.txt`.

## Separate the crawler decisions

Do not use one blanket "allow AI" or "block AI" decision. Separate at least:

- Search/discovery crawlers that may surface current product information.
- Training crawlers that may use content to improve models.
- User-triggered browsing agents that fetch pages on behalf of a shopper.
- Ads/commercial crawlers if relevant.

## Practical checklist

- Confirm current `robots.txt` is intentional and documented.
- Keep sitemap directives present and current.
- Avoid blocking product/category pages needed for merchant discovery unless there is a deliberate policy reason.
- Ensure `llms.txt` does not contradict `robots.txt`.
- Document policy owners and review cadence.

## Acceptance checks

- `/robots.txt` returns HTTP 200.
- `/llms.txt` returns HTTP 200 if adopted.
- Sitemap URLs referenced in robots resolve directly.
- Crawler policy is reviewed by SEO/platform/legal as appropriate.
