---
title: "Cloudflare's AI Training \"Disallow\" policy returns 403 at the edge while robots.txt still says Allow"
url: "https://www.dariuszparys.com/notes/cloudflare-ai-training-policy-blocks-crawlers"
author: "Dariusz Parys"
published: 2026-11-11T00:00:00.000Z
description: "AI crawlers got 403 on every path of a Cloudflare site whose robots.txt allowed them. The block came from the zone's AI Training policy, not from the settings that read as bot protection."
---

# Cloudflare's AI Training "Disallow" policy returns 403 at the edge while robots.txt still says Allow

## Context

A static site on Cloudflare's Free plan published `llms.txt` and Markdown copies of its posts for agents. Its `robots.txt` said `Allow: /`. Requests with GPTBot, ClaudeBot, CCBot, Amazonbot or PerplexityBot user agents still got `403 Your request was blocked` on every path except `robots.txt`.

## Insight

The zone's `bot_management` object showed `ai_bots_protection: disabled` and `bot_preference_sync_enabled: false`, which reads as "not blocking". The setting that blocked was `ai_training: "disallow"`, enforced by a managed rule named "Block AI training crawlers for Bot Preference Sync disallow".

In the dashboard this is the domain's **Security → Settings → Bot traffic → AI Crawler and Agent access policies**, row **Training**. Setting it to "Allow (do not block)" removed the 403s.

Bot Fight Mode is a second, separate source of challenges. On the Free plan it cannot be exempted per bot; it is only on or off.

## Diagnosis

1. Request the same URL with several bot user agents and compare status codes:

   ```sh
   curl -s -o /dev/null -w '%{http_code}\n' -A '<bot UA>' https://example.com/llms.txt
   ```

2. Name the rule that fired with a GraphQL query on `firewallEventsAdaptive`, grouped by `description` and `userAgent`.

## Did any agent fetch the Markdown files?

The same GraphQL API answers this without extra logging: `httpRequestsAdaptiveGroups`, filtered on `clientRequestPath` and `verifiedBotCategory`. It works on the Free plan with 31 days of retention.

Count only rows with a `verifiedBotCategory`. Most unverified requests carrying ChatGPT-User or Claude-User user agents were scanners probing for `.env` files.
