T139
robots.txt contains a blanket
Disallow: / for GooglebotUser-agent: Googlebot
Disallow: /
3,410
clicks at risk
DB_HOST, DB_USER and DB_PASS in .env.Full crawl · 5,000 URLs · 62 rendered · completed 17 Aug 2026, 09:14 GST
severity × log(1 + affected clicks) × confidence ÷ fix effort
Disallow: / for Googlebotar-AE → en-AE)Google-Extended is blocked — this does not suppress AI Overviewsnosnippet, data-nosnippet, max-snippet and noindex.robots.txt verdict cross-checked with a live user-agent probe — a WAF block is invisible in robots.txt
| Crawler | / | /products/ | /heat-transfer/ | /blog/ | /downloads/ | /ar/ |
|---|---|---|---|---|---|---|
| OAI-SearchBot retrieval | ✓ | ✓ | ✓ | ✕ | ✕ | ✓ |
| PerplexityBot retrieval | ✓ | ✕ | ✓ | ✕ | ✕ | ✓ |
| Claude-SearchBot retrieval | ✓ | ✓ | ✓ | ✓ | 402 | ✓ |
| Googlebot gates AIO + AI Mode | ✕ | ✕ | ✕ | ✕ | ✕ | ✕ |
| GPTBot training — policy choice | ◐ | ◐ | ◐ | ◐ | ◐ | ◐ |
| Bytespider ignores robots.txt | ? | ? | ? | ? | ? | ? |
Volume shown three ways because GCC industrial volumes are thin: country · GCC aggregate · global English
| Intent | AE | GCC | Global EN | CPC | KD | Rank | SERP | ||
|---|---|---|---|---|---|---|---|---|---|
| plate heat exchanger supplier dubai | Transactional | 40 | 210 | 1,900 | $3.10 | 18 | 6 | Local pack | 0.86 |
| shell and tube heat exchanger uae | Commercial | 20 | 140 | 2,400 | $2.75 | 24 | 14 | Directory 60% | 0.74 |
| air cooled vs water cooled chiller | Informational | null | 90 | 8,100 | $1.20 | 31 | — | AI Overview −58% | 0.51 |
| مورد مبادلات حرارية الإمارات | Commercial | 30 | 170 | — | $2.40 | 12 | — | No AIO | 0.68 |
null means unknown — kept, never treated as zero.Connect a backlink provider in Integrations to populate referring domains, anchor distribution and broken inbound links.