Free Tool

Web Crawler Simulator

Test how 20+ search engine and AI crawlers see your website. Verify robots.txt rules, check content visibility, and optimize for better indexing.

Test any crawler

Select a crawler and enter your URL to see what the bot sees.

AI Assistants & LLMs

Crawlers used by AI chatbots and large language models to fetch web content for answers and training.

ChatGPT Fetch

User agent used by ChatGPT to fetch pages for AI answers and...

Claude Fetch

User agent used by Claude web to fetch pages for AI answers ...

Grok Search

User agent used by Grok to fetch pages for AI answers and pr...

PerplexityBot

Perplexity's crawler user agent used to answer questions and...

CohereAI

Cohere's primary crawler user agent in their AI services/too...

AI2Bot

Allen Institute's AI2Bot crawler, used for research corpora ...

Search Engines

Traditional search engine crawlers that index your content for search results.

Googlebot

Primary user agent used by Google's indexing bot, Gemini, AI...

Bingbot

User agent used by Bing's indexing bot, Microsoft Copilot an...

DuckDuckBot

DuckDuckGo's primary indexing bot user agent....

Bravebot

Brave's primary crawler user agent for their search engine....

YandexBot

Yandex's primary indexing bot user agent....

YouBot

You.com's bot used for search and preview generation, fetchi...

Social Media

Crawlers that fetch Open Graph data for link previews on social platforms.

Facebook Crawler

Fetches pages to render Open Graph previews on Facebook, Ins...

LinkedInBot

Fetches pages to render Open Graph previews on LinkedIn....

Bytespider

ByteDance's web crawler for AI model training and content di...

Voice Assistants & Enterprise

Crawlers for voice search, smart speakers, and enterprise applications.

Applebot

Apple's crawler used for Siri and Spotlight suggestions....

Amazonbot

Amazon's crawler used for product and content discovery....

Diffbot

Diffbot's primary crawler user agent, used for web scraping ...

Data & Research

Crawlers used for web archiving, research datasets, and AI training corpora.

CCBot

Open web-scale archive crawler used to build public datasets...

Omgili

Aggregated discussion indexer bot....

TimpiBot

Independent web indexer bot....

All Supported Crawler User Agents

Crawler	User-Agent	Purpose	Simulate
ChatGPT Fetch	`Mozilla/5.0 (compatible; ChatGPT-User/1....`	User agent used by ChatGPT to fetch pages for AI a...	Test
Claude Fetch	`Claude-Web/1.0...`	User agent used by Claude web to fetch pages for A...	Test
Grok Search	`Grok-Search/1.0...`	User agent used by Grok to fetch pages for AI answ...	Test
PerplexityBot	`PerplexityBot/1.0 (https://www.perplexit...`	Perplexity's crawler user agent used to answer que...	Test
Googlebot	`Mozilla/5.0 (compatible; Googlebot/2.1; ...`	Primary user agent used by Google's indexing bot, ...	Test
Bingbot	`Mozilla/5.0 (compatible; bingbot/2.0; +h...`	User agent used by Bing's indexing bot, Microsoft ...	Test
Facebook Crawler	`facebookexternalhit/1.1 (+http://www.fac...`	Fetches pages to render Open Graph previews on Fac...	Test
LinkedInBot	`LinkedInBot/1.0 (+http://www.linkedin.co...`	Fetches pages to render Open Graph previews on Lin...	Test
Amazonbot	`Amazonbot/1.0 (+https://developer.amazon...`	Amazon's crawler used for product and content disc...	Test
Applebot	`Mozilla/5.0 (compatible; Applebot/0.1; +...`	Apple's crawler used for Siri and Spotlight sugges...	Test
Bytespider	`Bytespider; (+http://www.bytedance.com/s...`	ByteDance's web crawler for AI model training and ...	Test
DuckDuckBot	`DuckDuckBot/1.0; (+http://duckduckgo.com...`	DuckDuckGo's primary indexing bot user agent....	Test
CohereAI	`CohereAI/1.0...`	Cohere's primary crawler user agent in their AI se...	Test
AI2Bot	`AI2Bot/1.0 (+http://www.allenai.org)...`	Allen Institute's AI2Bot crawler, used for researc...	Test
CCBot	`CCBot/2.0 (+https://commoncrawl.org/faq/...`	Open web-scale archive crawler used to build publi...	Test
Diffbot	`Diffbot/1.0 (+http://www.diffbot.com)...`	Diffbot's primary crawler user agent, used for web...	Test
Omgili	`omgili...`	Aggregated discussion indexer bot....	Test
TimpiBot	`TimpiBot/1.0...`	Independent web indexer bot....	Test
YouBot	`YouBot/1.0 (+https://you.com/bot)...`	You.com's bot used for search and preview generati...	Test
Bravebot	`Bravebot/1.0...`	Brave's primary crawler user agent for their searc...	Test
YandexBot	`Mozilla/5.0 (compatible; YandexBot/3.0; ...`	Yandex's primary indexing bot user agent....	Test