Grok Crawler Simulator
Test how xAI's Grok AI crawler sees your website. Ensure your content is accessible for Grok's AI-powered search and answers.
Test your site with Grok Search
Enter your URL to see how Grok Search views your website.
What is Grok Search?
Grok is xAI's AI assistant, integrated into X (formerly Twitter). Grok uses web crawlers to access content beyond X's platform, allowing it to answer questions with real-time web information. The Grok-Search and related crawlers fetch web content to provide comprehensive AI answers to X users.
Why Allow Grok Search?
Grok Search robots.txt Configuration
Control how Grok Search accesses your website using robots.txt directives. Add these rules to your robots.txt file at the root of your domain.
Allow Grok Search
# Allow Grok/xAI crawlers User-agent: Grok-Search Allow: / User-agent: xAI-Grok Allow: / User-agent: Grok-DeepSearch Allow: /
Block Grok Search
# Block Grok/xAI crawlers User-agent: Grok-Search Disallow: / User-agent: xAI-Grok Disallow: / User-agent: Grok-DeepSearch Disallow: /
User-Agent String: Grok-Search/1.0
Grok Search FAQ
Related Crawler Simulators
Make Your Site Crawlable
JavaScript websites often have indexing issues. LovableHTML pre-renders your SPA into crawler-friendly HTML so Grok Search and other bots can read your content.
