xAI Publishes Details on Its Grok Web Crawler
xAI has launched a dedicated page describing "Grokbot," the automated crawler it uses to gather web content for training and grounding its Grok AI models. The page lists the bot's user-agent string and IP ranges, and explains how webmasters can allow or disallow it via robots.txt, mirroring the transparency approach other AI labs like OpenAI and Google have taken with their own crawlers.
The move comes as scrutiny over AI companies scraping the open web intensifies, with many publishers and site owners increasingly blocking or rate-limiting bots they don't want feeding commercial AI products. By formalizing its crawler's identity, xAI is giving site operators an easier way to manage that access rather than relying on generic user-agent detection or IP blocking.
The Hacker News discussion focused heavily on how aggressively the bot crawls, whether it respects robots.txt in practice, and comparisons to other AI scrapers' behavior on smaller sites and forums.