// HACKER NEWS — CYBERSECURITY
Someone is running mass vulnerability scans, spoofing AI bots like ClaudeBot
The internet is rapidly evolving from an environment built primarily for humans, into one increasingly used by machines. See how AI agents, crawlers, scrapers, and other bots are reshaping the way information is discovered, accessed, and used across the web.
Key ecosystem metrics across 5,000+ websites using Agent Analytics and AI Chat Referral Tracking.
Hover over each agent type for more information about what they do
These bots scrape website content to train AI models. Some belong to AI companies, while others belong to third-party services that resell the data. Automatic Robots.txt can block unwanted scraping. Included agent types include AI Data Providers and AI Data Scrapers.
These bots fetch website content in real time to power AI assistants, coding agents, and other retrieval-augmented generation (RAG) tasks. Pages inform responses on the spot, such as when an assistant summarizes an article or a coding agent references documentation. Included agent types include AI Assistants and AI Coding Agents.
These bots crawl website content so it can be surfaced in AI search engines and AI-generated answers. Those answers often include citations or links back to the source pages. Included agent types include AI Search Crawlers.
AI search indexing activity by agent type over time
These bots use browsers to autonomously navigate websites, click through pages, and make decisions to complete tasks for people. Agentic UX best practices and Google PageSpeed Insights help evaluate how well websites support them. Included agent types include AI Agents.
See which robots.txt rules are set across the web and how well agents follow them. An agent's Robots.txt Effectiveness measures the effectiveness of a disallow rule for it by estimating how much the agent reduces its traffic after it's blocked.
Agents with the best Robots.txt Effectiveness percentages