Crawler-Verzeichnis

The Leipzig Corpora Collection (LCC) is a project of the Natural Language Processing Group of the University of Leipzig. The LCC offers access to monolingual dictionaries in more than 200 languages.

Paqlebot indexes news articles to discover who's mentioned in the media.

PMGCrawler is the Presse-Monitor crawler used to automatically fetch, index, and analyze publisher content for media monitoring and press-review services.

BunnyOptimizer-v01 is a custom crawler/automation client that performs high-volume URL fetching and optimization checks, typically used for automated content retrieval, performance profiling, or bulk scraping across websites.

The Meta Web Indexer crawler navigates the internet to improve the quality of Meta AI search results for users. Meta analyzes online content to improve the relevance and accuracy of Meta AI. By allowing Meta WebIndexer in your robots.txt file, you help us cite and link to your content in Meta AI's responses.

Moreover Bot is a crawler developed by Moreover Technologies, Inc., built for the specific purpose of processing RSS feeds rather than full web crawling. It identifies itself via the user-agent string as an RSS-fed bot and is likely utilized for aggregating or indexing distributed news content.

Gemini-Deep-Research is the agent responsible for collecting and scanning resources used in Google Gemini's Deep Research feature, which acts as a personal research assistant.

bigsur.ai is a web crawler operated by Big Sur AI that fetches website content to enable AI-powered web agents, sales assistants, and content marketing solutions for businesses.

anthropic-ai is a unconfirmed agent possibly used by Anthropic to download training data for its LLMs (Large Language Models) that power AI products like Claude.

Claude-Web is an AI-related agent operated by Anthropic. It's currently unclear exactly what it's used for, since there's no official documentation. If you can provide more detail, please contact us.

Claude-SearchBot navigates the web to improve search result quality for users. It analyzes online content specifically to enhance the relevance and accuracy of search responses.

Claude-User is dispatched by Anthropic's Claude AI assistant in response to user prompts, when it needs to fetch content to include in its answers.

ClaudeBot is Anthropic's web crawler that collects data for training Claude AI models. It respects robots.txt directives.

SentiBot is a sentiment analysis crawler that monitors web content for opinion mining and brand sentiment tracking.

Yokoy is a spend management SAAS solution. Webhooks generate requests to book expenses or invoices to customer's ERP system whenever the processing and approval process has been completed in Yokoy.

We use Project Honeypot for IP info.

Missinglettr Bot analyzes blog content to automatically create social media campaigns, extracting quotes, images, and key points for scheduling.

Macrobondbot implements custom web crawlers to fetch macroeconomic and financial data from 2,400+ official data sources for Macrobond's integrated research platform.

Over 50 active Cloudflare users are using Integromat to automate their workflow.

Cloudflare Digicert DCV service.

Buttondown RSS-Feed-Parser is a feed reader that fetches and processes RSS, Atom, and other content feeds.

A web crawler by Agency Analytics that allows their clients to check their own sites for SEO

Splunk Attack Analyzer (formerly known as TwinWave), visits URLs submitted by customers using a headless Chrome browser. DOM (Document Object Model), HAR (HTTP Archive), and other relevant data from these visits are analyzed to determine if the page is hosting malicious content.

Snipcart is an e-commerce solution for developers.

Readable is a website analysis crawler that scans customers' websites to extract and analyze text content for readability, clarity, and plain language compliance, providing comprehensive content optimization insights.

The Orlo Link Preview bot is used by the Orlo social media management platform. It fetches previews of links that are scheduled to be published in social media posts.

LinksIndexerBot crawls websites to help ensure backlinks are discovered and indexed by search engines, improving SEO visibility through automated link submission.

HelloWork is a French job search platform crawler that indexes job postings from company career pages.

Amazon Kendra is a highly accurate intelligent search service that enables your users to search unstructured data using natural language. It returns specific answers to questions, giving users an experience that's close to interacting with a human expert. It is highly scalable and capable of meeting performance demands, tightly integrated with other AWS services such as Amazon S3 and Amazon Lex, and offers enterprise-grade security.

AccessStatus is a monitoring bot that checks HTTP status codes of web pages to determine if URLs are active, redirected, or returning errors for website uptime monitoring.