Directorio de Crawlers

New York Times Newsgathering is a content aggregator that collects and indexes data from multiple sources.

GuestpostsBot is a Web Crawler that has several functions to facilitate the website owner who has registered his site on the guestposts.com.br platform to monitor his site. The bot constantly tracks the sites registered on the platform in order to check if the partnerships made on the guestpost platform are still active, in addition to validating if the site exists to allow registration and also monitoring the status of the site from time to time to warn the website owner in case of any inoperability.

Cloudtrellis automatically scans entire websites for broken links, accessibility issues, SEO problems, and security risks to ensure optimal user experience and search engine performance.

Spark Shipping is eCommerce automation software for retailers running WooCommerce

Qualys SSL Scanner is a monitoring crawler that checks website availability, performance, and health.

Artemis Web Reader is a feed reader that fetches and processes RSS, Atom, and other content feeds.

MgidBot is MGID's web crawler that analyzes website content to detect context categories and topics for targeted native advertising recommendations and content discovery.

Website Managed - MainWP Control Dashboard for accessing MainWP child sites.

LoomlyBot is used to extract metadata from web pages in order to show a social media post preview within Loomly so that clients can see what their social media posts will look like when published.

Since we offer sales and marketing information we need to enrich the company information. To provide crucial company information inside our service we need to provide a preview of visitor websites. Therefore we need to visit the websites.

Notabot is a web crawler that integrates with Helpfeel's AI-powered FAQ search system, indexing website content to provide intelligent search and auto-complete capabilities for customer support.

Fastmail Bot is a link preview crawler that fetches page metadata to generate rich previews when URLs are shared.

Black Duck Fast Dynamic is a security scanner for software composition analysis (SCA), identifying open source vulnerabilities and license compliance issues in web applications.

Hype Machine crawls music publications and blogs since 2005 to discover new artist posts and builds curated playlists from this metadata for music listeners and discovery.

Coinbase Webhooks are automated messages sent from the Coinbase platform to a user's server, used for notifying users about events such as receiving crypto payments.

EasyDNS' uptime monitoring probe.

Crawler to extract the newest articles in the publisher's website (via feed or parsing html) to make a carrousel with images, links and text for our native ads module in order to improve recirculation in the publisher's web. Only crawls our publisher's webpages.

WebPageTest is one of the most popular and free tools for measuring webpage performance and enables you to run web performance tests on your site from a number of different locations across the world in a number of different browsers.

honeybadgerbot is a monitoring crawler that checks website availability, performance, and health.

RetroListeCOM is a monitoring bot that regularly visits gaming websites, particularly Habbo Hotel retros, to collect and track real-time user count statistics for display on the RetroListe.com ranking platform.

The Wikimedia Foundation's Zotero Translation Server is a customized metadata extraction tool that powers Citoid (see wikimedia-citoid), retrieving citation data from URLs, DOIs, and ISBNs using Zotero translators, on demand from website visitor requests.

Cloudflare Validator makes requests to verify IPs for Cloudflare Bots Directory

OpenRSS is a nonprofit RSS feed service bot that crawls websites to generate and provide free RSS feeds, allowing users to follow website updates in a reader app.

MotoMinerBot crawls and indexes vehicle detail pages from automotive websites to build MotoMiner's searchable vehicle database and search engine platform.

Innguma Fetcher is a feed reader that fetches and processes RSS, Atom, and other content feeds.

ConvermaxBot is a feed reader that fetches and processes RSS, Atom, and other content feeds.

FeedWind Crawler is a feed reader that fetches and processes RSS, Atom, and other content feeds.

Simple crawler focussing on only job postings for job search site.

SequelWP is a monitoring agent that checks website uptime on 5-minute intervals for verified customers with valid x-sequelwp headers, helping ensure web hosting service reliability.

Help Center Export is a Zendesk-approved app that integrates with any Zendesk help center and helps the customers with these tasks: Export all your articles and any meta-data: title, section, link, labels, updated time. Export all references to internal and external docs. Detect and export broken links and images for each article. In order to check for broken links the app is using a bot that attempts to access each link present in help center articles and check the response for errors.