Blog

Notabot is a web crawler that integrates with Helpfeel's AI-powered FAQ search system, indexing website content to provide intelligent search and auto-complete capabilities for customer support.

Fastmail Bot is a link preview crawler that fetches page metadata to generate rich previews when URLs are shared.

Black Duck Fast Dynamic is a security scanner for software composition analysis (SCA), identifying open source vulnerabilities and license compliance issues in web applications.

Hype Machine crawls music publications and blogs since 2005 to discover new artist posts and builds curated playlists from this metadata for music listeners and discovery.

Coinbase Webhooks are automated messages sent from the Coinbase platform to a user's server, used for notifying users about events such as receiving crypto payments.

EasyDNS' uptime monitoring probe.

Crawler to extract the newest articles in the publisher's website (via feed or parsing html) to make a carrousel with images, links and text for our native ads module in order to improve recirculation in the publisher's web. Only crawls our publisher's webpages.

WebPageTest is one of the most popular and free tools for measuring webpage performance and enables you to run web performance tests on your site from a number of different locations across the world in a number of different browsers.

honeybadgerbot is a monitoring crawler that checks website availability, performance, and health.

RetroListeCOM is a monitoring bot that regularly visits gaming websites, particularly Habbo Hotel retros, to collect and track real-time user count statistics for display on the RetroListe.com ranking platform.

The Wikimedia Foundation's Zotero Translation Server is a customized metadata extraction tool that powers Citoid (see wikimedia-citoid), retrieving citation data from URLs, DOIs, and ISBNs using Zotero translators, on demand from website visitor requests.

Cloudflare Validator makes requests to verify IPs for Cloudflare Bots Directory

OpenRSS is a nonprofit RSS feed service bot that crawls websites to generate and provide free RSS feeds, allowing users to follow website updates in a reader app.

MotoMinerBot crawls and indexes vehicle detail pages from automotive websites to build MotoMiner's searchable vehicle database and search engine platform.

Innguma Fetcher is a feed reader that fetches and processes RSS, Atom, and other content feeds.

ConvermaxBot is a feed reader that fetches and processes RSS, Atom, and other content feeds.

FeedWind Crawler is a feed reader that fetches and processes RSS, Atom, and other content feeds.

Simple crawler focussing on only job postings for job search site.

SequelWP is a monitoring agent that checks website uptime on 5-minute intervals for verified customers with valid x-sequelwp headers, helping ensure web hosting service reliability.

Help Center Export is a Zendesk-approved app that integrates with any Zendesk help center and helps the customers with these tasks: Export all your articles and any meta-data: title, section, link, labels, updated time. Export all references to internal and external docs. Detect and export broken links and images for each article. In order to check for broken links the app is using a bot that attempts to access each link present in help center articles and check the response for errors.