Directorio de Crawlers

RSS API is a feed reader that fetches and processes RSS, Atom, and other content feeds.

Huckabot is Huckabuy’s main crawler which is utilized by almost all of Huckabuy’s products. The primary purpose of Huckabot is to crawl and index a customer’s website, which is then rendered and optimized with our Dynamic Rendering Product. Several of the Page Speed product boosters, such as Fold Prioritization, also leverage Huckabot in order to optimize and improve a website’s performance.

The Worldline Bot is associated with Worldline, a payment and transactional services company. It handles notifications and callbacks related to payment processing.

Watchful is a monitoring crawler that checks website availability, performance, and health.

ZumBot is a web crawler operated by Zum that systematically indexes webpages and content to provide search results for Zum's Korean portal and search engine services.

W3C Validator validates HTML, CSS, and web accessibility standards for the World Wide Web Consortium's validation services.

Critical CSS Bot extracts critical above-the-fold CSS from web pages to optimize page load performance.

SemrushBot-OCOB is the web crawler component used by Semrush's Content Toolkit for gathering and analyzing website content, supporting content marketing research, competitive analysis and SEO optimization.

Google-Adwords-Instant is the web crawler bot connected to the Google Ads platform that visits advertiser landing pages to perform quality checks, verification, and compliance assessment for advertising campaigns.

meta-externalfetcher is used by Meta to perform user-initiated fetches of individual links from AI assistant product functions.

Mediavine Metadata Parser extracts page content and metadata for Mediavine's ad management platform, optimizing ad placement for premium publishers.

Cloudflare Browser Rendering

MistralAI-User is Mistral's AI assistant bot that performs web browsing and data gathering tasks for users in Le Chat, including opening web pages and retrieving information to answer user queries.

Bling ERP is a crawler for Bling, a Brazilian ERP and e-commerce management platform, used for product and inventory synchronization.

Our microservice downloads js files from our users servers in order to format them and show them a human readable file. This is done to facilitate solving errors associated with said file

Spectate is an uptime monitoring bot that performs health checks on websites and web applications to ensure service availability and track performance metrics.

SmartologyBot is a contextual advertising crawler that analyzes web content to generate semantic vectors for matching publisher content with relevant advertisements on their SmartMatch platform using real-time bidding.

StatsDroneBot is StatsDrone's web scraper that collects and aggregates affiliate marketing statistics and performance data from various websites and platforms.

MediaMonitoringBot crawls and indexes news and media publishers websites for a new materials and try to match it against keywords provided by our customers (subscribers) and send them updates based on that information.

ServerHunterSpider is a specialized web crawler that indexes pricing, specifications, and stock information of hosting plans from 850+ providers, fully respecting robots.txt while building a comprehensive server comparison database.

netumo is Netumo's website uptime monitoring bot that checks website availability and performance, alerting users to downtime and providing detailed analytics on site reliability and response times.

SiteSearch360 crawls and indexes website content to provide custom site search functionality, offering semantic search capabilities as a Google Site Search replacement.

We scrape full article/page content to ensure we can optimally automate the content distribution for the digital publishers we work with. Every single article a publisher releases will get scraped approx. 2-4 times by independent services.

Medialogia Bot is a monitoring crawler that checks website availability, performance, and health.

MarketGoo is a white-label SEO tool provider that crawls websites to analyze search engine optimization metrics, providing comprehensive SEO insights and domain intelligence for service providers and their customers.

Project Shield, created by Google Cloud and Jigsaw and powered by Google Cloud Armor, provides free unlimited protection against DDoS attacks, a type of digital attack used to censor information by taking websites offline.

2Checkout Payment Platform's webhooks integration for notifying websites about payment events. Now operated by Verifone.

Mars Finder is a website search service designed to utilize the maximum potential of a website. MARS FINDER has held the top share of website search service market of Japan in 2017.

online-webceo-bot is WebCEO's crawler that audits websites for their comprehensive SEO platform, providing 24 professional tools for agency-level analysis, reports, and digital marketing insights.

Bitbucket Webhooks for CI/CD