An almost generic web crawler built using Scrapy and Python 3.7 to recursively crawl entire websites.
Crawler is a C++ & Node.js application that allows you to crawl web pages, save them locally, and extract hyperlinks from the page body. It provides a simple command-line interface where you can enter the starting URL and specify the maximum number of crawls. The crawler follows the hyperlinks recursively, saves the web pages in specified directo
A multithreaded web crawler written in Go that recursively discovers and visits webpage links up to a configurable depth.
A lightweight Java-based web crawler designed to explore and analyze web pages recursively, performing keyword-based discovery and depth-limited link extraction.
legacy web crawler automation tool
To associate your repository with the recursive-crawling topic, visit your repo's landing page and select "manage topics."