Search Results for "webspider"

...It consists of * crawler (spider) and indexer * search for your website(s) * search statistics * website analyzer (http header, short titles and keywords, linkchecker, ...) You need to install it on your own server. So all crawled data stay in your environment. You never know when an external webspider updated your content. Trigger a rescan whenever you want - you always have under control what data of what time were checked. The spider is a CLI tool and must be added as a cronjob. In a web based backend you can control all data and analyze your data. You can handle multiple websites in the same backend. PHP 7 or 8 + PDO (Mysql/ Sqlite)

PHPCrawl is a high configurable webcrawler/webspider-library written in PHP. It supports filters, limiters, cookie-handling, robots.txt-handling, multiprocessing and much more.

Webfear Structiatella is a C# written webspider or mirroring engine, which allowes to specify (even multilevel) content of websites and mirror it into a XML file. It provides a GUI, mirroring engine and a DLL-api for integration into own programs.

webspider provides a mechanism to get contents from web. With the extended classes, you can do the following things: 1. grab urls from a specified base url 2. analyze the contents of a list of urls 3. get specific files from web 4. blablabla

Pr0nSpider is an example (multi-threaded porn downloading) application of the included WebSpider C# library. The WebSpider library is intended to make multi-threaded webspidering in C# as easy as possible. See the home page (link above) for details.