Search Results for "text crawler"
Sort By:
Showing 29 open source projects for "text crawler"
View related business solutions-
Gemini 3 and 200+ AI Models on One PlatformBuild generative AI apps with Vertex AI. Switch between models without switching platforms.
-
AI-generated apps that pass security reviewRetool lets you generate dashboards, admin panels, and workflows directly on your data. Type something like “Build me a revenue dashboard on my Stripe data” and get a working app with security, permissions, and compliance built in from day one. Whether on our cloud or self-hosted, create the internal software your team needs without compromising enterprise standards or control.
-
1
Weibo Crawler
Python crawler for collecting and downloading Sina Weibo user data
weibo-crawler is a Python-based data collection tool designed to retrieve information from Sina Weibo user accounts. It automates the process of gathering posts, user profile details, and engagement metrics from one or more target accounts. weibo-crawler can extract comprehensive information about users, including profile attributes such as nickname, follower count, following count, and account metadata. It also captures detailed data about each post, including the content, publishing time,... -
2
fess
Open source enterprise search server for websites, files, and data
Fess is an open source enterprise search server designed to provide powerful full-text search capabilities across multiple data sources. It enables organizations to quickly deploy a scalable search environment without requiring deep knowledge of underlying search technologies. Fess is built on top of OpenSearch and offers an integrated solution for crawling, indexing, and searching documents from websites, file systems, and various data stores. -
3
PaSa
An advanced paper search agent powered by large language models
..., PaSa decomposes the task: the Crawler generates search queries, retrieves candidate papers (via search tools and citation expansion), then adds them to a “paper queue.” The Selector then reads abstracts or full text (depending on what’s available) and decides which papers are relevant. -
4
news-please
Python tool for crawling and extracting structured data from news site
news-please is an open source news crawler and information extraction tool designed to collect and structure articles from online news websites. It provides an integrated pipeline that crawls news sites, retrieves article pages, and extracts structured information such as headlines, authors, publication dates, and article text. news-please can recursively follow internal links and read RSS feeds to gather both recent and archived articles from a news outlet when given only the root URL of a site. ... -
Auth0 B2B Essentials: SSO, MFA, and RBAC Built InAuth0's B2B Essentials plan gives you everything you need to ship secure multi-tenant apps. Unlimited orgs, enterprise SSO, RBAC, audit log streaming, and higher auth and API limits included. Add on M2M tokens, enterprise MFA, or additional SSO connections as you scale.
-
5
Web Spider, Web Crawler, Email Extractor
Free Extracts Emails, Phones and custom text from Web using JAVA Regex
In Files there is WebCrawlerMySQL.jar which supports MySql Connection Free Web Spider & Crawler. Extracts Information from Web by parsing millions of pages. Store data into Derby Database and data are not being lost after force closing the spider. - Free Web Spider , Parser, Extractor, Crawler - Extraction of Emails , Phones and Custom Text from Web - Export to Excel File - Data Saved into Derby and MySQL Database - Written in Java Cross Platform Also See Free email Sender : https://sourceforge.net/projects/gitst-free-email-ender/ Please install Microsoft OpenJDK to start the application https://www.microsoft.com/openjdk -
6
Wizards Castle
Text adventure: Battle monsters & cast spells to claim the Orb of Zot!
Wizard's Castle - A Classic Text-Based Dungeon Crawler Reborn Embark on an epic quest through a procedurally generated castle in search of the legendary Orb of Zot! This modern reimagining of the 1980 BASIC game combines classic ASCII graphics with enhanced features and multiple implementations. Choose from five unique races (Human, Elf, Hobbit, Dwarf, or Dark Elf), battle fearsome monsters from Kobolds to Dragons, cast powerful spells, and explore 512 rooms filled with treasures, traps, and mysteries. ... -
7
WFDownloader App
Free batch downloader for image, wallpaper, video, audio, document,
Use as an image gallery, wallpaper, audio/music, video, document, and other media bulk downloader from supported websites. Also use to download sequential website urls that have a certain pattern (e.g. image01.png to image100.png). Also use app's built-in site crawler for advanced link search or extraction. There is also special support for forum media and open directory downloading. It's a programmable downloader and also works with password protected sites. Say goodbye to downloading one... -
8
Easyspider - Distributed Web Crawler
Easy Spider is a distributed Perl Web Crawler Project from 2006
...If you want to write text automatically try https://www.artikelschreiber.com/en/ or https://www.unaique.net/en/! -
9
Web Spider, Web Crawler, Email Extractor
Free Extracts Emails, Phones and custom text from Web using JAVA Regex
In Files there is WebCrawlerMySQL.jar which supports MySql Connection Please follow this link to get latest version https://sourceforge.net/projects/web-spider-web-crawler-extract/ Free Web Spider & Crawler. Extracts Information from Web by parsing millions of pages. Store data into Derby OR MySQL Database and data are not being lost after force closing the spider. - Free Web Spider , Parser, Extractor, Crawler - Extraction of Emails , Phones and Custom Text from Web - Export to Excel File - Data Saved into Derby Database - Written in Java Cross Platform See also Free Email Sender in this link: https://sourceforge.net/projects/gitst-free-email-ender/ Please install Microsoft OpenJDK to start the application https://www.microsoft.com/openjdk -
AI-powered service management for IT and enterprise teamsGive your IT, operations, and business teams the ability to deliver exceptional services—without the complexity. Maximize operational efficiency with refreshingly simple, AI-powered Freshservice.
-
10
Hakrawler
Fast Go web crawler for discovering URLs and web app endpoints
hakrawler is a lightweight command-line web crawler built in Go that is designed to quickly discover URLs, endpoints, and assets within web applications. It is primarily used during the reconnaissance phase of security testing, bug bounty hunting, and penetration testing. It works by automatically crawling web pages and extracting links, JavaScript file locations, and other resources that may reveal additional attack surface or hidden functionality. hakrawler is implemented as a simple and... -
11
Universal Proxy Software 2.0
Universal Proxy Software 2.0 is the most advanced proxy software.
...Features: Proxy Grabber Proxy Checker Proxy(IP) Changer Mega Proxy Grabber Mega Proxy Checker Auto Proxy Changer Proxy Editor Proxy Leecher Proxy Scrapper Mega Proxy Editor Mega Proxy Leecher Mega Proxy Scrapper Proxy Combiner Proxy Lookup Text Proxy Leecher Mega Proxy Combiner Mega Proxy Lookup Mega Text Proxy Leecher Mor Crawler Proxy Viewer Proxy URL Grabber Proxy Grabber: Proxy Grabber is one of the unique features of Universal Proxy Software.This grabber can grab https proxies upto 17k and socks 5 upto 37k and socks 4 also. Mega Proxy Checker Mega Proxy Checker is very fast and it can support a large amount of proxies. ... -
12
mzitu
Python crawler that downloads image galleries and analyzes titles
mzitu is a Python-based web crawling project designed to automatically download and organize image galleries from a specific photography site. It demonstrates how to build a scraper that navigates gallery pages, retrieves image links, and saves the images locally in a structured directory layout. It focuses on automating the collection of large sets of images by programmatically parsing page content and iterating through gallery entries. mzitu also includes a simple analysis script that... -
13
OpenSearchServer Search Engine
An open source search engine with RESTFul API and crawlers
OpenSearchServer is a powerful, enterprise-class, search engine program. Using the web user interface, the crawlers (web, file, database, etc.) and the client libraries (REST/API , Ruby, Rails, Node.js, PHP, Perl) you will be able to integrate quickly and easily advanced full-text search capabilities in your application: Full-text with basic semantic, join queries, boolean queries, facet and filter, document (PDF, Office, etc.) indexation, web scrapping,etc. OpenSearchServer runs on... -
14
crawler4j
Open source web crawler for Java
crawler4j is an open source web crawler for Java which provides a simple interface for crawling the Web. Using it, you can setup a multi-threaded web crawler in few minutes. You need to create a crawler class that extends WebCrawler. This class decides which URLs should be crawled and handles the downloaded page. shouldVisit function decides whether the given URL should be crawled or not. -
15
Products of the project: Java HTMLParser - VietSpider Web Data Extractor - Extractor VietSpider News. Click on "Show project details" to see more feature about each product.
-
16
go_spider
An awesome Go concurrent Crawler(spider) framework
An awesome Go concurrent Crawler(spider) framework. The crawler is flexible and modular. It can be expanded to an Individualized crawler easily or you can use the default crawl components only. Spider gets a Request in Scheduler that has url to be crawled. Then Downloader downloads the result(html, json, jsonp, text) of the Request. The result is saved in Page for parsing in PageProcesser. -
17
sitecheck
Modular web site spider for web developers.
More than just a link checker, sitecheck is a website spider (also known as a crawler) which can assist with SEO by testing an entire site plus both inbound links from search engines and outbound links to other sites for the following issues: looping redirects (HTTP 301/302), broken links (HTTP 404), server errors (HTTP 500), spelling mistakes, low readability scores (using the Flesch Reading Ease test), missing/empty/duplicate meta tags, duplicate content, slow page speed, W3C validation... -
18
Zoozle Search & Download Suchmaschine
Zoozle 2008 - 2010 Webpage, Tools and SQL Files
Download search engine and directory with Rapidshare and Torrent - zoozle Download Suchmaschine All The files that run the World Leading German Download Search Engine in 2010 with 500 000 unique visitors a day - all the tools you need to set up a clone. Code Contains: - PHP Files for zoozle - Perl Crawler for gathering new content to database and all other cool tools i have... -
19
Ancient World Of Generica
A Current Text combat game transitioning to Dungeon Crawler/Roguelike
This is a little game i have programmed for a while in python and compiled with pyinstaller. It is a very early alpha with only a few feature to boot in the first internet release version. These include: -Rudimentry Saving and loading -Weapons, armor and potions(but no inventory to store them in as of yet) - A modding interface for the above -Basic Text combat and morale implemented for a basic monster logic - Basic procedural generation for the above -Classes and Races This at the... -
20
RiverGlass EssentialScanner is an open source web and file system crawler which indexes the text content of discovered files so they can be retrieved and analyzed. It provides simple scanner capabilities as part of larger enterprise search solutions.
-
21
Pytarl is a console/terminal text adventure-like rougelike dungeon crawler game written in Python 2.6.x. (this site is only a placeholder for the win32 binary. development happens over here: http://bitbucket.org/seamot/pytarl/)
-
22
elk is a powerful open-source python based command-line web crawler that can recursively search for files and text on websites.
-
23
Allows you to create your own notes using contend from Wikipedia. A public version of the text crawler cum note maker used for Selfplatform - a project of HBCSE, Tata Institute of Fundamental Research (TIFR), India. It allows users to create their own no
-
24
A toolkit for crawling information from web pages by combining different kinds of "actions". Actions are simple operations such as navigation to a specified url or extraction of text from the html. Also available is a graphic user interface.
-
25
Web Textual eXtraction Tools C++ Parallel web crawler, noun phrase idenification, Multi-lingual Part of Speech Tagging, Tarjan's Algorithm, Co-RelationShip Mappings...
