Open Source Search Engines

The Lemur Project develops search engines, browser toolbars, text analysis tools, and data resources that support research and development of information retrieval and text mining software, including the Indri search engine in C++, the Galago search engine research framework in Java, the RankLib learning to rank library, ClueWeb09 and ClueWeb12 datasets and the Sifaka data mining application.

The stuff here has no documentation and some of it may never be completed. This is my playground, use at your own risk.

PDFBox is a Java PDF Library. This project will allow access to all of the components in a PDF document. More PDF manipulation features will be added as the project matures. This ships with a utility to take a PDF document and output a text file.

XODA is a KISSed (Keep Simple and Stupid) System for Organizing Documents using AJAX. This is a Document Management System without backend database, though making possible organizing files/directories by descriptions, filters and more. Visit xoda.org

PHP Crawler is a simple website search script for small-to-medium websites. The only requrements are PHP and MySQL, no shell access required.

Web data extraction (web data mining, web scraping) tool. It leverages well proved XML and text processing techologies in order to easely extract useful data from arbitrary web pages.

Project moved to GitHub! https://github.com/carrot2/carrot2 Carrot2 is an Open Source Search Results Clustering Engine. It can automatically organize small collections of documents, e.g. search results, into thematic categories. Carrot2 integrates very well with both Open Source and proprietary search engines.

SkyCD is a program for indexing your files in CDs and CDs also. All indexing information is saved in text files, so anyone can edit or view with existing text editor/viewer. You can also send these files to your friends & they will know what CD you have.

Sperowider Website Archiving Suite is a set of Java applications, the primary purpose of which is to spider dynamic websites, and to create static distributable archives with a full text search index usable by an associated Java applet.

NOTICE: This code repository is deprecated. Please visit https://github.com/cdlib/xtf for the latest updates. Obsolete Description: The eXtensible Text Framework (XTF) is an architecture that supports searching across collections of heterogeneous textual data (XML, PDF, HTML, text, and more), and the presentation of results and documents in a highly configurable manner. Includes highly customized versions of the proven open-source components Lucene and Saxon.

A drop-in framework for adding tagging (folksonomy) capabilities to existing applications

QZARCH - Quick free-text search The project aims to deliver a light-weight file-based free-text search engine for Java based websites to adopt easily. The features include: - Search for one or more keywords in the content of one or more files -

"Swish-e is a fast, flexible, and free open source system for indexing collections of Web pages or other files" (http://swish-e.org/ ) This module provides a Python API for this software.

A Microformatted Registry Alternative (AMRA) is an alternative web service description registry using simple, microformatted documents.

Robust featureful multi-threaded CLI web spider using apache commons httpclient v3.0 written in java. ASpider downloads any files matching your given mime-types from a website. Tries to reg.exp. match emails by default, logging all results using log4j.

Aperture is a Java framework for extracting and querying full-text content and metadata from various information systems (file systems, web sites, mail boxes, ...) and the file formats (documents, images, ...) occurring in these systems.

A web app for creating a repository of pictures (our focus is birds). Users submit pictures, with a wizard that generates RDF descriptiors. Sumissions are forwarded to Admins for aproval. Instances will export the RDF so that repositories may cooperate.

Blogometro is simplified implementation of "Blodgex", to track the weblog's links and updates.

BlueBox is PHP-MySQL powered search engine. It can be installed on every webserver without any permission. Only FTP and database management rights are required. BlueBox is very fast even on more than 1'000'000 pages scanned.

The Cornell Web Lab Collaboration Server is a suite of tools and services for GUI-based extraction, analysis and sharing of archived web data. See http://weblab.infosci.cornell.edu/ and http://www.cs.cornell.edu/~weigel for details about the project.

CoverYourASP.com - complete Active Server Pages source (JScript) for this popular web site. Includes full membership system, diary, online db admin, banner ad system and loads more.

Port of the Google sitemap generator, from Python to Csharp aka C-Sharp aka C# aka .NET aka dotNet.

DVDWeb is a Web Service which provides organization/search/lookup services through JAX-RPC API. The search can be done against the builtin DB (the user\'s private list of DVDs according to UPC codes) or against other Internet sites such as imdb or yahoo.

An extensible framework for the automated harvesting, indexing and collation of publicly available data from internet resources for example RSS feeds and webservices.

Ever heard of Deep Web, thats the real Internet. The Deep Web is several orders of magnitude larger than the surface web and using Google's Advanced operators its now possible to search this unexplored part of the Internet.