DeDuplicator (Heritrix add-on) download

The DeDuplicator is an add-on module (plug-in) for the web crawler Heritrix. It offers a means to reduce the amount of duplicate data collected in a series of snapshot crawls.

Project Activity

See All Activity >

License

GNU Library or Lesser General Public License version 2.0 (LGPLv2)

Follow DeDuplicator (Heritrix add-on)

DeDuplicator (Heritrix add-on) Web Site

Other Useful Business Software

HR Onboarding Software

WorkBright streamlines form collection to get your new team members on the job in a quick, compliant, and 100% remote process.

WorkBright is a cloud-based new hire onboarding solution that provides assistance for the processing and induction of new employees before their first day on the job. Simple and easy-to-use, this paperless digital onboarding platform enables new employees to upload photos of relevant documents, fill out their W4s, capture signatures electronically, and complete all paperwork from their tablets, laptops, or smartphones. With WorkBright, organizations can seamlessly eliminate manual data entry, streamline the form correction workflow efficiently, deliver automated reminders, and more.

Learn More

Rate This Project

User Reviews

Be the first to post a review of DeDuplicator (Heritrix add-on)!

Additional Project Details

Languages

English

Intended Audience

Advanced End Users, Developers, System Administrators

User Interface

Plugins

Programming Language

Java

Related Categories

Java Internet Software, Java Web Scrapers

Registered

2006-11-06

Similar Business Software

Oxylabs

Oxylabs is a market leader in web intelligence with enterprise-grade, ethical, and compliant solutions. Its proxy infrastructure spans one of the largest global networks, offering residential, ISP, mobile, datacenter, and dedicated datacenter proxies, along with Web Unblocker – an AI-driven...

See Software
PathSolutions TotalView

PathSolutions TotalView network monitoring and troubleshooting software bridges the gap between NETWORK MONITORING and TROUBLESHOOTING RESOLUTION telling you WHEN, WHERE and WHY network errors occur. PathSolutions TotalView continuously monitors and tracks the performance of every device and...

See Software
Google Cloud BigQuery

BigQuery is a serverless, multicloud data warehouse that simplifies the process of working with all types of data so you can focus on getting valuable business insights quickly. At the core of Google’s data cloud, BigQuery allows you to simplify data integration, cost effectively and securely...

See Software
JDisc Discovery

JDisc Discovery is a comprehensive network inventory and IT asset management solution designed to help organizations gain clear, up-to-date visibility into their IT environment. It automatically scans and maps devices across the network, including servers, workstations, virtual machines, and...

See Software
Google Cloud SQL

Fully managed relational database service for MySQL, PostgreSQL, and SQL Server with rich extension collections, configuration flags, and developer ecosystems. New customers get $300 in free credits to spend on Cloud SQL. You won’t be charged until you upgrade. Reduce maintenance costs with...

See Software
Apify

Apify is a full-stack web scraping and automation platform helping anyone get value from the web. At its core is Apify Store, a marketplace with over 10,000 Actors where developers build, publish, and monetize automation tools. Actors are serverless cloud programs that extract data, automate...

See Software