Robots.io

Robots.io is a Java library designed to make parsing a websites 'robots.txt' file easy.

The RobotsParser class provides all the functionality to use robots.io.

Domains passed to RobotsParser are normalised to always end in a forward slash. Disallowed Paths returned will never begin with a forward slash. This is so that URL's can easily be constructed.

Project Activity

See All Activity >

License

GNU General Public License version 3.0 (GPLv3)

Follow Robots.io

Robots.io Web Site

Other Useful Business Software

Gen AI apps are built with MongoDB Atlas

Build gen AI apps with an all-in-one modern database: MongoDB Atlas

MongoDB Atlas provides built-in vector search and a flexible document model so developers can build, scale, and run gen AI apps without stitching together multiple databases. From LLM integration to semantic search, Atlas simplifies your AI architecture—and it’s free to get started.

Start Free

Rate This Project

User Reviews

Be the first to post a review of Robots.io!

Additional Project Details

Programming Language

Java

Related Categories

Java Search Engines, Java Site Management Software

Registered

2014-12-24

Similar Business Software

Easy Hosting Control Panel (EHCP)

Ehcp is a tool in a server to facilitate the process of hosting domains, emails, adding domains, FTP users, and so on. Full PHP, opensource, easily modifiable, customizable, Ehcp templates, domains, subdomains, FTP, MySQL, DNS (automatic), email, email forwarding, autoreply, catch-all emails,...

See Software
H-Sphere

H-Sphere is a scalable multiserver web hosting solution. It has many advanced features and a sophisticated billing system to automate and improve your web hosting tasks. H-Sphere was designed to work on many servers and can be scaled by adding more web, mail, database, and DNS servers without...

See Software
Perplexity

Where knowledge begins. Perplexity is an AI search engine that gives you quick answers. Available for free at as a web app, desktop app, or on the go on iPhone or Android. Perplexity AI is an advanced search and question-answering tool that leverages large language models to provide...

See Software
ispmanager

Ispmanager – websites and web environment Linux control panel that makes life easier for web developers and hosting professionals. The main features: - Work and download a popular CMS: WordPress, Drupal, Joomla - Manage files through the built-in file manager - Choose from various web...

See Software
aaPanel

aaPanel, a simple but powerful control panel, can manage the web server through web-based GUI(Graphical User Interface). aaPanel provides the one-click function such as one-click install LNMP/LAMP developing environment and software. Our main goal is to help users to save the time of deploying,...

See Software
Google

Our mission is to organize the world's information so that everyone can access and use it. Every time you search, there are thousands, sometimes millions, of webpages with helpful information. How Google figures out which results to show starts long before you even type, and is guided by a...

See Software