Welcome to ftp.nluug.nl Current directory: /os/Linux/distr/salix/sbo/14.0/perl/perl-www-robotrules/ |
|
Contents of README:This module parses /robots.txt files as specified in "A Standard for Robot Exclusion", at <http://www.robotstxt.org/wc/norobots.html> Webmasters can use the /robots.txt file to forbid conforming robots from accessing parts of their web site. The parsed files are kept in a WWW::RobotRules object, and this object provides methods to check if access to a given URL is prohibited. The same WWW::RobotRules object can be used for one or more parsed /robots.txt files on any number of hosts. |
Name Last modified Size
Parent Directory - README 29-Sep-2012 06:34 488 perl-www-robotrules.SlackBuild 30-Apr-2012 05:51 2.8K perl-www-robotrules.info 01-Oct-2012 03:07 326 slack-desc 29-Sep-2012 06:34 1.0K
NLUUG - Open Systems. Open Standards
Become a member
and get discounts on conferences and more, see the NLUUG website!