All Packages  Class Hierarchy  This Package  Previous  Next  Index

Class Acme.NoRobots 

java.lang.Object
   |
   +----Acme.NoRobots
public class NoRobots
extends Object
Implements the Robot Exclusion Standard.

The basic idea of the Robot Exclusion Standard is that each web server can set up a single file called "/robots.txt" which contains pathnames that robots should not look at. See the full spec for details. Using this class is very simple - you create the object using your robot's name, and then you call ok() on each URL. For efficiency, the class caches entries for servers you've visited recently.

Fetch the software.
Fetch the entire Acme package.

See Also:
Spider

Constructor Index

o NoRobots(String)
Constructor.

Method Index

o ok(URL)
Check whether it's ok for this robot to fetch this URL.

Constructors

o NoRobots 
 public NoRobots(String robotName)
Constructor.

Methods

o ok 
 public boolean ok(URL url)
Check whether it's ok for this robot to fetch this URL. 

All Packages  Class Hierarchy  This Package  Previous  Next  Index
ACME Java  ACME Labs