Const
How slowly to go on a host whose robots.txt the operator overrode: one request every four seconds, against a default of 500 ms. A server that asked not to be crawled at all gets the slowest rate this tool offers.
How slowly to go on a host whose robots.txt the operator overrode: one request every four seconds, against a default of 500 ms. A server that asked not to be crawled at all gets the slowest rate this tool offers.