|Subject:||wish: respect robots.txt by default|
Let's be a good netizen and support robots.txt by default (or at least as an option!). jeffa has already written the code for this. It's listed here with more details:Mark
This queue is for tickets about the WWW-Mechanize CPAN distribution.
This service is sponsored and maintained by Best Practical Solutions and runs on Perl.org infrastructure.
Please report any issues with rt.cpan.org to firstname.lastname@example.org.