Drew DeVault a8069bb73b Increase default delay to 5 seconds		2022-07-08 20:56:00 +02:00
..
crawler.go	Increase default delay to 5 seconds	2022-07-08 20:56:00 +02:00
index.go	crawler: respect robots.txt	2022-07-08 20:30:09 +02:00