sources/searchhut
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Add README.md

Browse source
This commit is contained in:
Drew DeVault 2022-07-08 20:55:55 +02:00
parent d6bc032d24
commit 92ca0ecf22
1 changed files with 7 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

7
README.md Normal file
View file

@ -0,0 +1,7 @@
WIP
## Why is this crawling my site?
This crawler is still under development. It respects robots.txt Disallow and
Crawl-Delay directives. But, if it's annoying you, email sir@cmpwn.com and I'll
knock it off.