Add README.md
This commit is contained in:
parent
d6bc032d24
commit
92ca0ecf22
1 changed files with 7 additions and 0 deletions
7
README.md
Normal file
7
README.md
Normal file
|
@ -0,0 +1,7 @@
|
|||
WIP
|
||||
|
||||
## Why is this crawling my site?
|
||||
|
||||
This crawler is still under development. It respects robots.txt Disallow and
|
||||
Crawl-Delay directives. But, if it's annoying you, email sir@cmpwn.com and I'll
|
||||
knock it off.
|
Loading…
Reference in a new issue