8 lines
210 B
Markdown
8 lines
210 B
Markdown
|
WIP
|
||
|
|
|
||
|
|
## Why is this crawling my site?
|
||
|
|
|
||
|
|
This crawler is still under development. It respects robots.txt Disallow and
|
||
|
|
Crawl-Delay directives. But, if it's annoying you, email sir@cmpwn.com and I'll
|
||
|
|
knock it off.
|