7 lines
210 B
Markdown
7 lines
210 B
Markdown
WIP
|
|
|
|
## Why is this crawling my site?
|
|
|
|
This crawler is still under development. It respects robots.txt Disallow and
|
|
Crawl-Delay directives. But, if it's annoying you, email sir@cmpwn.com and I'll
|
|
knock it off.
|