No description
cmd | ||
crawler | ||
database | ||
graph | ||
.gitignore | ||
COPYING | ||
go.mod | ||
go.sum | ||
gqlgen.yml | ||
README.md | ||
schema.sql |
WIP
Why is this crawling my site?
This crawler is still under development. It respects robots.txt Disallow and Crawl-Delay directives. But, if it's annoying you, email sir@cmpwn.com and I'll knock it off.