sources/searchhut
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
No description
6 commits 1 branch 0 tags 139 KiB
Go 53.4%
Python 25.1%
HTML 12.1%
CSS 8.4%
Makefile 1%
Find a file
Drew DeVault a8069bb73b Increase default delay to 5 seconds
2022-07-08 20:56:00 +02:00
cmd cmd/sh-search: initial commit 2022-07-08 20:04:37 +02:00
crawler Increase default delay to 5 seconds 2022-07-08 20:56:00 +02:00
database Initial commit 2022-07-08 19:46:11 +02:00
COPYING Initial commit 2022-07-08 19:46:11 +02:00
go.mod Initial commit 2022-07-08 19:46:11 +02:00
go.sum Initial commit 2022-07-08 19:46:11 +02:00
README.md Add README.md 2022-07-08 20:55:55 +02:00
schema.sql Initial commit 2022-07-08 19:46:11 +02:00

README.md

WIP

Why is this crawling my site?

This crawler is still under development. It respects robots.txt Disallow and Crawl-Delay directives. But, if it's annoying you, email sir@cmpwn.com and I'll knock it off.