5 commits

Author	SHA1	Message	Date
Drew DeVault	35a4faa05b	sh-index: fetch user agent from config	2022-07-09 18:14:06 +02:00
Drew DeVault	a8069bb73b	Increase default delay to 5 seconds	2022-07-08 20:56:00 +02:00
Drew DeVault	d6bc032d24	crawler: respect robots.txt	2022-07-08 20:30:09 +02:00
Drew DeVault	fbd0492ef1	cmd/sh-search: initial commit	2022-07-08 20:04:37 +02:00
Drew DeVault	050694c4f2	Initial commit	2022-07-08 19:46:11 +02:00