Drew DeVault
|
c15f968a28
|
crawler: re-schedule after HTTP 429
Fixes: https://todo.sr.ht/~sircmpwn/searchhut/5
|
2022-07-09 19:14:55 +02:00 |
|
Drew DeVault
|
baf82f9bb8
|
crawler: perform HEAD before GET
Implements: https://todo.sr.ht/~sircmpwn/searchhut/8
|
2022-07-09 18:59:23 +02:00 |
|
Drew DeVault
|
35a4faa05b
|
sh-index: fetch user agent from config
|
2022-07-09 18:14:06 +02:00 |
|
Drew DeVault
|
a8069bb73b
|
Increase default delay to 5 seconds
|
2022-07-08 20:56:00 +02:00 |
|
Drew DeVault
|
d6bc032d24
|
crawler: respect robots.txt
|
2022-07-08 20:30:09 +02:00 |
|
Drew DeVault
|
fbd0492ef1
|
cmd/sh-search: initial commit
|
2022-07-08 20:04:37 +02:00 |
|
Drew DeVault
|
050694c4f2
|
Initial commit
|
2022-07-08 19:46:11 +02:00 |
|