sources/searchhut
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
23 commits 1 branch 0 tags 139 KiB
Commit graph

5 commits

Author SHA1 Message Date
Drew DeVault
baf82f9bb8 crawler: perform HEAD before GET 
Implements: https://todo.sr.ht/~sircmpwn/searchhut/8
 2022-07-09 18:59:23 +02:00
Drew DeVault
759ad758af crawler: improve index settings 2022-07-09 18:57:39 +02:00
Drew DeVault
d6bc032d24 crawler: respect robots.txt 2022-07-08 20:30:09 +02:00
Drew DeVault
eb6769c904 crawler: follow links regardless of readability 2022-07-08 20:13:32 +02:00
Drew DeVault
050694c4f2 Initial commit 2022-07-08 19:46:11 +02:00