sources/searchhut
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
No description
61 commits 1 branch 0 tags 139 KiB
Go 53.4%
Python 25.1%
HTML 12.1%
CSS 8.4%
Makefile 1%
Find a file
Umar Getagazov cbd3732deb Store page size in the database 
Implements: https://todo.sr.ht/~sircmpwn/searchhut/33
2022-07-13 10:14:37 +02:00
cmd sh-index: add -u flag to add URLs to schedule 2022-07-11 20:57:59 +02:00
config Fix searchut typo in the config file path 2022-07-11 13:17:16 +02:00
crawler Store page size in the database 2022-07-13 10:14:37 +02:00
database database: add middleware 2022-07-09 13:52:55 +02:00
graph API: add index size to stats 2022-07-11 21:38:29 +02:00
import Store page size in the database 2022-07-13 10:14:37 +02:00
query web: add total pages indexed to home page 2022-07-11 20:40:53 +02:00
static web: match alert's dark theme colors with sr.ht 2022-07-13 10:14:35 +02:00
templates web: add total pages indexed to home page 2022-07-11 20:40:53 +02:00
.gitignore Add Makefile 2022-07-09 18:14:00 +02:00
config.example.ini sh-api: expand top-level server riggings 2022-07-09 15:39:04 +02:00
COPYING Initial commit 2022-07-08 19:46:11 +02:00
go.mod web: add total pages indexed to home page 2022-07-11 20:40:53 +02:00
go.sum web: add total pages indexed to home page 2022-07-11 20:40:53 +02:00
gqlgen.yml API: Implement search resolver 2022-07-09 15:48:03 +02:00
Makefile Add Makefile 2022-07-09 18:14:00 +02:00
README.md Add README.md 2022-07-08 20:55:55 +02:00
schema.sql Store page size in the database 2022-07-13 10:14:37 +02:00

README.md

WIP

Why is this crawling my site?

This crawler is still under development. It respects robots.txt Disallow and Crawl-Delay directives. But, if it's annoying you, email sir@cmpwn.com and I'll knock it off.