Serverjonas/search
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
7 Commits 1 Branch 0 Tags
0adf9eb42c3a236ca48e62701f94cafdba77e006
T
Clone
Open with VS Code Open with VSCodium Open with Intellij IDEA
Download ZIP Download TAR.GZ Download BUNDLE
jhkaktus 0adf9eb42c Skip over-long URLs before inserting into crawl_queue 
Extremely long URLs exceeded the PostgreSQL B-tree index row limit
(~2704 bytes) on crawl_queue.url and crashed workers with
ProgramLimitExceeded. queue_urls() now drops URLs whose UTF-8
representation exceeds MAX_QUEUE_URL_BYTES (2000), logs a warning with
the skipped count and limit, and returns 0 early when nothing remains.
The INSERT with ON CONFLICT (url) DO NOTHING and the DB schema are
unchanged, so other crawler instances stay compatible.

🤖 Generated with Codebuff
Co-Authored-By: Codebuff <noreply@codebuff.com>
2026-09-06 10:42:19 +02:00
search
add search
2026-09-02 20:06:04 +02:00
web
add better analysis
2026-09-05 21:50:42 +02:00
.env.example
add search
2026-09-02 19:54:49 +02:00
.gitignore
add better analysis
2026-09-05 21:50:42 +02:00
crawler.py
Skip over-long URLs before inserting into crawl_queue
2026-09-06 10:42:19 +02:00
LICENSE
Initial commit
2026-09-02 16:14:24 +00:00
README.md
add search
2026-09-02 19:54:49 +02:00

README.md

search

das komplette serverjonas search

  • crawler.py – PostgreSQL-Crawler, baut den Suchindex auf
  • search/ – Suchmaschine darüber (CLI + HTTP-API), siehe search/README.md
  • web/ – Statistik-Webseite
Reference in New Issue View Git Blame Copy Permalink
S
Description
das komplette serverjonas search
Readme MIT
538 KiB
Languages
Python 98.1%
Shell 1.3%
PLpgSQL 0.6%