Extremely long URLs exceeded the PostgreSQL B-tree index row limit
(~2704 bytes) on crawl_queue.url and crashed workers with
ProgramLimitExceeded. queue_urls() now drops URLs whose UTF-8
representation exceeds MAX_QUEUE_URL_BYTES (2000), logs a warning with
the skipped count and limit, and returns 0 early when nothing remains.
The INSERT with ON CONFLICT (url) DO NOTHING and the DB schema are
unchanged, so other crawler instances stay compatible.
🤖 Generated with Codebuff
Co-Authored-By: Codebuff <noreply@codebuff.com>
jhkaktus
tech2m