Skip to content

fix(crawler,content): harden XHS search + utf8mb4#19

Merged
hongzexin merged 1 commit into
mainfrom
fix/dev-crawler-stability-utf8mb4
Feb 24, 2026
Merged

fix(crawler,content): harden XHS search + utf8mb4#19
hongzexin merged 1 commit into
mainfrom
fix/dev-crawler-stability-utf8mb4

Conversation

@hongzexin

Copy link
Copy Markdown
Contributor

Summary

  • Crawler: avoid unhandled page.waitForResponse rejections in XHS search when navigation fails
  • Content: add Flyway migration V7__utf8mb4.sql to convert text tables to utf8mb4 (emoji-safe)

Why

  • Dev crawler sometimes crashed on XHS timeouts due to an unhandled promise.
  • Dev content returned HTTP 500 on ingest with MySQL error 1366 (emoji in body_md).

Test plan

  • backend-CHEK-content: mvn test
  • Crawler: manual CronJob run in dev

Made with Cursor

Prevent unhandled waitForResponse rejections in XHS search flow and migrate content text tables to utf8mb4 to accept emojis.

Co-authored-by: Cursor <cursoragent@cursor.com>
@hongzexin hongzexin merged commit a49eb8d into main Feb 24, 2026
12 checks passed
@hongzexin hongzexin deleted the fix/dev-crawler-stability-utf8mb4 branch February 24, 2026 11:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant