Fix Prisma container error and crawler not finding posts

[konard]

Summary

Fixes #1 - Prisma "table does not exist" error, crawler returning empty results, data not being saved to database, and adds date-based crawling.

Bug Fix: Database table not created on container start

The prisma migrate deploy command in the Dockerfile CMD can fail silently in Prisma 7 Docker environments due to TypeScript config resolution issues. Root causes and fix:

1. Prisma 7 .ts config loading — prisma migrate deploy needs to load prisma.config.ts at runtime, which can fail if the TS execution environment isn't properly set up in the container.
2. Stale migration state — If the database already has a _prisma_migrations record from a previous (broken) deployment, Prisma skips the migration even though the table doesn't exist.
3. Added SQL fallback — TaskService.ensureTablesExist() checks if the Task table exists and creates it via raw SQL if not. Called on app startup and CLI crawl.
4. Dockerfile CMD — Now uses || instead of && for the migration step, so the app starts even if Prisma migrations fail.

Bug Fix: Data not saved to database

The crawl endpoint (POST /api/crawl) was not reliably persisting data. Root causes:

1. Missing UNIQUE index on id_post — The migration didn't match the schema (@unique on id_post), breaking Prisma's skipDuplicates behavior.
2. API only saved to DB, not files — If the DB save failed, no data was persisted. Now saves to both database and data/ folder.
3. API response hid crawled posts on DB error — Old code returned "No new posts found" when DB save failed, even though posts were crawled.
4. CLI didn't save to database — npm run crawl only saved to files. Now saves to both DB and files when DATABASE_URL is set.

Bug Fix: Crawler returns "No new posts found" when posts exist

Root cause was 6 compounding bugs:

1. ContentValidator too strict (src/services/validate/validator.ts): Required BOTH job keywords AND specific emojis. Fixed to accept keywords OR emojis.
2. Post ID extraction broken (src/services/parser/htmlParser.ts): Used a[href*='/s/'] selector but Telegram post links don't contain /s/. Fixed to use data-post attribute.
3. Timestamp selector wrong (src/services/parser/htmlParser.ts): time.datetime looked for class "datetime", but actual <time> elements use datetime attribute. Fixed to time[datetime].
4. Timestamps not extracted (src/services/extractor.ts): parsePost() never called extractTimestamp(). Now uses actual post dates.
5. AD filter too broad (src/services/validate/validator.ts): /pinned/i matched "pinned" anywhere. Changed to ^pinned (start-of-text only).
6. Invalid post URLs (src/services/extractor.ts): Generated ${baseUrl}?q=${postId} instead of proper t.me/channel/postId URLs.

Bug Fix: Prisma container error

1. Fixed DATABASE_URL (docker-compose.yaml): Changed hardcoded telegram-scrapper to ${POSTGRES_DB}.
2. Removed unused init-db volume.
3. Removed url from schema.prisma for Prisma 7 compatibility.
4. Added missing UNIQUE index on Task.id_post in migration.

New Feature: Date-based crawling

• API: POST /api/crawl accepts optional {date: "YYYY-MM-DD"} body
• CLI: npm run crawl -- --date 2025-01-15

Test Plan

• docker-compose up --build -d starts without errors
• docker-compose logs telegram-scraper shows "Database table 'Task' already exists" or "Table 'Task' created successfully via SQL fallback"
• curl -X POST http://localhost:3000/api/crawl returns posts (not empty [])
• Posts are saved in both database (GET /api/jobs) and data/ folder
• curl -X POST http://localhost:3000/api/crawl -d '{"date":"2025-01-01"}' crawls since Jan 1
• No "table does not exist" error
• Post IDs are stable numeric IDs (not post_TIMESTAMP)
• Post timestamps reflect actual publication dates

🤖 Generated with Claude Code

[konard]

🤖 Solution Draft Log

This log file contains the complete execution trace of the AI solution draft process.

💰 Cost estimation:

• Public pricing estimate: $1.679226 USD
• Calculated by Anthropic: $1.138122 USD
• Difference: $-0.541104 (-32.22%)
  📎 Log file uploaded as Gist (491KB)
  🔗 View complete solution draft log

---

Now working session is ended, feel free to review and add any feedback on the solution draft.

[lion-lef]

1.073 > tg-crawler@1.0.0 prisma:generate
1.073 > prisma generate
1.073
4.821 Loaded Prisma config from prisma.config.ts.
4.821
5.458 Prisma schema loaded from prisma/schema.prisma.
5.467 Error: Prisma schema validation - (get-config wasm)
5.467 Error code: P1012
5.467 error: The datasource property url is no longer supported in schema files. Move connection URLs for Migrate to prisma.config.ts and pass either adapter for a direct database connection or accelerateUrl for Accelerate to the PrismaClient constructor. See https://pris.ly/d/config-datasource and https://pris.ly/d/prisma7-client-config
5.467 --> prisma/schema.prisma:8
5.467 |
5.467 7 | provider = "postgresql"
5.467 8 | url = env("DATABASE_URL")
5.467 |
5.467
5.467 Validation Error Count: 1
5.467 [Context: getConfig]
5.467
5.467 Prisma CLI Version : 7.3.0

[konard]

🤖 AI Work Session Started

Starting automated work session at 2026-02-01T18:25:40.047Z

The PR has been converted to draft mode while work is in progress.

This comment marks the beginning of an AI work session. Please wait working session to finish, and provide your feedback.

[konard]

Thanks for catching that! You're right — Prisma 7 no longer supports the url property in schema.prisma.

I've pushed a fix in commit df39a6e that removes the url = env("DATABASE_URL") line from prisma/schema.prisma. The DATABASE_URL is already correctly configured in:

• prisma.config.ts (line 12) — used by Prisma CLI for migrations
• src/services/database/task.ts (line 23-30) — used at runtime via the PrismaPg adapter

The docker-compose.yaml fix (using ${POSTGRES_DB} instead of hardcoded telegram-scrapper) is still in place.

[konard]

🤖 Solution Draft Log

This log file contains the complete execution trace of the AI solution draft process.

💰 Cost estimation:

• Public pricing estimate: $1.141214 USD
• Calculated by Anthropic: $0.659049 USD
• Difference: $-0.482165 (-42.25%)
  📎 Log file uploaded as Gist (169KB)
  🔗 View complete solution draft log

---

Now working session is ended, feel free to review and add any feedback on the solution draft.

[lion-lef]

Add endpoint to crawler with a data, or write a output command to crawl with a date

[konard]

🤖 AI Work Session Started

Starting automated work session at 2026-02-01T18:43:44.689Z

The PR has been converted to draft mode while work is in progress.

This comment marks the beginning of an AI work session. Please wait working session to finish, and provide your feedback.

[konard]

Added date-based crawling support as requested. Here's what's new:

API endpoint — POST /api/crawl now accepts an optional date field in the JSON body:

# Crawl since a specific date
curl -X POST http://localhost:3000/api/crawl \
  -H "Content-Type: application/json" \
  -d '{"date": "2025-01-15"}'

# Default behavior (last 7 days) still works
curl -X POST http://localhost:3000/api/crawl

CLI command — npm run crawl now supports --date:

npm run crawl -- --date 2025-01-15

Invalid dates return a 400 error (API) or exit with an error message (CLI). When no date is provided, the default 7-day window is used.

Changes in commit bc21f81:

• src/services/srcaper/scraper.ts — scrape() accepts optional sinceDate parameter
• src/index.ts — /api/crawl parses date from request body, passes to runCrawlAndSave()
• src/crawler/crawl-only.ts — parses --date CLI argument

[konard]

🤖 Solution Draft Log

This log file contains the complete execution trace of the AI solution draft process.

💰 Cost estimation:

• Public pricing estimate: $2.225630 USD
• Calculated by Anthropic: $1.572955 USD
• Difference: $-0.652675 (-29.33%)
  📎 Log file uploaded as Gist (815KB)
  🔗 View complete solution draft log

---

Now working session is ended, feel free to review and add any feedback on the solution draft.

[lion-lef]

{"success":true,"message":"No new posts found","posts":[]}
Fix, Post exist 100%

[konard]

🤖 AI Work Session Started

Starting automated work session at 2026-02-01T18:57:22.936Z

The PR has been converted to draft mode while work is in progress.

This comment marks the beginning of an AI work session. Please wait working session to finish, and provide your feedback.

[konard]

Fixed the "No new posts found" issue in commit a2f9b2d. The root cause was 6 compounding bugs that prevented the crawler from extracting posts:

1. Validator too strict — Required both job keywords AND specific emojis (👔/💼/📌). Changed to accept posts matching either condition.

2. Post ID extraction broken — The a[href*='/s/'] CSS selector never matched any links (Telegram post URLs don't contain /s/), so all posts got random IDs like post_1738434567890. This broke deduplication across crawls. Fixed by reading the data-post attribute from message elements.

3. Timestamp CSS selector wrong — time.datetime looked for <time class="datetime"> but the actual elements are <time class="time" datetime="2026-01-19T14:23:19+00:00">. Fixed to time[datetime].

4. Timestamps never extracted — parsePost() hardcoded new Date().toISOString() instead of calling extractTimestamp(). Now uses real post dates.

5. Overly broad ad filter — /pinned/i matched "pinned" anywhere in post text. Changed to ^pinned (start-of-text only).

6. Invalid post URLs — Generated baseUrl?q=postId instead of proper t.me/channel/postId links.

All changes verified to compile cleanly with tsc --noEmit.

[konard]

🤖 Solution Draft Log

This log file contains the complete execution trace of the AI solution draft process.

💰 Cost estimation:

• Public pricing estimate: $5.170977 USD
• Calculated by Anthropic: $3.526991 USD
• Difference: $-1.643987 (-31.79%)
  📎 Log file uploaded as Gist (805KB)
  🔗 View complete solution draft log

---

Now working session is ended, feel free to review and add any feedback on the solution draft.

[konard]

🔄 Auto-restart 1/3

Detected uncommitted changes from previous run. Starting new session to review and commit them.

Uncommitted files:

M package-lock.json

---

Auto-restart will stop after changes are committed or after 2 more iterations. Please wait until working session will end and give your feedback.

[konard]

🤖 Solution Draft Log

This log file contains the complete execution trace of the AI solution draft process.

💰 Cost estimation:

• Public pricing estimate: $0.311520 USD
• Calculated by Anthropic: $0.192360 USD
• Difference: $-0.119160 (-38.25%)

Changes in this session:

• Committed the uncommitted package-lock.json (lockfile updated after dependency resolution changes from prior commits)
• Verified TypeScript compilation passes cleanly (tsc --noEmit)
• Reviewed full PR diff — all changes are correct and complete
• Working tree is clean, no uncommitted changes remain

---

Now working session is ended, feel free to review and add any feedback on the solution draft.

[konard]

🔄 Auto-restart 1/3 Log

This log file contains the complete execution trace of the AI solution draft process.

💰 Cost estimation:

• Public pricing estimate: $1.205436 USD
• Calculated by Anthropic: $0.652153 USD
• Difference: $-0.553283 (-45.90%)
  📎 Log file uploaded as Gist (1006KB)
  🔗 View complete solution draft log

---

Now working session is ended, feel free to review and add any feedback on the solution draft.

[konard]

🤖 Solution Draft Log

This log file contains the complete execution trace of the AI solution draft process.

💰 Cost estimation:

• Public pricing estimate: $1.205436 USD
• Calculated by Anthropic: $0.652153 USD
• Difference: $-0.553283 (-45.90%)
  📎 Log file uploaded as Gist (1010KB)
  🔗 View complete solution draft log

---

Now working session is ended, feel free to review and add any feedback on the solution draft.

[lion-lef]

Save a database to data with api. Is now not saved

[konard]

🤖 AI Work Session Started

Starting automated work session at 2026-02-01T19:16:35.628Z

The PR has been converted to draft mode while work is in progress.

This comment marks the beginning of an AI work session. Please wait working session to finish, and provide your feedback.

[konard]

Fixed the data saving issue in commit 61e327c. Here's what was wrong and what changed:

Root causes:

1. Missing UNIQUE index on id_post — The init migration created the Task table but didn't include the UNIQUE constraint on id_post that's defined in schema.prisma. This meant skipDuplicates: true in Prisma's createMany couldn't properly detect duplicates. Fixed by adding CREATE UNIQUE INDEX "Task_id_post_key" ON "Task"("id_post") to the migration.

2. API only saved to database, not to files — The /api/crawl endpoint saved posts to the database but NOT to the data/ folder. If the DB save failed (silently caught), no data was persisted anywhere. Now saves to both database and files.

3. API response hid crawled posts on DB error — The old code only included posts in the API response if the DB save succeeded (allCollected.push(...newPosts.slice(0, savedCount))). If savedCount=0 due to a DB error, the response was "No new posts found" even though posts were crawled. Now all crawled posts are always returned.

4. CLI (npm run crawl) didn't save to database — The crawl-only.ts script only saved to files via DataStorage. Now it also saves to the database when DATABASE_URL is available.

Changes:

• prisma/migrations/20260125085858_init/migration.sql — Added UNIQUE index on id_post
• src/index.ts — API crawl saves to both DB and files; response always includes crawled posts
• src/crawler/crawl-only.ts — CLI crawl also saves to database

Note: If you're running the existing database, you'll need to either:

• Run npx prisma migrate deploy to apply the updated migration (on a fresh DB), or
• Manually add the index: CREATE UNIQUE INDEX IF NOT EXISTS "Task_id_post_key" ON "Task"("id_post");

[konard]

🤖 Solution Draft Log

This log file contains the complete execution trace of the AI solution draft process.

💰 Cost estimation:

• Public pricing estimate: $6.348995 USD
• Calculated by Anthropic: $3.790383 USD
• Difference: $-2.558612 (-40.30%)
  📎 Log file uploaded as Gist (887KB)
  🔗 View complete solution draft log

---

Now working session is ended, feel free to review and add any feedback on the solution draft.

[lion-lef]

Run of container, table is not created and empty. Fix please

[konard]

🤖 AI Work Session Started

Starting automated work session at 2026-02-01T19:38:05.458Z

The PR has been converted to draft mode while work is in progress.

This comment marks the beginning of an AI work session. Please wait working session to finish, and provide your feedback.

[konard]

Fixed the table creation issue in commit 4f3710c. The root cause is that prisma migrate deploy can fail in Docker containers with Prisma 7 due to TypeScript config resolution issues (the prisma.config.ts file needs to be loaded at runtime by the Prisma CLI).

What changed:

1. TaskService.ensureTablesExist() (src/services/database/task.ts) — New method that checks if the Task table exists using SELECT to_regclass('public."Task"'), and if it doesn't, creates it via raw SQL (CREATE TABLE IF NOT EXISTS + CREATE UNIQUE INDEX IF NOT EXISTS). This uses the same pg Pool that the service already creates for the Prisma adapter.

2. App startup (src/index.ts) — Calls ensureTablesExist() immediately when the app starts, before any API requests can hit the database.

3. CLI crawl (src/crawler/crawl-only.ts) — Also calls ensureTablesExist() before attempting to save to the database.

4. Dockerfile — The CMD now uses || instead of && for the migration step, so if prisma migrate deploy fails, the app still starts and the SQL fallback creates the table.

This means even if Prisma migrations can't run (wrong database state, TS config issues, missing migration history), the app will self-heal by creating the table on startup.

[konard]

🤖 Solution Draft Log

This log file contains the complete execution trace of the AI solution draft process.

💰 Cost estimation:

• Public pricing estimate: $4.374411 USD
• Calculated by Anthropic: $2.971525 USD
• Difference: $-1.402886 (-32.07%)
  📎 Log file uploaded as Gist (858KB)
  🔗 View complete solution draft log

---

Now working session is ended, feel free to review and add any feedback on the solution draft.

[lion-lef]

5.878 Error: P1013: The provided database string is invalid. The scheme is not recognized in database URL. Please refer to the documentation in https://pris.ly/d/config-url for constructing a correct connection string. In some cases, certain characters must be escaped. Please check the string for any illegal characters.

[+] up 0/1
⠙ Image tgcrawler-telegram-scraper Building 20.8s
Dockerfile:31

---

29 | RUN npm run prisma:generate

30 |

31 | >>> RUN npx prisma migrate deploy

32 |

33 | RUN npm run build

---

failed to solve: process "/bin/sh -c npx prisma migrate deploy" did not complete successfully: exit code: 1

[lion-lef]

Is changed docker file(npx generate before start and build app please place)

[konard]

🤖 AI Work Session Started

Starting automated work session at 2026-02-01T19:49:25.152Z

The PR has been converted to draft mode while work is in progress.

This comment marks the beginning of an AI work session. Please wait working session to finish, and provide your feedback.

[konard]

Fixed the Docker build failure in commit e853a57.

Root cause: RUN npx prisma migrate deploy cannot run during docker build because the PostgreSQL database isn't available at build time — it only starts at runtime via docker-compose. The P1013: The provided database string is invalid error happens because DATABASE_URL is empty/missing during the build phase.

Fix: Created a docker-entrypoint.sh startup script that runs at container start (not during build):

1. npx prisma generate    → ensures Prisma client matches current schema
2. npx prisma migrate deploy → applies database migrations (with fallback if it fails)
3. node dist/index.js     → starts the app

The Dockerfile build phase now only does:

• npm run prisma:generate (line 29) — generates Prisma client for TypeScript compilation
• npm run build (line 31) — compiles TypeScript to JavaScript

Both steps work without a database connection.

Important: Do NOT add RUN npx prisma migrate deploy to the Dockerfile — it will always fail during build because there's no database available. Migrations must run at startup time.

[konard]

🤖 Solution Draft Log

This log file contains the complete execution trace of the AI solution draft process.

💰 Cost estimation:

• Public pricing estimate: $2.948948 USD
• Calculated by Anthropic: $2.061593 USD
• Difference: $-0.887355 (-30.09%)
  📎 Log file uploaded as Gist (410KB)
  🔗 View complete solution draft log

---

Now working session is ended, feel free to review and add any feedback on the solution draft.