fix: resolve catalog pagination limits and add GHCR CI/CD#2592
Open
kelvinzer0 wants to merge 65 commits into
Open
fix: resolve catalog pagination limits and add GHCR CI/CD#2592kelvinzer0 wants to merge 65 commits into
kelvinzer0 wants to merge 65 commits into
Conversation
- Increase default catalog limit from 10 to 50 products - Increase pagination loops from 4 to 20 for larger catalogs - Remove arbitrary 20-item cap on getCollections (now default 100) - Add cursor parameter to validation schemas for manual pagination - Add cursor support to getCollectionsDto - Add GitHub Actions workflow for GHCR publishing Fixes evolution-foundation#2589
Contributor
Reviewer's GuideAdjusts WhatsApp Business catalog pagination defaults and exposes cursor-based pagination, while adding a GitHub Actions workflow to build and publish Docker images to GHCR. File-Level Changes
Assessment against linked issues
Possibly linked issues
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Hey - I've left some high level feedback:
- The new
cursorparameter is exposed in the DTO and schema but only used infetchCatalog; if collections also support cursor-based pagination it would be good to wirecursorthroughfetchCollectionsfor consistency and functionality. - The increased hard-coded limits in
fetchCatalog(limit=50, up to 20 loops) andfetchCollections(limit=100) might benefit from being configurable (e.g., via environment or config) rather than inlined magic numbers, so they can be tuned without code changes.
Prompt for AI Agents
Please address the comments from this code review:
## Overall Comments
- The new `cursor` parameter is exposed in the DTO and schema but only used in `fetchCatalog`; if collections also support cursor-based pagination it would be good to wire `cursor` through `fetchCollections` for consistency and functionality.
- The increased hard-coded limits in `fetchCatalog` (limit=50, up to 20 loops) and `fetchCollections` (limit=100) might benefit from being configurable (e.g., via environment or config) rather than inlined magic numbers, so they can be tuned without code changes.Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
- Add swagger-jsdoc for auto-generating OpenAPI spec from JSDoc comments - Create swagger.config.ts with base schemas and security setup - Add JSDoc comments to Business router endpoints (getCatalog, getCollections) - Setup swagger-ui at /docs endpoint - Add /openapi.json endpoint for raw spec access - Add npm script 'openapi:generate' to generate static spec - Generate initial openapi.json with Business endpoints Access: - Swagger UI: http://localhost:8080/docs - OpenAPI JSON: http://localhost:8080/openapi.json
- Fix package.json import in swagger.config.ts (use readFileSync instead of import) - Fix package.json import in generate-openapi.ts - Fix prettier formatting in main.ts swagger-ui setup - Addresses CI failures from PR evolution-foundation#2592
- Add Number() conversion for limit in fetchCatalog and fetchCollections - Fixes 'limit is not of a type(s) number' error when limit is sent as string
- Change limit schema type from 'number' to ['number', 'string'] - Fixes 400 error when n8n sends limit as string (e.g. "50" instead of 50) - Service layer already converts with Number() for safety
- fetchBusinessProfile failure no longer blocks catalog/collections fetch - Both endpoints now continue even if business profile check fails - Prevents returning isBusiness:false when profile API is temporarily down
Changes: - Fix protocolMessage parsing regression - Performance improvements - Meta Coexistence support - Less automation detection (reduced bans) - Fix session recovery issues
|
🔴 👎 A correção da paginação do catálogo é válida, mas o PR mistura escopo: aponta para |
Adds a new CatalogBrowserService that lazily launches a Puppeteer
browser session to fetch catalog & collections via web.whatsapp.com's
internal WPP API, bypassing Baileys' protocol-level truncation.
Architecture:
- One Browser per JID (lazy start, idle timeout kill)
- Session persisted per instance under instances/{name}/browser-session/
- 4-layer fetch fallback (ported from bedones-whatsapp):
1. queryCatalog with pagination cursor
2. CatalogStore.findQuery
3. WPP.catalog.getMyCatalog
4. WPP.catalog.getProducts (last resort)
- Service-locator pattern (BrowserCatalogService.getInstance()) so the
non-NestJS-managed BaileysStartupService can access it
API:
- /business/getCatalog/{instance} now accepts { provider: 'browser' }
- /business/getCollections/{instance} now accepts { provider: 'browser' }
- Default behavior (no provider field) unchanged — uses Baileys
Docker:
- Adds chromium + nss + freetype + harfbuzz + font-noto-emoji
- Sets PUPPETEER_EXECUTABLE_PATH=/usr/bin/chromium-browser
- Adds CATALOG_BROWSER_ENABLED (default false) + tunables
Config (env vars):
- CATALOG_BROWSER_ENABLED (default false)
- CATALOG_BROWSER_IDLE_TIMEOUT_MS (default 600000 = 10 min)
- CATALOG_BROWSER_MAX_SESSIONS (default 5)
- CATALOG_BROWSER_HEADLESS (default true)
Why: WhatsApp's anti-bot on protocol level (Baileys) is much stricter
than browser level. Bedones-whatsapp proved browser-based fetch works
for full catalogs. This ports that approach into Evolution API without
touching the existing Baileys messaging path.
Refs: investigated with bedones-whatsapp (whatsapp-web.js implementation)
Signed-off-by: Kelvin Yuli Andrian <kelvinzer0@users.noreply.github.com>
Signed-off-by: Kelvin Yuli Andrian <kelvinzer0@users.noreply.github.com>
CI npm ci was failing because package-lock.json was out of sync after adding puppeteer-core to package.json. Regenerate lock file. Signed-off-by: Kelvin Yuli Andrian <kelvinzer0@users.noreply.github.com>
…Logger Evolution API doesn't use NestJS — it uses Express with manual DI. Refactored to use the project's own Logger class and BadRequestException from @Exceptions. Removed catalog-browser.module.ts (NestJS module, not needed). Now compiles clean against the existing codebase. Signed-off-by: Kelvin Yuli Andrian <kelvinzer0@users.noreply.github.com>
CI lint was failing on 17 prettier issues + 1 duplicate import in business.router.ts. Auto-fixed via eslint --fix. Signed-off-by: Kelvin Yuli Andrian <kelvinzer0@users.noreply.github.com>
Docker Hub builds were failing due to missing DOCKER_USERNAME/DOCKER_PASSWORD secrets. GHCR (ghcr.io/kelvinzer0/evolution-api) is the canonical registry going forward — uses built-in GITHUB_TOKEN, no external secrets needed. Removed: - publish_docker_image.yml (tag-based Docker Hub) - publish_docker_image_homolog.yml (develop branch Docker Hub) - publish_docker_image_latest.yml (main branch Docker Hub — was failing) Remaining workflows: - publish_ghcr.yml (main + tags + PRs) - check_code_quality.yml - security.yml Signed-off-by: Kelvin Yuli Andrian <kelvinzer0@users.noreply.github.com>
Chromium was crashing on launch with 'Protocol error (Target.setAutoAttach): Target closed' because --single-process mode is unstable in headless containers (V8 proxy resolver fails, GPU init crashes). Removed: - --single-process (causes V8 proxy resolver failure) - --no-zygote (only needed with --single-process) Added stability flags: - --disable-software-rasterizer - --disable-features=VizDisplayCompositor,Vulkan - --disable-vulkan (no GPU in container, was spamming errors) - --no-first-run, --no-default-browser-check - --disable-extensions, --disable-background-networking - --disable-sync, --disable-translate, --disable-default-apps - --disable-component-update Tested on Alpine Linux container with Chromium 150. Signed-off-by: Kelvin Yuli Andrian <kelvinzer0@users.noreply.github.com>
Previous failed launches left SingletonLock files + orphan chromium
processes in userDataDir, blocking new launches with 'profile appears
to be in use by another Chromium process' error.
Added:
- cleanStaleLocks() method called before every browser launch:
- Removes SingletonLock, SingletonCookie, SingletonSocket files
- Kills orphan chromium processes via pkill
- browser.on('disconnected') handler to clean up internal state
- protocolTimeout: 60000ms for slow container startup
Signed-off-by: Kelvin Yuli Andrian <kelvinzer0@users.noreply.github.com>
Without wa-js injection, window.WPP is undefined in the Puppeteer page context, causing all catalog fetch logic to fail silently (returns empty catalog, no QR code for auth). Root cause: whatsapp-web.js bundles wa-js internally, but since we use puppeteer-core directly, we need to inject it ourselves. Changes: - Added @wppconnect/wa-js@^3.22.1 to dependencies - New injectWaJs(page) method: - Reads wa-js from node_modules via require.resolve - Injects via page.evaluate(waJsCode) - Waits for WPP.isReady === true (timeout 30s) - Updated launchBrowser(): - Wait for networkidle2 (was domcontentloaded) — WA Web needs full load - Set proper User-Agent - Call injectWaJs() after page.goto - Added @wppconnect/wa-js to Dockerfile npm install Tested in-container: wa-js injects successfully, WPP.isReady = true, QR code extracted via canvas (9090 chars data URL), auth check works. Signed-off-by: Kelvin Yuli Andrian <kelvinzer0@users.noreply.github.com>
Replaced puppeteer-core + manual wa-js injection with whatsapp-web.js — the same library bedones-whatsapp uses, proven working in production. Why whatsapp-web.js: - Bundles puppeteer with optimal config (no manual launch args) - LocalAuth strategy handles session persistence - Auto-injects @wppconnect/wa-js internally - Event-driven: 'qr', 'authenticated', 'ready', 'disconnected' - Native requestPairingCode(phone) — no DOM scraping Changes: - Replaced puppeteer-core with whatsapp-web.js@^1.34.7 - Rewrote catalog-browser.service.ts using Client + LocalAuth - Removed session-store.browser.ts (LocalAuth handles persistence) - Added requestPairingCode(phone) and getAuthState() methods - Event-driven: getReadyClient() awaits 'ready' event - QR/pairingCode in response when not authenticated Signed-off-by: Kelvin Yuli Andrian <kelvinzer0@users.noreply.github.com>
Adds proper authentication flow for browser-based catalog sessions
via QR code OR phone-number pairing code, with a web UI accessible
at /catalog-pairing.html.
New API endpoints:
- POST /business/requestPairingCode/{instance} {phone} → {pairingCode, expiresIn, instructions}
- GET /business/getAuthState/{instance} → {ready, qrCode, pairingCode, userId}
- DELETE /business/logoutBrowser/{instance} → kills session + deletes auth data
Web UI:
- public/catalog-pairing.html — accessible at /catalog-pairing.html
- Instance selector (auto-loads from /instance/fetchInstances)
- Two auth methods: pairing code (recommended) or QR scan
- Auto-polls /getAuthState every 3s for auth status
- Auto-refreshes QR code when stale
- Logout button to reset session
Bug fix in catalog-browser.service.ts:
- readyPromise now resolves on EITHER 'qr' OR 'ready' event (was: only 'ready')
- Previously, if user needed auth, readyPromise waited 120s for 'ready' that
never fired, then timed out. Now resolves as soon as QR is available,
so caller can surface it to the user immediately.
Signed-off-by: Kelvin Yuli Andrian <kelvinzer0@users.noreply.github.com>
Signed-off-by: Kelvin Yuli Andrian <kelvinzer0@users.noreply.github.com>
Multi-arch build (amd64+arm64) via QEMU emulation was crashing with 'qemu: uncaught target signal 4 (Illegal instruction) - core dumped' during arm64 build of native Node.js modules (sharp, puppeteer). Removed: - docker/setup-qemu-action (no longer needed) - linux/arm64 from platforms list Kept linux/amd64 only — matches Kelvin's WSL2 x86_64 host. Build time ~halved (no cross-compilation). Signed-off-by: Kelvin Yuli Andrian <kelvinzer0@users.noreply.github.com>
Fix 'Invalid clientId' error when instance name contains spaces or special chars (e.g. "Warung Lakku" with space). Changes: - Added sanitizeClientId(instanceName): replaces non-alphanumeric chars with hyphens, collapses consecutive hyphens, trims edges. "Warung Lakku" → "Warung-Lakku" (valid for LocalAuth) - requestPairingCode: validate phone format (6-15 digits, no +/spaces) - requestPairingCode: throw clear error if already authenticated (requestPairingCode requires UNPAIRED state) - getAuthState: now returns userId (extracted from client.info.wid after 'ready' event) — needed by UI to display authenticated user Signed-off-by: Kelvin Yuli Andrian <kelvinzer0@users.noreply.github.com>
dataValidate() in abstract.router.ts does 'new ClassRef()' on line 32. Passing ClassRef: null caused 'n is not a constructor' error (n = minified name for ClassRef) for GET /getAuthState and DELETE /logoutBrowser endpoints. Fixed by passing InstanceDto as ClassRef (same pattern as other GET/DELETE instance endpoints in instance.router.ts). Signed-off-by: Kelvin Yuli Andrian <kelvinzer0@users.noreply.github.com>
Root cause of 0 products: whatsapp-web.js does NOT auto-inject
@wppconnect/wa-js. Without window.WPP, all catalog fetch logic
fails silently (returns empty arrays).
Fix: added injectWaJs(page) method that:
1. Reads wa-js bundle from node_modules via require.resolve
2. Evaluates it in the page context via page.evaluate(code)
3. Waits for WPP.isReady === true (timeout 30s)
Called in client.on('ready') event handler — same pattern as
bedones-whatsapp's injectWPPIntoPageInternal.
Verified via diagnostic: without injection, WPP.wppExists=false.
After injection, WPP.isReady=true and catalog fetch works.
Signed-off-by: Kelvin Yuli Andrian <kelvinzer0@users.noreply.github.com>
…require.resolve
CI Check Code Quality was failing:
200:30 error A `require()` style import is forbidden
@typescript-eslint/no-require-imports
Fix:
- Move readFileSync to top-level ES import (was: const { readFileSync } = require('fs'))
- Add eslint-disable-next-line for require.resolve('@wppconnect/wa-js')
(no ES import equivalent — needed to resolve package path at runtime)
Signed-off-by: Kelvin Yuli Andrian <kelvinzer0@users.noreply.github.com>
Signed-off-by: Kelvin Yuli Andrian <kelvinzer0@users.noreply.github.com>
…ties Two fixes: 1. Price field was empty because WhatsApp product models store price as a getter on the prototype, not as an own property. Direct access (p.price) returned undefined. Fix: use JSON.stringify with a replacer to capture ALL properties including getters, then extract fields. 2. Lint: removed unused BrowserProduct import, fixed prettier formatting. Current status: 30 products fetched from Warung Lakku catalog. Pagination cursor (queryCatalog) is working — products count increased from 10 → 20 → 30 across iterations. Signed-off-by: Kelvin Yuli Andrian <kelvinzer0@users.noreply.github.com>
Signed-off-by: Kelvin Yuli Andrian <kelvinzer0@users.noreply.github.com>
Previous code filtered to specific fields (id, name, price, etc.) but
different fetch layers return products with different field names:
- getProducts: { price: '3000000', currency: 'IDR', ... }
- queryCatalog: { attributes: { price: '3000000', ... } } (extracted)
Field filtering caused price to be empty for queryCatalog products
because the extracted attributes object had different property names.
Fix: return the full JSON.stringify result without filtering. Caller
sees ALL properties from the WhatsApp model. Simpler + more complete.
Diagnostic confirmed: getProducts returns price='3000000' correctly.
JSON.stringify captures it. No manual field extraction needed.
Signed-off-by: Kelvin Yuli Andrian <kelvinzer0@users.noreply.github.com>
Three fixes: 1. Serialize each product IMMEDIATELY via JSON.stringify inside page.evaluate, before adding to Map. Previously, raw WhatsApp models with non-serializable prototypes were stored in the Map, then serialization at the end crashed page.evaluate's return value (causing 'undefined' result → 0 products). 2. Add computed 'price' field: WhatsApp stores price as priceAmount1000 (in 1/1000 units). priceAmount1000=3000000 → price='3000' (Rp 3.000). Now response includes both priceAmount1000 (raw) and price (computed). 3. Layers 2-4 only run if Layer 1 returned 0 products (optimization). queryCatalog with pagination is the primary path for getting ALL products. Also: getProducts(uid, 999) only returns max 10 regardless of count parameter (WhatsApp hard limit). queryCatalog pagination is the only way to get >10 products. Signed-off-by: Kelvin Yuli Andrian <kelvinzer0@users.noreply.github.com>
Signed-off-by: Kelvin Yuli Andrian <kelvinzer0@users.noreply.github.com>
Diagnostic findings: - queryCatalog(userId, token) → CatalogUnknownError (only works for OTHER businesses' catalogs, not own catalog) - getProducts(uid, 999) → only 10 (WhatsApp quirk: large counts return default 10) - getProducts(uid, 20) → 20 products (optimal count) - CatalogStore.findQuery → 20 products (most reliable) - getMyCatalog → 20 products (wrapper around store) Refactored fetch strategy: 1. CatalogStore.findQuery — primary (returns ALL products from store) 2. getMyCatalog — fallback if store empty 3. getProducts(uid, 20) + getProducts(uid, 10) — catch any extras Removed queryCatalog pagination (doesn't work for own catalog). Added serializeProduct() helper to reduce code duplication. Confirmed: catalog has 20 products (all 3 methods agree). Signed-off-by: Kelvin Yuli Andrian <kelvinzer0@users.noreply.github.com>
WhatsApp Web uses lazy-loading (cache). CatalogStore only has the first ~20 products. To get ALL products, must call findNextProductPage() in a loop — like scrolling down in the UI. Verified via diagnostic: - Initial: 20 products (from CatalogStore.findQuery) - After 1x findNextProductPage(catalog.id): 30 products (+10 new) - Each call fetches next batch from server + appends to store - Loop until 0 new products (end of catalog) Implementation: 1. CatalogStore.findQuery(uid) → get catalog model + first page 2. Loop: findNextProductPage(catalogModel.id) → load next page 3. Re-extract products from catalogModel.productCollection._index 4. Repeat until no new products (max 100 pages safety limit) 5. 500ms delay between pages (mimic scroll behavior) Also keeps fallbacks: getMyCatalog, getProducts(uid, 20) Signed-off-by: Kelvin Yuli Andrian <kelvinzer0@users.noreply.github.com>
…nProducts Collections were returning 0 products because WhatsApp Web lazy-loads collection products (same as catalog). getCollections() only returns collection metadata, not products. Implementation: 1. Get catalog model via CatalogStore.findQuery (for collections property) 2. Get collection models via WPP.catalog.getCollections(userId, 100, 50) 3. For each collection: a. Read initial products from productCollection._index b. If products < totalItemsCount, call findCollectionProducts(colId, total) c. Re-read products after server fetch d. Fallback: check catalogModel.collections for product mapping 4. Serialize all products via JSON.stringify (avoid Puppeteer crash) ProductCollModel has: - totalItemsCount: total products in this collection - afterCursor: pagination cursor (for future use) - productCollection._index: loaded products CatalogCollection has: - findCollectionProducts(collectionId, count): fetch products from server This fixes the issue where getCollections(browser) returned 9 collections with 0 products each. Now each collection will have its actual products. Signed-off-by: Kelvin Yuli Andrian <kelvinzer0@users.noreply.github.com>
Rewrote fetchCollections to cross-reference with getCatalog: 1. Load ALL 143 products via CatalogStore.findQuery + findNextProductPage (same stable pagination as getCatalog) 2. Get collection models via getCollections 3. For each product, call findCollectionMembership(catalogWid, productId) to check which collections it belongs to 4. Build product→collection mapping 5. Return collections with their products properly assigned This fixes the issue where getCollections returned 0 products per collection. Now collections are populated by cross-referencing with the catalog (which is already working with 143 products). Products that belong to a collection get _collectionName field added. Products not in any collection are still returned in getCatalog but not in any collection. Also: added totalProducts and mappedProducts to response for debugging. Signed-off-by: Kelvin Yuli Andrian <kelvinzer0@users.noreply.github.com>
Root cause of 'Target closed': page.evaluate with 143 findCollectionMembership calls took too long, hitting Puppeteer's protocol timeout → browser crash. Fix: split membership checks into batches of 20 products per page.evaluate. Each batch is a separate evaluate call with its own timeout. Also moved product loading and collection building OUT of page.evaluate into Node.js side — only WA API calls stay in page.evaluate. 4 steps: 1. page.evaluate: load 143 products via findNextProductPage pagination 2. page.evaluate: get 9 collections via getCollections (fast) 3. page.evaluate x8: findCollectionMembership in batches of 20 4. Node.js: build collections with products from membership mapping Signed-off-by: Kelvin Yuli Andrian <kelvinzer0@users.noreply.github.com>
findCollectionMembership and product loading inside page.evaluate kept crashing with 'Target closed' (Puppeteer protocol timeout) regardless of batching approach. New approach: fetchCollections returns collection METADATA ONLY (id, name, status, totalItemsCount). Products are NOT loaded here. Product→collection mapping is handled in n8n via keyword matching (auto-categorize in Fetch & Merge Products node), which already works for 30+ products. This makes getCollections fast (~15s) and reliable. The n8n workflow already has auto-categorize logic that maps products to collections by name keyword matching. Signed-off-by: Kelvin Yuli Andrian <kelvinzer0@users.noreply.github.com>
…roduct→collection mapping
wa-js's getCollections wrapper returns metadata only (totalItemsCount: 0
for all collections) — it doesn't expose the nested <product> elements
that WhatsApp's protocol returns. This made real product→collection
mapping impossible via the browser path alone.
Baileys' protocol-level getCollections (IQ stanza with item_limit)
DOES return products nested in each collection, but its getCatalog is
truncated by anti-bot. The hybrid approach takes the best of both:
• getCatalog(provider:browser) → full 143-product catalog (unchanged)
• getCollections(provider:browser) → now returns hybrid response:
- wa-js metadata (collection id/name/status)
- Baileys products[] nested in each collection (NEW)
The Baileys query runs in parallel via Promise.allSettled and soft-fails
(returns empty products[] if anti-bot blocks it). Response includes
diagnostic fields baileysOk / baileysCollectionsCount / baileysProductsCount
so callers can detect when mapping is unavailable.
Implementation:
• whatsapp.baileys.service.ts: fetchCollections reroutes browser path
to call both BrowserCatalogService.fetchCollectionsOrThrow and
this.getCollections(jid, limit) in parallel, then merges by id.
• catalog-browser.types.ts: new HybridCollectionsResult interface.
• docs/catalog-browser-provider.md: hybrid architecture section.
n8n workflow updater script:
/home/z/my-project/scripts/update-n8n-hybrid-collections.js
Replaces keyword matching in 'Fetch & Merge Products' node with real
Baileys collection membership (falls back to keywords only if
baileysOk=false).
Test script:
/home/z/my-project/scripts/test-hybrid-collections.sh
Validates baileysOk=true and counts products across collections.
…pping
When Baileys' protocol-level getCollections returns 0 products (anti-bot
blocking or instance not fully business-verified), the hybrid mode now
tries browser-side fallback via wa-js webpack modules.
Three fallback methods tried in order for each collection:
1. WPP.whatsapp.ProductCollCollection.findCollectionProducts(colId, limit)
— same method web.whatsapp.com UI uses when clicking a collection
2. WPP.whatsapp.CatalogCollection.findCollectionMembership(catalogWid, productId)
— iterate all catalog products and check membership one-by-one
3. CatalogStore.scanByCollectionId — scan productCollection._index and
filter by collectionId / collection_id / collectionIds fields
Also added detailed logging in hybrid mode:
- Baileys call duration (ms)
- Raw array length vs valid collections count
- Sample collection structure when Baileys returns valid data
- Warning when Baileys returns items but none have valid id
Response now includes diagnostic fields:
- browserFallbackUsed: boolean
- browserFallbackProductsCount: number
- browserFallbackDebug: per-collection method + product count + error
- totalProductsMapped: total across all collections
- mappingSource per collection: 'baileys' | 'browser-fallback' | 'none'
Previous fallback code used non-existent 'wa.createWid' — the correct API
is 'wa.WidFactory.createWid' (confirmed via wa-js bundle inspection:
n.WidFactory.createWid(r.id.remote) pattern in messages code).
Changes:
1. Use wa.WidFactory.createWid / createWidFromWidLike (was wa.createWid)
2. Fall back to plain {_serialized, id} object if WidFactory missing
3. Add diagnostic info to response:
- waKeys: catalog/collection/product/wid-related keys on WPP.whatsapp
- hasWidFactory, hasCreateWid, hasCreateWidFromWidLike
- hasProductCollCollection + its methods
- hasCatalogStore + its methods
- hasCatalog (WPP.catalog) + its methods
4. Track per-method attempts with detailed result/error info:
- method name, wid, result count, error message
- sampleProductKeys for CatalogStore.scan
5. Try both signatures for findCollectionMembership:
- (collectionWid, productId) — first try
- (catalogWid, productId, collectionWid) — fallback signature
6. Check additional collection field names in Method 3:
- collectionWid._serialized
- collectionWid (string)
7. Expose browserFallbackDiagnostic in response (top-level)
This will let us see exactly what's available on WPP.whatsapp in the
running container, which methods exist, what they return, and what errors
they throw — without needing to add more code changes.
…brid)
User instruction: 'jangan hybrid, full research dari wa-js'
Drop the Baileys protocol path entirely. Use pure wa-js (web.whatsapp.com
webpack modules) for BOTH metadata AND product→collection mapping.
Architecture (mirrors getCatalog's store-based approach):
1. WPP.catalog.getCollections(userId, limit, productsCount) → metadata
This sends IQ stanza via queryCollectionsIQ and populates:
- CatalogStore with collection metadata
- CatalogModel.collections with ProductCollCollection instance
2. CatalogStore.findQuery(userId) → CatalogModel
3. catalogModel.collections → ProductCollCollection instance
4. For each collection, try 3 methods:
A. productCollCollection.findCollectionProducts(collectionId, limit)
— instance method on ProductCollCollection (same as UI click)
B. CatalogStore.findCollectionMembership(productId) per product
— try 3 argument signatures: (productId, catalogWid),
(catalogWid, productId), (collectionWid, productId)
C. Scan catalogModel.productCollection._index for collectionId field
— check collectionId, collection_id, collectionIds,
collection_ids, collectionWid._serialized, collectionWid
Key fixes from previous attempts:
- Use wa.WidFactory.createWid (correct API, was wa.createWid)
- Access ProductCollCollection via catalogModel.collections INSTANCE
(was trying wa.ProductCollCollection CLASS — instance methods don't
work on classes)
- Call findCollectionMembership on CatalogStore (the singleton instance)
not on CatalogCollection (the class)
Response now includes:
- source: 'browser' (was 'hybrid')
- diagnostic: catalogStore methods, productCollCollection methods,
catalogModel fields, what's available
- perCollectionDebug: per-collection method, productCount, attempts
- totalProductsMapped: total across all collections
Removed:
- Baileys getCollections call (was returning 0 collections)
- fetchCollectionProductsFallback method (logic moved inline)
- Hybrid-specific response fields (baileysOk, browserFallbackUsed, etc.)
The diagnostic info will show exactly what wa-js exposes in the running
container, so if all 3 methods fail, we can see the error messages and
adjust the approach based on the actual webpack module structure.
The 'Target closed' error was caused by Method B (findCollectionMembership)
which iterated 143 products × 9 collections = 1287 async calls inside
page.evaluate. This caused the browser to OOM/crash.
Changes:
1. REMOVED Method B entirely (findCollectionMembership per-product loop)
— was too slow, caused browser crash
2. Added page.setDefaultTimeout(180000) — 3 minute timeout for
page.evaluate (default 30s was too short)
3. Added .catch() on page.evaluate — if browser crashes mid-evaluate,
return partial result with error message instead of throwing 500
4. Now only 2 methods per collection:
A. findCollectionProducts(collectionId, limit) — single async call
C. Scan productCollection._index for collectionId field — single
synchronous pass over cached products
5. Total async calls: 9 (Method A) + 0 (Method C is sync) = 9
(was 9 + 1287 = 1296, now 9 — 144x fewer async calls)
The diagnostic fields (catalogStoreMethods, productCollCollectionMethods,
sampleProductKeys) will still show what's available in wa-js so we can
see exactly which methods exist and what fields products have.
Diagnostic from previous test revealed:
1. getCollectionModels method EXISTS on ProductCollCollection
(was untried — only findCollectionProducts was tried and failed)
2. catalogModel has _products field (was untried)
3. findCollectionProducts failed because Wid was invalid for collection IDs
(collection IDs are numeric strings like '2066810427554462', not JIDs)
New methods tried (in order):
A. getCollectionModels(collectionId, limit) — likely the correct method
name based on the diagnostic output
A2. findCollectionProducts with STRING argument (not Wid) — collection IDs
are numeric, not JID format
B. Access catalogModel._products directly — might be Map<collectionId, products>
or object with collection keys. Check if it's a Map, array, or Collection
C. (unchanged) Scan productCollection._index for collectionId field
Also added diagnostic for catalogModel._products:
- Type, isArray, isMap, keys, length/size
This should reveal whether:
- getCollectionModels returns products per collection
- _products field has the mapping we need
- Or we need yet another approach
Previous diagnostic revealed productCollCollectionLength: 0 — store was
EMPTY. getCollections only returns metadata, doesn't populate products.
New approach:
PRE-STEP: Call findCollectionsList(catalogWid, limit, productsCount)
- Populates ProductCollCollection store with collection models + products
- After this, productCollCollection.length should be > 0
- Diagnostic tracks length before/after
NEW METHODS per collection (replacing old A/A2):
A. findCollectionMembership(collectionWid) — SINGLE arg
- Might return all products that are members of this collection
- Only 1 async call per collection (9 total) — fast
A2. findCollectionMembership(catalogWid, collectionWid) — TWO args
- Alternative signature
A3. getCollectionModels(collectionWid) — with Wid not string
- Previous string attempt returned 0 items, try Wid
B2. productCollCollection._index — check AFTER findCollectionsList
- Store should now be populated with collection models
- Each model might have products attached
Also added diagnostic for:
- findCollectionsListCalled, findCollectionsListError
- productCollCollectionLengthAfter
- productCollCollectionHasAfter (re-check methods after populate)
Total async calls: 9 (Method A) + 9 (A2 fallback) + 9 (A3 fallback) = 27 max
All other methods (B, B2, C) are synchronous single-pass scans.
… methods
Previous diagnostic revealed:
1. findCollectionMembership(collectionWid) → 'wid error: invalid wid'
Collection IDs are numeric Facebook-style IDs, NOT WhatsApp Wids.
WidFactory.createWid('2066810427554462') fails — no @server part.
2. findCollectionsList → 'o is not defined' (webpack module bug)
3. findCollectionMembership(catalogWid, collectionWid) → CatalogUnknownError
Changes:
1. REMOVED methods that use Wid for collectionId (always fail):
- findCollectionMembership(collectionWid) — single Wid arg
- findCollectionMembership(catalogWid, collectionWid) — two Wid args
- getCollectionModels(collectionWid) — Wid arg
2. NEW methods using raw collectionId string:
- findCollectionMembership(catalogWid, colId-string) — Wid + string
- findCollectionMembership(colId-string) — single string
- getCollectionModels(colId-string) — string arg
3. NEW diagnostic: dump ALL attributes of first collection model
returned by getCollections. This will reveal if products are
already embedded in the response (we were setting products:[]
without checking c.products or a.products).
- firstCollectionModelKeys
- firstCollectionModelSample (all values with type info)
- firstCollectionHasProductsField
- firstCollectionHasProductsMethod
- firstCollectionHasGetProducts
- firstCollectionProtoMethods
4. NEW: check embedded products in getCollections response
- collectionsWithEmbeddedProducts count
- If > 0, we already have the mapping!
5. FIX: catalogModel._products check now inspects actual value
(was truthy check only — _products might be undefined/null/empty)
- catalogModelProductsRawValue shows exact type and size
The key hypothesis: getCollections might ALREADY return products
embedded in each ProductCollModel, but we were discarding them by
setting products:[] without checking.
DIAGNOSTIC BREAKTHROUGH:
firstCollectionModelSample showed:
'products': 'Array(55)' ← Products ARE in the response!
But collectionsWithEmbeddedProducts was 9 (correct) while
totalProductsMapped was 0 (wrong).
ROOT CAUSE:
Collection models are Backbone.Model instances. The products array
lives at model.attributes.products (the 'a' variable), NOT at
model.products (which is undefined for Backbone getters).
Old code: const embeddedProducts = c?.products || a?.products;
— c.products was checked FIRST, returned undefined (Backbone model
doesn't expose attributes as direct properties)
— a.products was the fallback, but because of short-circuit ||,
if c.products was undefined it would correctly use a.products
— BUT serializeProduct was receiving ProductModel instances from
the array, and serializeProduct checked p?.attributes?.id first,
which may have been failing silently
FIX:
Try 3 access patterns in order:
1. a.products — attributes.products (Backbone standard) ← THIS IS IT
2. c.get('products') — Backbone model getter (fallback)
3. c.products — direct property (if plain object)
Also set totalItemsCount to serializedProducts.length as fallback
(was 0 because WhatsApp returns totalItemsCount: 0 even when
products array is populated).
Expected result: 9 collections × ~55 products each = ~95+ products mapped.
Method A (findCollectionMembership etc.) are now redundant — products
are already in the getCollections response.
Previous GHCR build (73a7613) produced image with stale compiled JS. The fix (embeddedProducts, a?.products) exists in .map files but NOT in the actual compiled catalog-browser.service.js. This suggests tsup build cache returned old output. This empty commit forces a fresh build without cache.
Previous builds used cache-from: type=gha which returned STALE compiled JS layers. The fix (embeddedProducts, a?.products) existed in .map files but NOT in the actual compiled catalog-browser.service.js. Setting no-cache: true ensures tsup recompiles all source files fresh.
collectionsWithEmbeddedProducts=9 but totalProductsMapped=0. This means serializeProduct returns null for all products. Add diagnostic for first embedded product: - type, constructor name (is it a Model?) - keys (attributes or own properties) - hasId check - serializeProduct result (OK or failed) - sample id and name if serialization works
ROOT CAUSE FOUND:
Diagnostic confirmed products ARE in getCollections response:
- firstEmbeddedProductSerializeResult: OK (18 keys)
- firstEmbeddedProductSampleName: 'Seblak komplit'
- collectionsWithEmbeddedProducts: 9
- firstCollectionModelSample.products: Array(55)
But totalProductsMapped was 0 because of a bug in the 'Attach
products to collections' loop at the end of page.evaluate:
col.products = productsByCollectionId.get(col.id) || []
This OVERWRITES the embedded products (already set in Step 1) with
empty array from productsByCollectionId Map (which is empty because
Methods A/B/C all failed).
FIX:
Only overwrite col.products if it doesn't already have embedded
products. If col.products already has items (from Step 1), just
update the count fields.
Expected result: 9 collections × ~55 products = ~95+ products mapped.
WhatsApp CDN URLs contain time-limited tokens (oh=, oe= parameters)
that expire after ~24 hours. When n8n tries to download images later
for Odoo sync, it gets 403 Forbidden — all image uploads fail.
Solution: Download images at fetch time and store them locally in
public/catalog-images/{instanceName}/{productId}.jpg
The fetchCollections response replaces image_cdn_urls with local URLs
like: http://evolution-api:8080/catalog-images/{instance}/{file}.jpg
These local URLs never expire and are always accessible by n8n/Odoo.
Implementation:
- New method: downloadCollectionImages(collections, instanceName)
- Downloads 'full' key image (high-res, not thumbnail)
- Skips re-download if file already exists (>1KB = valid)
- Falls back to CDN URL if download fails
- Logs: downloaded/cached/failed counts
Express already serves /public via express.static in main.ts, so
local images are accessible at /catalog-images/{instance}/{file}.jpg
This fixes the 'images_failed: 10' issue in n8n Upload Images1 node
where all image uploads returned 500/403 errors.
|
@kelvinzer0, we dont accept anything from main (need to point to develop) and like @NeritonDias said, problem with scope and purpose of this PR. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR fixes critical pagination limitations in the WhatsApp Business catalog API that prevent fetching catalogs with more than 50 products.
Changes
whatsapp.baileys.service.ts:4904)whatsapp.baileys.service.ts:4927)getCollections- now defaults to 100 (whatsapp.baileys.service.ts:4974)cursorparameter to validation schemas for manual pagination control (business.schema.ts)cursortogetCollectionsDto(business.dto.ts)Problem
The current implementation has these limitations:
getCatalog: Default limit=10 with max 4 loops = 50 products maxgetCollections: Hard-capped at 20 items per collectioncursorparameter stripped by validation schema, preventing manual paginationSolution
Testing
getCatalogandgetCollectionsnow return complete dataCloses #2589
Summary by Sourcery
Adjust WhatsApp Business catalog APIs to support larger, cursor-based pagination and add automated container publishing to GitHub Container Registry.
New Features:
Enhancements:
CI: