Skip to content

feat: add caching-logic, extend data-model for shared listings#5

Merged
chenkel-data merged 3 commits into
mainfrom
feat/extend-data-model
Apr 3, 2026
Merged

feat: add caching-logic, extend data-model for shared listings#5
chenkel-data merged 3 commits into
mainfrom
feat/extend-data-model

Conversation

@chenkel-data
Copy link
Copy Markdown
Owner

@chenkel-data chenkel-data commented Apr 3, 2026

Summary

This PR extends the data model to support shared listings across multiple agents, improves scrape efficiency with cache-aware early stopping, and updates the UI so the new backend behavior is reflected cleanly in filtering, stats, and listing details.

Main Changes

  • Introduce a listing_agents junction table to support many-to-many relationships between listings and agents, allowing a single listing to be associated with multiple agents in overlapping search regions (shared listings).
  • Propagate shared-listing tags such as favorites and blacklist status across all linked agents, so actions taken in one agent are reflected everywhere the listing appears.
  • Add cache-aware early stopping to scraping, stopping pagination once consecutive pages contain no new listings.
  • Update persistence, cleanup, and stats logic to correctly support the shared-listing model.
  • Store scrape rank and run metadata to better preserve each provider’s original listing order.
  • Update the frontend to support shared listings and show extra context such as listing ID, first seen, and last seen.

Why It Matters

  • Listings found in overlapping regions now appear for every relevant agent, instead of being assigned only to the first agent that scraped them.
  • The listing order stays closer to the provider’s original order.
  • Scraping is now faster by skipping unnecessary pagination for already-known listings.

@chenkel-data chenkel-data merged commit 0e546a2 into main Apr 3, 2026
1 check passed
@chenkel-data chenkel-data deleted the feat/extend-data-model branch April 3, 2026 15:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant