Skip to content

CLOPS-776 Proposed performance optimisation for migration of log_visit#92

Open
marcel-innocraft wants to merge 1 commit into
5.x-devfrom
CLOPS-776-log_visit-enhancement
Open

CLOPS-776 Proposed performance optimisation for migration of log_visit#92
marcel-innocraft wants to merge 1 commit into
5.x-devfrom
CLOPS-776-log_visit-enhancement

Conversation

@marcel-innocraft

@marcel-innocraft marcel-innocraft commented Jun 25, 2026

Copy link
Copy Markdown

Description

  • Replaces offset-based batching for log_visit with index-based pagination.

Previous SQL:

SELECT * FROM log_visit WHERE idsite = 1 ORDER BY idvisit ASC OFFSET 1000000 LIMIT 10000;

New SQL:

SELECT * FROM log_visit WHERE idsite = 1 AND idvisit > 1234567 ORDER BY idvisit ASC LIMIT 10000;

The original SQL iterates through OFFSET records before collection of LIMIT results, which causes a degradation in SQL performance with each batch.
The modified SQL uses the existing index to move straight to the appropriate index position before collecting LIMIT results for that selected idsite. This avoids the iteration time, providing a faster and consistent query time.

The changes in this PR have previously been applied to production Cloud servers to facilitate large migrations by the Support team.

Issue No

Steps to Replicate the Issue

  1. Support team initiates IDSite migration for a site with a lot of log_visit records (i.e. 32M)
  2. Observe slow performance (i.e. 48h)

Checklist

  • [✔] I have understood, reviewed, and tested all AI outputs before use
  • [✔] All AI instructions respect security, IP, and privacy rules

@marcel-innocraft marcel-innocraft changed the title CLOPS-776 Replace offset-based batching for log_visit with index-based pagination CLOPS-776 Proposed performance optimisation for migration of log_visit Jun 25, 2026
@AltamashShaikh

Copy link
Copy Markdown
Contributor

@marcel-innocraft we will need to updated the Changelog.md and plugins.json file too, next release date is of Monday 2026-06-29

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants