Question regarding scaling options.

[DucTuLeAnh]

Hi, I'm currently experimenting with devlake. Reading the docs, it looks like devlake only supports single instance setups. So there are no horizontal scaling options by increasing replicas, correct?

Because I was wondering, if devlake is a fitting product to provide company wide engineering metrics, covering 1000+ gitlab repositories. Having skimmed through some Discussions, you can kinda improve performance with proper indices, but I'm not super sure yet. What has been your experience so far?

[dosubot[bot]]

Hi @DucTuLeAnh! I'm Dosu and I’m helping the incubator-devlake team.

You're correct — DevLake is designed as a single-instance application and doesn't support horizontal scaling by increasing replicas. All blueprints are processed sequentially by a single runner, which becomes the primary bottleneck at enterprise scale.

For 1000+ GitLab repositories, there have been community reports of significant challenges:

• Full collection cycles taking ~2 weeks for 80+ projects
• MySQL hitting 100% CPU at ~1,000 projects even with 16 cores / 64 GB RAM
• Raw tables growing to 21+ GB with 2000 repos due to user data duplication

What helps at scale:

1. Database indexes — Community-tested indexes reduced DORA calculations from hours to minutes
2. GitLab-specific optimization — Set GITLAB_SERVER_COLLECT_ALL_USERS=false to collect members per-project instead of globally (reduced collection time by ~70%)
3. Memory allocation — Minimum 6 GB container memory recommended
4. Collection strategy — Use incremental collection, narrow date ranges, and consider SKIP_COMMIT_STAT=true / SKIP_COMMIT_FILES=true

The honest answer is that at 1000+ repositories, you'll likely face performance challenges even with optimizations. Parallel runners are the most requested architectural enhancement for this reason.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}