GitLab CI Explorer

SQLite-first pipeline importer and local UI for exploring GitLab CI timings as an end-to-end waterfall.

What This Does

• Imports one GitLab pipeline run into SQLite.
• Stores:
  • pipeline timestamps
  • job timestamps
  • parsed trace sections
• Renders one imported pipeline as a browser-network-tab style timeline.

SQLite is the system of record.

Repo Layout

• scraper.py: CLI importer (SQLite write path)
• data/: local DB (ignored by git)
• frontend/: Next.js app (SQLite read path)

Requirements

• Python 3.11+
• GITLAB_TOKEN in the environment

Quick Start

1. Create a virtualenv and install dependencies:

python3 -m venv .venv
. .venv/bin/activate
pip install -r requirements.txt

2. Initialize the database:

python3 scraper.py db-init

3. Import one pipeline by project path + pipeline id:

python3 scraper.py import-pipeline acme/frontend-app 2439631130

4. Import the most recent pipelines for one project:

python3 scraper.py import-recent acme/frontend-app --limit 5

5. Start the frontend:

cd frontend
npm install
npm run dev

6. Use a custom database path if needed:

python3 scraper.py --db-path ./data/custom.sqlite import-pipeline acme/frontend-app 2439631130

Environment

• GITLAB_TOKEN: personal access token with read access to the relevant GitLab projects
• GITLAB_URL: optional, defaults to https://gitlab.com
• CI_EXPLORER_DB_PATH: optional frontend override for the SQLite database path

CLI Commands

Default DB path:

• ./data/gitlab-ci.sqlite

Initialize schema:

python3 scraper.py db-init

Import one pipeline:

python3 scraper.py import-pipeline <project-path> <pipeline-id>

Import the most recent pipelines for one project:

python3 scraper.py import-recent <project-path> --limit 10

SQLite Schema

• gitlab_project: project identity
• gitlab_pipeline: pipeline-level timing envelope
• gitlab_job: job-level timeline anchors, including derived started_queueing_at_utc
• gitlab_job_section: parsed trace sections, plus optional diagnostic details for enriched phases

Timing Model

The viewer will treat timestamps as canonical.

Important notes:

• job.started_queueing_at_utc is derived from:
  • job.started_at - job.queued_duration
• job.created_at_utc -> job.started_queueing_at_utc represents blocked / dependency wait time.
• job.started_queueing_at_utc -> job.started_at_utc represents runnable queue time.

Frontend View

The primary view is one pipeline run:

• global time axis from pipeline.created_at_utc to pipeline.finished_at_utc
• one row per job
• nested sections inside jobs
• visible blocked / queued / execution phases

Current enriched breakdowns:

• prepare_script: scheduling, scale-up wait, pod startup, service startup, finalize
• restore_cache: download, extract
• archive_cache: pack, upload