feat: add LSP server for editor integration (pygls) by jimisola · Pull Request #323 · reqstool/reqstool-client

jimisola · 2026-03-15T20:25:06Z

Important

This branch is based on feat/313-sqlite-storage (#321) but targets main. PR #321 must be merged first before this PR can be merged.

Summary

Add a Python LSP server to reqstool-client using pygls, backed by the SQLite pipeline from #313. The server provides editor-agnostic support for reqstool annotations in Java, Python, TypeScript, and JavaScript.

Depends on: #321 (SQLite storage)

Implemented features

Hover: Show requirement/SVC details when hovering over @Requirements/@SVCs annotations + YAML field descriptions from JSON schemas
Diagnostics: Red squiggles on unknown/deprecated/obsolete requirement/SVC IDs + YAML schema validation
Completion: Autocomplete requirement/SVC IDs inside annotations + enum values in YAML files (significance, categories, lifecycle, verification, etc.)
Go-to-definition: Navigate from annotations in source code to YAML definitions and vice versa
Code lens: Inline requirement/SVC stats on annotated methods
Document symbols: Outline view for requirements.yml, software_verification_cases.yml, manual_verification_results.yml with cross-references
Multi-project workspaces: Discover and track multiple reqstool projects (e.g., Gradle multi-module with system + microservices)
File watching: Auto-reload on static file changes + manual refresh command for remote dependency changes
Language support: Java/Python (source-level @Requirements/@SVCs annotations) + TypeScript/JavaScript (JSDoc @Requirements/@SVCs tags)
.reqstoolignore: Place in any directory to exclude it from LSP workspace discovery (ecosystem-agnostic — works for Python, Java, JS/TS projects)

Architecture

Editor (VS Code / Neovim / IntelliJ / etc.)
  ↕ LSP protocol (stdio)
ReqstoolLanguageServer (pygls)
  → WorkspaceManager (manages multiple projects)
      → ProjectState[] (one per reqstool project found)
          → build_database() pipeline (reused from #313)
          → RequirementsRepository (reused from #313)
  → Features: hover, diagnostics, completion, go-to-definition, document symbols, code lens

New CLI command

pip install reqstool   # pygls and lsprotocol are now core dependencies
reqstool lsp           # start LSP server in stdio mode

New files

File	Purpose
`src/reqstool/lsp/annotation_parser.py`	Regex-based annotation detection (Java/Python decorators + JSDoc tags)
`src/reqstool/lsp/project_state.py`	Wraps `build_database()` pipeline for a single reqstool project
`src/reqstool/lsp/root_discovery.py`	Finds root reqstool projects in workspace folders; honours `.reqstoolignore`
`src/reqstool/lsp/workspace_manager.py`	Per-folder isolation managing ProjectState instances
`src/reqstool/lsp/server.py`	pygls LanguageServer with lifecycle, file watching, refresh
`src/reqstool/lsp/yaml_schema.py`	JSON Schema loading, field descriptions, enum values
`src/reqstool/lsp/features/hover.py`	Hover for source annotations + YAML fields
`src/reqstool/lsp/features/diagnostics.py`	Source + YAML schema diagnostics
`src/reqstool/lsp/features/completion.py`	ID + YAML enum completion
`src/reqstool/lsp/features/definition.py`	Source ↔ YAML go-to-definition
`src/reqstool/lsp/features/document_symbols.py`	YAML outline view
`src/reqstool/lsp/features/codelens.py`	Inline stats lens on annotated methods

Test coverage

198 LSP unit tests, 530 total unit tests — all passing.

Closes

Closes feat: Language Server Protocol (LSP) for editor integration #314

Test plan

Unit tests for annotation parser
Unit tests for project state
Unit tests for workspace manager
Unit tests for YAML schema validation and completion
Unit tests for hover, diagnostics, go-to-definition, document symbols, code lens
Unit tests for root discovery (incl. .reqstoolignore behaviour)
Regression smoke test (CLI output identical to main)
Manual testing with VS Code reqstool extension

jimisola · 2026-03-15T20:36:09Z

@Jonas-Werne Iterating over the plan now: https://github.com/reqstool/reqstool-client/blob/feat/314-lsp-server/docs/lsp-server-plan.md

…nd multi-root workspace support (#314) Adds refined Q5 root project discovery algorithm, document symbols feature (Step 9), separate root_discovery.py module, workspace/didChangeWorkspaceFolders support, and YAML schema utility (Step 10). Signed-off-by: jimisola <jimisola@users.noreply.github.com>

Move pygls/lsprotocol to project.optional-dependencies instead of main dependencies. Users install with `pip install reqstool[lsp]` only when they need the LSP server; CI pipelines keep the lighter base install. Signed-off-by: jimisola <jimisola@users.noreply.github.com>

…annotation parser (#314) - Add pygls/lsprotocol as optional [lsp] extra in pyproject.toml - Add `reqstool lsp` subcommand with ImportError guard for missing extra - Create annotation_parser module detecting @Requirements/@svcs in Java/Python (source decorators) and TypeScript/JavaScript (JSDoc tags) - Add 33 unit tests for annotation parser covering both syntaxes, position lookup, completion context, and multi-line annotations Signed-off-by: jimisola <jimisola@users.noreply.github.com>

- ProjectState wraps build_database() pipeline for a single reqstool project with query helpers for requirements, SVCs, and MVRs - Root discovery finds root projects in workspace folders by parsing requirements.yml metadata and building a local reference graph - WorkspaceManager provides per-folder isolation with add/remove/rebuild operations and file-to-project resolution - Add 27 unit tests covering project lifecycle, root discovery algorithm, and workspace management Signed-off-by: jimisola <jimisola@users.noreply.github.com>

… command (#314) - ReqstoolLanguageServer (pygls subclass) with workspace manager integration - Lifecycle handlers: initialized, shutdown, didOpen/Change/Save/Close - Workspace folder change tracking (add/remove folders dynamically) - File watcher for static YAML files triggers project rebuild - reqstool.refresh command for manual rebuild of all projects - Diagnostic publishing placeholders for Step 6 Signed-off-by: jimisola <jimisola@users.noreply.github.com>

- Hover on @Requirements/@svcs annotations shows requirement/SVC details - Hover on YAML fields shows JSON Schema descriptions - YAML schema utilities: load schemas, resolve $ref, get field descriptions/enum values - Wire hover handler into LSP server Signed-off-by: Jimisola Laursen <jimisola@jimisola.com>

- Source code diagnostics: unknown requirement/SVC IDs, deprecated/obsolete lifecycle warnings - YAML diagnostics: parse errors and JSON Schema validation against reqstool schemas - Wire diagnostics into server on didOpen, didChange, didSave, and rebuild events Signed-off-by: Jimisola Laursen <jimisola@jimisola.com>

- Source completion: offer requirement/SVC IDs inside @Requirements/@svcs annotations - YAML completion: offer enum values for fields like significance, variant, categories - Wire completion handler into server with trigger characters Signed-off-by: Jimisola Laursen <jimisola@jimisola.com>

- Source → YAML: navigate from @Requirements/@svcs annotations to YAML definitions - YAML → YAML: navigate from requirement IDs to SVC references and vice versa - Wire definition handler into LSP server Signed-off-by: Jimisola Laursen <jimisola@jimisola.com>

- Outline view for requirements.yml, software_verification_cases.yml, manual_verification_results.yml - Shows requirement/SVC/MVR items with titles, significance, and cross-references as children - Wire document symbol handler into LSP server Signed-off-by: Jimisola Laursen <jimisola@jimisola.com>

_find_id_in_yaml only matched `id: <raw_id>` lines, so cross-reference navigation (REQ→SVC, SVC→MVR) returned empty results because IDs appear inside array fields like `requirement_ids: ["REQ_PASS"]`. Add _find_reference_in_yaml with a \b word-boundary pattern and strengthen the integration test to assert actual navigation results. Signed-off-by: Jimisola Laursen <jimisola@jimisola.com>

Add location_type and location_uri columns to urn_metadata table to record where each URN's data originated (local, git, maven, pypi). Carry resolved file paths through CombinedRawDataset so the LSP go-to-definition handler uses actual paths from reqstool_config.yml instead of hard-coded filenames. Signed-off-by: Jimisola Laursen <jimisola@jimisola.com>

Add regression-python fixture project with requirements, SVCs, MVRs, annotations, and test results for end-to-end LSP integration tests. Configure pytest-asyncio and add test client infrastructure. Signed-off-by: Jimisola Laursen <jimisola@jimisola.com>

Signed-off-by: Jimisola Laursen <jimisola@jimisola.com>

…#314) - Accept --stdio flag (passed by VS Code LSP client) in lsp subcommand - Add --tcp, --host, --port args for TCP transport support - Wrap start_server() call with exception handler in command.py - Wrap server.start_io()/start_tcp() with exception logging in server.py

…_path (#314)

…okens, codeAction LSP features (#314) - Add reqstool/details custom request for structured REQ/SVC/MVR data - Extend hover with "Open Details" command link - Add textDocument/codeLens (verification status above annotations) - Add textDocument/inlayHint (title inline after ID) - Add textDocument/references (find all usages across open docs + YAML) - Add workspace/symbol (quick-search REQ/SVC IDs) - Add textDocument/semanticTokens/full (color-code deprecated/obsolete) - Add textDocument/codeAction (quick fixes for unknown/deprecated IDs) - Add get_mvr() and get_yaml_paths() to ProjectState - Add _get() and _first_project() shared helpers to server.py Signed-off-by: Jimisola Laursen <jimisola@jimisola.com>

… references (#314) - get_requirement_details: adds `references` (cross-refs) and `implementations` (annotation impls) - get_svc_details: adds `test_annotations` and `test_results` (automated test status per annotation) - RequirementsRepository: adds get_test_results_for_svc for targeted per-SVC test result lookup - ProjectState: exposes get_impl_annotations_for_req, get_test_annotations_for_svc, get_test_results_for_svc Signed-off-by: Jimisola Laursen <jimisola@jimisola.com>

…s LSP features (#314) - completion: filter DEPRECATED/OBSOLETE IDs from suggestions - codeLens: add ⚠/✕ lifecycle badge per ID; pass full ids list in command args - hover: show implementation count for requirements; tests passed/failed/missing and MVR counts for SVCs - details: add test_summary aggregate, enrich requirement_ids with title+lifecycle_state, add source_paths to all responses - semanticTokens: 4 distinct token types in lifecycle order — reqstoolDraft(0), reqstoolValid(1), reqstoolDeprecated(2), reqstoolObsolete(3) Signed-off-by: Jimisola Laursen <jimisola@jimisola.com>

- Add static lsp.adoc covering all LSP capabilities, commands, and annotation languages - Add reqstool-lsp.openrpc.json (OpenRPC 1.2.6) as formal spec for the custom reqstool/details method - Fix details response: return id+urn separately (urn = project URN only, not composite UrnId) - Update nav.adoc to include LSP Server page - Update CLAUDE.md with LSP documentation guidance Signed-off-by: Jimisola Laursen <jimisola@jimisola.com>

jimisola · 2026-03-19T09:23:19Z

Note: LSP variant changes needed

PR #328 removes variant as a behavioral gate and makes it optional. The LSP module (src/reqstool/lsp/) is not on main yet, but when it lands it will need these updates:

lsp/root_discovery.py: Change DiscoveredProject.variant: VARIANTS → Optional[VARIANTS]. On ValueError, set variant = None instead of returning None.
lsp/workspace_manager.py: Handle root.variant being None in logging (root.variant.value if root.variant else None).

Existing if project.variant == VARIANTS.EXTERNAL: continue logic remains correct since None != VARIANTS.EXTERNAL.

…314) Extract LSP try/except handling into Command.command_lsp() to bring main() from complexity 13 down to the allowed maximum of 10. Signed-off-by: Jimisola Laursen <jimisola@jimisola.com>

Jonas-Werne · 2026-03-31T20:42:55Z

There are some tests that fails

Move pygls and lsprotocol from optional [lsp] extra into core dependencies so no reqstool[lsp] install is needed. Fix test_definition.py import of renamed public function find_id_in_yaml (was _find_id_in_yaml). Signed-off-by: Jimisola Laursen <jimisola@jimisola.com>

Signed-off-by: Jimisola Laursen <jimisola@jimisola.com>

Jonas-Werne · 2026-04-03T17:32:51Z

Tried to run this together with the VSCode extension but ran into some issues when trying it out on the reqstool client project. It did not get the correct requirements

When running the status command the wrong requirements appear as errors so that might explain it, but I'm a little unsure

jimisola · 2026-04-04T06:50:18Z

I suspect that this is due to previous changes and not the LSP stuff per se.

I'll handle it as a separate issue (bug) and deal with that when working on #325 #326 #327

Tried to run this together with the VSCode extension but ran into some issues when trying it out on the reqstool client project. It did not get the correct requirements
When running the status command the wrong requirements appear as errors so that might explain it, but I'm a little unsure

jimisola · 2026-04-04T06:51:19Z

Actually it seems that this could be the cause:

When a user opens the reqstool-client project itself in VS Code with the LSP server, the LSP shows wrong/unrelated requirements. The root cause: discover_root_projects() in root_discovery.py recursively walks the entire workspace and finds all requirements.yml files — including test fixtures under tests/resources/test_data/. Since those fixture files are not referenced by the main docs/reqstool/requirements.yml, they each become "root" projects. The LSP then loads them all, and the wrong one may be selected for a given file.

Introduce a .reqstoolignore sentinel file convention for the LSP's workspace discovery. When _walk_dir encounters a directory containing .reqstoolignore it skips that directory entirely, preventing test fixture requirements.yml files from being loaded as root projects. Add .reqstoolignore to tests/resources/test_data/ so that the reqstool-client project's own test fixtures are not discovered when the project is opened in VS Code. Fixes: #323 Signed-off-by: Jimisola Laursen <jimisola@jimisola.com>

jimisola · 2026-04-04T11:43:38Z

Thanks for testing, @Jonas-Werne!

I believe that the issue you hit is now fixed. When the reqstool-client project itself is opened in VS Code, the LSP was discovering test fixture requirements.yml files alongside the real docs/reqstool/requirements.yml, causing the wrong requirements to be shown.

The fix introduces a .reqstoolignore sentinel file (analogous to .gitignore): any directory containing this file is skipped entirely during LSP workspace discovery. A .reqstoolignore has been added to tests/resources/test_data/ so the test fixtures are excluded.

Could you pull the latest feat/314-lsp-server and try again?

Jonas-Werne

Tested it out initially I had the same issue noticed that we had missed to add a .reqstoolignore file. Added it and now it works 👍

jimisola · 2026-04-04T14:29:11Z

Tested it out initially I had the same issue noticed that we had missed to add a .reqstoolignore file. Added it and now it works 👍

My mistake. Thanks for the commit.

So, we can merge it then.

Jonas-Werne · 2026-04-04T14:31:27Z

Now some tests fail, probably because I ignored the regression requirements.

jimisola · 2026-04-04T14:33:35Z

Now some tests fail, probably because I ignored the regression requirements.

I'm working on them.

…ation tests Placing .reqstoolignore inside tests/fixtures/reqstool-regression-python/ caused the LSP integration tests to fail: _walk_dir finds the file at depth=0 (the workspace root) and skips the entire directory, so the regression fixture project is never discovered and all LSP features return 'project not loaded'. Move the file up one level to tests/fixtures/ so it only blocks discovery when the full reqstool-client project is opened as a workspace. When the integration test uses reqstool-regression-python as the workspace root directly, _walk_dir starts at that directory (depth=0, no .reqstoolignore present) and correctly discovers requirements.yml. Signed-off-by: Jimisola Laursen <jimisola@jimisola.com>

jimisola · 2026-04-04T14:38:44Z

@Jonas-Werne Can you try again? Moved the .reqstoolignore one level.

Jonas-Werne

Works now 👍

jimisola · 2026-04-04T14:45:14Z

Works now 👍

I'm making an additional change. The .reqstoolignore should allow/include a glob pattern from day 1 so that we don't have two different versions of .reqstoolignore later. I'll let you know when it's implemented so that you can try it again (hit my limit). Ok?

Patterns (one per line, # comments ignored) are matched with fnmatch against immediate child directory names. An empty or comments-only file has no effect; use ** to skip all children. Update tests/resources/test_data/.reqstoolignore and tests/fixtures/.reqstoolignore to use the explicit ** pattern with an explanatory comment instead of relying on an empty file. Signed-off-by: Jimisola Laursen <jimisola@jimisola.com>

jimisola · 2026-04-04T17:55:28Z

Works now 👍

I'm making an additional change. The .reqstoolignore should allow/include a glob pattern from day 1 so that we don't have two different versions of .reqstoolignore later. I'll let you know when it's implemented so that you can try it again (hit my limit). Ok?

@Jonas-Werne Please try again when you have the time.

Jonas-Werne

Tried it again, Works as expected

jimisola mentioned this pull request Mar 15, 2026

feat: Language Server Protocol (LSP) for editor integration #314

Closed

jimisola requested a review from Jonas-Werne March 15, 2026 20:26

jimisola self-assigned this Mar 15, 2026

jimisola added 20 commits March 18, 2026 23:21

docs: add LSP server implementation plan (#314)

94a8dcb

style: apply black formatting and fix flake8 unused import (#314)

7ae81a5

Signed-off-by: Jimisola Laursen <jimisola@jimisola.com>

fix: replace PyYAML with ruamel.yaml in LSP diagnostics (#314)

a8b6f99

feat: add --log-file option to lsp command for debugging (#314)

9b58fc5

fix: match source files to project via workspace folder, not reqstool…

5331f5d

…_path (#314)

jimisola force-pushed the feat/314-lsp-server branch from 21e5991 to 7fc2a9c Compare March 18, 2026 22:22

jimisola added 3 commits March 18, 2026 23:27

fix: reduce cyclomatic complexity of main() below flake8 C901 limit (#…

d08eb0b

…314) Extract LSP try/except handling into Command.command_lsp() to bring main() from complexity 13 down to the allowed maximum of 10. Signed-off-by: Jimisola Laursen <jimisola@jimisola.com>

jimisola dismissed Jonas-Werne’s stale review via cd172cd April 3, 2026 07:58

jimisola added 2 commits April 3, 2026 10:01

Merge branch 'main' into feat/314-lsp-server

bb36f02

fix: include requirement IDs in codelens title for multi-ID annotations

233716c

Signed-off-by: Jimisola Laursen <jimisola@jimisola.com>

jimisola requested a review from Jonas-Werne April 3, 2026 09:50

Jonas-Werne mentioned this pull request Apr 3, 2026

feat: add initial LSP client extension reqstool/reqstool-vscode#1

Merged

8 tasks

jimisola changed the title ~~feat: add LSP server for editor integration~~ feat: add LSP server for editor integration (pygls) Apr 4, 2026

fix: add missing .reqstoolignore file

2f85032

Jonas-Werne previously approved these changes Apr 4, 2026

View reviewed changes

jimisola dismissed Jonas-Werne’s stale review via bcd430d April 4, 2026 14:37

Jonas-Werne previously approved these changes Apr 4, 2026

View reviewed changes

jimisola dismissed Jonas-Werne’s stale review via fe3d291 April 4, 2026 16:11

jimisola force-pushed the feat/314-lsp-server branch from fe3d291 to 553a98f Compare April 4, 2026 17:54

Jonas-Werne approved these changes Apr 4, 2026

View reviewed changes

jimisola merged commit 7297deb into main Apr 4, 2026
7 checks passed

jimisola deleted the feat/314-lsp-server branch April 4, 2026 18:13

Conversation

jimisola commented Mar 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Implemented features

Architecture

New CLI command

New files

Test coverage

Closes

Related

Test plan

Uh oh!

jimisola commented Mar 15, 2026

Uh oh!

jimisola commented Mar 19, 2026

Note: LSP variant changes needed

Uh oh!

Jonas-Werne commented Mar 31, 2026

Uh oh!

Jonas-Werne commented Apr 3, 2026

Uh oh!

jimisola commented Apr 4, 2026

Uh oh!

jimisola commented Apr 4, 2026

Uh oh!

jimisola commented Apr 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Jonas-Werne left a comment

Choose a reason for hiding this comment

Uh oh!

jimisola commented Apr 4, 2026

Uh oh!

Jonas-Werne commented Apr 4, 2026

Uh oh!

jimisola commented Apr 4, 2026

Uh oh!

jimisola commented Apr 4, 2026

Uh oh!

Jonas-Werne left a comment

Choose a reason for hiding this comment

Uh oh!

jimisola commented Apr 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jimisola commented Apr 4, 2026

Uh oh!

Jonas-Werne left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jimisola commented Mar 15, 2026 •

edited

Loading

jimisola commented Apr 4, 2026 •

edited

Loading

jimisola commented Apr 4, 2026 •

edited

Loading