Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions LIMITATIONS.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,10 +70,10 @@ do. Read this before trusting any single output.
get_context_pack). No write tools, no network listener, no source code returned.
The rest of the planned tool surface is not implemented yet.

## 6b. Verification limitations (v0.2, experimental)
## 6b. Verification limitations (experimental)

- One built-in claim (billing-webhook-signature). User-defined claims are not
supported yet, deliberately.
- Two built-in claims (billing-webhook-signature, jwt-authentication).
User-defined claims are not supported yet, deliberately.
- Verification is rule-driven over scanner signals; it inherits every scanner
coverage limitation listed here.
- Statuses mean "per DevTime's evidence rules", not formal proof or a security
Expand Down
4 changes: 2 additions & 2 deletions QUICKSTART.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ python -m venv .venv
source .venv/bin/activate # Windows (PowerShell): .venv\Scripts\Activate.ps1
# Windows (Git Bash): source .venv/Scripts/activate
pip install -e ".[dev]"
pytest # optional: all tests pass (120 at v0.2.0)
pytest # optional: all tests pass (126 at v0.3.0)
```

## 3. Create the demo repo
Expand Down Expand Up @@ -132,7 +132,7 @@ A fresh-clone check was run on the current candidate:
- **OS:** Windows 11 (Git Bash)
- **Python:** 3.11.9
- **Install:** `pip install -e ".[dev]"`
- **Tests:** all passing (120 at v0.2.0)
- **Tests:** all passing (126 at v0.3.0)
- **Demo:** `dtc init` / `dtc scan` / `dtc concepts` / `dtc explain "Billing Webhooks"`
all produced the expected output from a clean `git clone`.

Expand Down
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -278,8 +278,8 @@ Contradictions:
```

Statuses are SUPPORTED, WEAK, CONTRADICTED, or UNKNOWN; contradictions always
show both sides; changed evidence marks a claim STALE. One built-in claim ships
in v0.2. See **[VERIFICATION.md](VERIFICATION.md)**.
show both sides; changed evidence marks a claim STALE. Two built-in claims ship
(billing webhook signatures, JWT authentication). See **[VERIFICATION.md](VERIFICATION.md)**.

## Example output

Expand Down
50 changes: 50 additions & 0 deletions RELEASE_NOTES_v0.3.0.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
# DevTime v0.3.0 - JWT claim and the docs-vs-implementation detector

Second step of the verification layer. Two changes and one honest bug fix. No
cloud, no telemetry, no AI, no code execution - unchanged.

## New built-in claim: jwt-authentication

```bash
dtc verify jwt-authentication
```

"Authentication uses JWT access tokens." Uses the JWT purpose classifier
(introduced in v0.0.6) to distinguish access tokens from invitation and
verification tokens.

## New: documentation-vs-implementation contradiction

The classic case: documentation says JWT, the code says otherwise.

```
Status: CONTRADICTED
claimed: docs/auth.md references JWT in a documentation or decision
context, implying JWT-based authentication.
observed: src/tokens/invite.ts uses JWT for invitation/verification tokens.
Invitation tokens are not access-token authentication, and no
access-token usage was found.
```

Honesty rule preserved: documentation with NO usage found at all is WEAK
(missing evidence), never CONTRADICTED - absence is not positive conflicting
evidence.

## Fixed: signal metadata was silently dropped at persistence

A latent V0 bug: every signal's metadata was hardcoded to `{}` when written to
the local database. Concept detection never noticed (it runs in-process on live
objects); the verification engine is the first feature that reads metadata back
from the database, and it exposed the bug. Metadata (including the JWT purpose
classification) now persists, with a regression test.

## Notes

- Two built-in claims now: billing-webhook-signature, jwt-authentication.
- Still deliberately no user-defined claim files, and no LLM in the truth path.
- 126 passing tests (6 new).
- No breaking changes.

## Names

- PyPI distribution: `devtime-ei`. Python import: `devtime`. CLI: `dtc`.
18 changes: 10 additions & 8 deletions VERIFICATION.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Claim verification (experimental, v0.2)
# Claim verification (experimental)

DevTime is evolving into the verification layer for repository understanding:
the layer developers and coding agents use to know whether a statement about a
Expand Down Expand Up @@ -55,13 +55,15 @@ same time: the last verification supported it, but its evidence changed since.
Freshness only tracks files that were evidence for the claim. Unrelated changes
never mark a claim stale.

## Built-in claims (v0.2)

One claim ships in this release, chosen because it is the best-covered domain
in the scanner and it has a real-world contradiction story:
## Built-in claims

- **billing-webhook-signature** - "Incoming billing webhooks verify the payment
provider's signature."
- **jwt-authentication** (v0.3) - "Authentication uses JWT access tokens."
Includes the documentation-vs-implementation detector: documentation claiming
JWT while the only JWT usage found is invitation/verification tokens is a
both-sided contradiction. Documentation with no usage at all is WEAK, not
contradicted: absence is not positive conflicting evidence.

Example contradiction, found in the wild: Cal.com's community edition ships
`pages/api/stripe/webhook.ts` as a handler that always returns 404 ("not
Expand All @@ -84,14 +86,14 @@ sides and exact paths.
- Heuristic scanner: evidence comes from static patterns, not execution.
- Signature verification is recognized for known provider patterns
(e.g. Stripe `constructEvent`); custom schemes may not be detected.
- One built-in claim in v0.2. User-defined claims are deliberately not
- Two built-in claims. User-defined claims are deliberately not
supported yet: the claim model must earn trust before it grows a
configuration language.
- Coverage follows scanner language support; see [LIMITATIONS.md](LIMITATIONS.md).

## Where this is going

Next candidates, in order: more built-in claims over well-covered domains
(webhook idempotency, JWT authentication), a documentation-vs-implementation
contradiction detector, and diff-aware claim staleness in `dtc risk`. User-defined
(webhook idempotency), more contradiction detectors, and diff-aware claim
staleness in `dtc risk`. User-defined
claims come after built-in claims prove trustworthy on real repositories.
4 changes: 2 additions & 2 deletions pyproject.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[project]
name = "devtime-ei"
version = "0.2.0"
version = "0.3.0"
description = "Local-first Engineering Intelligence for software repositories"
readme = "README.md"
requires-python = ">=3.11"
Expand Down Expand Up @@ -50,7 +50,7 @@ dev = [
Homepage = "https://github.com/Shakargy/devtime"
Repository = "https://github.com/Shakargy/devtime"
Issues = "https://github.com/Shakargy/devtime/issues"
"Release Notes" = "https://github.com/Shakargy/devtime/releases/tag/v0.2.0"
"Release Notes" = "https://github.com/Shakargy/devtime/releases/tag/v0.3.0"
Demo = "https://youtu.be/1Hiu3Y9J_SI"

[project.scripts]
Expand Down
4 changes: 2 additions & 2 deletions server.json
Original file line number Diff line number Diff line change
Expand Up @@ -8,12 +8,12 @@
"source": "github"
},
"websiteUrl": "https://github.com/Shakargy/devtime",
"version": "0.2.0",
"version": "0.3.0",
"packages": [
{
"registryType": "pypi",
"identifier": "devtime-ei",
"version": "0.2.0",
"version": "0.3.0",
"transport": {
"type": "stdio"
}
Expand Down
2 changes: 1 addition & 1 deletion src/devtime/__init__.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
"""DevTime - local-first Engineering Intelligence for repository memory."""

__version__ = "0.2.0"
__version__ = "0.3.0"

# Version metadata (Builder Edition, Chapter 20).
EVIDENCE_MODEL = "2026.06.1"
Expand Down
141 changes: 140 additions & 1 deletion src/devtime/intelligence/verification.py
Original file line number Diff line number Diff line change
Expand Up @@ -149,8 +149,17 @@ class ClaimDefinition:
statement="Incoming billing webhooks verify the payment provider's signature.",
category="billing",
),
"jwt-authentication": ClaimDefinition(
slug="jwt-authentication",
name="JWT Authentication",
statement="Authentication uses JWT access tokens.",
category="authentication",
),
}

# JWT library names for dependency evidence.
_JWT_DEP_TOKENS = ("jsonwebtoken", "pyjwt", "jose", "njwt", "jwt-decode", "fast-jwt")

_LIMITATIONS = [
"Heuristic scanner: evidence comes from static patterns, not execution.",
"Signature verification is recognized for known provider patterns "
Expand Down Expand Up @@ -225,7 +234,8 @@ def verify_claim(conn: sqlite3.Connection, slug: str) -> VerificationResult:
)

rows = _load_signals(conn, scan_id)
return _verify_billing_webhook_signature(definition, rows, scan_id)
evaluator = _EVALUATORS[slug]
return evaluator(definition, rows, scan_id)


def _verify_billing_webhook_signature(
Expand Down Expand Up @@ -348,6 +358,135 @@ def _verify_billing_webhook_signature(
)


def _verify_jwt_authentication(
definition: ClaimDefinition, rows: list[sqlite3.Row], scan_id: str
) -> VerificationResult:
"""Verify JWT access-token authentication, with the documentation-vs-
implementation contradiction detector.

The purpose classifier (Trust Repair v0.0.6) distinguishes access tokens
from invitation/verification tokens. Documentation claiming JWT while the
only JWT usage found is invitation-purpose is a real, both-sided conflict.
Documentation with NO usage found at all is missing evidence (WEAK), not a
contradiction: absence is not positive conflicting evidence.
"""
access_usage: list[EvidenceRef] = []
invitation_usage: list[EvidenceRef] = []
unclear_usage: list[EvidenceRef] = []
jwt_docs: list[EvidenceRef] = []
jwt_deps: list[EvidenceRef] = []

for row in rows:
hay = _hay(row)
kind = row["kind"]

if kind == "token_usage":
try:
purpose = json.loads(row["metadata_json"] or "{}").get("purpose", "unclear")
except json.JSONDecodeError:
purpose = "unclear"
if purpose == "access":
access_usage.append(
_ref(row, "JWT used as an access token (login/bearer context).", "strong")
)
elif purpose == "invitation":
invitation_usage.append(
_ref(
row,
"JWT used for invitation/verification tokens, not access.",
"moderate",
)
)
else:
unclear_usage.append(
_ref(row, "JWT usage found; its purpose is unclear.", "weak")
)
elif kind in ("doc", "decision") and "jwt" in hay:
jwt_docs.append(
_ref(
row,
"Documentation or decision record references JWT.",
"moderate" if kind == "decision" else "weak",
)
)
elif kind == "dependency" and any(t in hay for t in _JWT_DEP_TOKENS):
jwt_deps.append(
_ref(row, "JWT library dependency is declared.", "weak")
)

why: list[str] = []
missing: list[str] = []
contradictions: list[Contradiction] = []
supporting: list[EvidenceRef] = []

if access_usage:
status = SUPPORTED
supporting = access_usage + jwt_docs + jwt_deps
why.append("JWT access-token usage evidence was found.")
if jwt_docs:
why.append("Documentation or a decision record corroborates JWT usage.")
else:
missing.append("A decision record explaining the JWT choice.")
elif jwt_docs and invitation_usage:
status = CONTRADICTED
supporting = invitation_usage
doc = jwt_docs[0]
inv = invitation_usage[0]
contradictions.append(
Contradiction(
summary="Documentation claims JWT authentication, but the only JWT "
"usage found is invitation/verification tokens.",
claimed_side=f"{doc.path} references JWT in a documentation or "
"decision context, implying JWT-based authentication.",
observed_side=f"{inv.path} uses JWT for invitation/verification "
"tokens. Invitation tokens are not access-token authentication, "
"and no access-token usage was found.",
evidence=[doc, inv],
)
)
why.append(
"Documentation references JWT, but no access-token usage exists; "
"the only JWT usage found is invitation-purpose."
)
missing.append("JWT access-token usage (login/bearer/authorization context).")
elif jwt_docs or jwt_deps or invitation_usage or unclear_usage:
status = WEAK
supporting = jwt_docs + jwt_deps + invitation_usage + unclear_usage
why.append(
"JWT surface exists (documentation, dependencies, or unclear usage), "
"but access-token authentication is not established."
)
missing.append("JWT access-token usage (login/bearer/authorization context).")
else:
status = UNKNOWN
why.append(
"No JWT evidence was found in the scanned files. The claim does not "
"apply, or the usage is outside scanner coverage."
)
missing.append("Any JWT usage, dependency, or documentation.")

return VerificationResult(
claim_slug=definition.slug,
claim_name=definition.name,
statement=definition.statement,
status=status,
why=why,
supporting=supporting,
contradictions=contradictions,
missing=missing,
limitations=_LIMITATIONS,
scan_id=scan_id,
verified_at=_now(),
engine_version=__version__,
)


_EVALUATORS = {
"billing-webhook-signature": _verify_billing_webhook_signature,
"jwt-authentication": _verify_jwt_authentication,
}


def _ref(row: sqlite3.Row, observation: str, strength: str) -> EvidenceRef:
return EvidenceRef(
path=row["path"],
Expand Down
6 changes: 5 additions & 1 deletion src/devtime/scanner/signals.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@
from __future__ import annotations

import hashlib
import json
import sqlite3
import uuid
from dataclasses import dataclass, field
Expand Down Expand Up @@ -214,7 +215,10 @@ def run_scan(
s.start_line,
s.end_line,
s.confidence,
"{}",
# v0.3.0: persist extractor metadata (was hardcoded '{}',
# silently dropping e.g. the JWT purpose classification
# that verification reads back from the database).
json.dumps(s.metadata or {}),
),
)
conn.commit()
Expand Down
8 changes: 4 additions & 4 deletions tests/integration/test_evidence_precision.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,14 +14,14 @@

# --- version ------------------------------------------------------------------

def test_version_is_release_0_2_0():
# v0.2.0 release: package metadata and __version__ agree on the release version.
def test_version_is_release_0_3_0():
# v0.3.0 release: package metadata and __version__ agree on the release version.
import importlib.metadata as m

assert devtime.__version__ == "0.2.0"
assert devtime.__version__ == "0.3.0"
# Distribution is published as "devtime-ei" (the name "devtime" is reserved on
# PyPI); the import package and the dtc command stay "devtime"/"dtc".
assert m.version("devtime-ei") == "0.2.0"
assert m.version("devtime-ei") == "0.3.0"


# --- P0 Authentication headline precision ------------------------------------
Expand Down
Loading
Loading