Pluggable graph interface for Hoptimator topology + Mermaid renderer

[jogrogan]

Summary

Adds a generic graph interface to Hoptimator — a pluggable model for representing the topology of pipelines, triggers, and logical tables — together with an initial Mermaid renderer that's wired into the sqlline !graph command. The interface is the point; Mermaid is one rendering of it.

The graph interface (hoptimator-api)

The data model and SPI live where the rest of Hoptimator's public surface lives, so a renderer or backend can depend on them without pulling K8s, util, or anything else:

• PipelineGraph / GraphNode / GraphEdge — the data model. Nodes are typed (External, Pipeline, Trigger, LogicalTable, View); edges carry from/to direction (OWNER_OF, DEPENDS_ON_SOURCE, DEPENDS_ON_SINK, TRIGGERS).
• GraphTarget — the entry point. Three concrete kinds: View(name), LogicalTable(name), Resource(database, path).
• GraphProvider — SPI that turns a GraphTarget + depth + JDBC connection into a PipelineGraph. Registered via META-INF/services. Mirrors the existing DeployerProvider shape.
• GraphRenderer — SPI that turns a PipelineGraph into a string in some format. Also ServiceLoader-discovered.

The dispatcher (hoptimator-jdbc)

GraphService in hoptimator-jdbc is the runtime entry point — buildGraph(identifier, depth, conn), render(graph, format), availableFormats(). Sits alongside the other service classes (ValidationService). Doesn't know about K8s or Mermaid; just ServiceLoaders providers and renderers and dispatches.

A future deployment substrate (non-K8s, RPC-backed, in-process) or a different rendering target (D3 JSON for an interactive web view, DOT for graphviz, etc.) plugs in without touching anything in this PR.

The K8s provider (hoptimator-k8s)

K8sGraphProvider is the only provider in this PR. It uses PipelineGraphBuilder to traverse the K8s state: starts at the target, uses the depends-on-<slug> labels to find Pipelines and TableTriggers in O(matches) on the server, parses the
directional source/sink annotations to draw edges, and recurses up to a configurable depth.

• Direction-aware: UPSTREAM follows producers, DOWNSTREAM follows consumers. Self-loops and cycles pinned with a visited set + a unit test.
• SQL-side identifier resolution for !graph foo.bar — resolves against the Calcite catalog so users don't have to know the underlying <database>_<path> slug shape.
• Trigger node detail comes from the rendered Job's YAML (container name, JobTemplate name derived from metadata.name's <trigger>-<template> shape) and the spec (cron schedule humanized, paused state). No extra annotation needed.

The Mermaid renderer (hoptimator-graph)

New module — exists solely to hold the Mermaid renderer (and any future renderers we don't want bloating hoptimator-util). MermaidRenderer in com.linkedin.hoptimator.graph.mermaid is registered as the default for format "mermaid". Produces a
flowchart with subgraphs for LogicalTable tiers, distinct node shapes per type, and dotted edges for trigger flows. Other renderers can register against the same SPI without changing this one.

Test plan

• gradle test green across hoptimator-api, hoptimator-k8s, hoptimator-graph, hoptimator-logical, hoptimator-jdbc, hoptimator-util.
• PipelineGraphBuilderTest — view / MV-on-MV / LogicalTable / external-resource entry points, direction-aware traversal, depth-limit termination, cycle handling, stale-label rejection via annotation cross-check, YAML-derivation helpers.
• MermaidRendererTest — Mermaid output shape for each node/edge type.
• GraphServiceTest — SPI dispatch happy path + the "no provider supports target" error message.
• Quidem integration scripts (k8s-trigger-graph.id, logical-graph.id, etc.) assert on the exact rendered Mermaid output for canonical scenarios end-to-end against a real K8s cluster.

Known limitations

• This visualizer relies on labels that will not be present for existing pipelines

Examples

Command:

create or replace table "LOGICAL"."testevent" ("KEY_id" int, "memberId" bigint, "pageKey" varchar);

Result:

!graph LOGICAL.testevent

flowchart TD
    subgraph n0["LogicalTable logical-testevent"]
        direction LR
        subgraph s_nearline["nearline"]
            n2[("KAFKA.testevent")]
        end
        subgraph s_offline["offline"]
            n3[("MYSQL.testdb.testevent")]
        end
        subgraph s_online["online"]
            n5[("VENICE.testevent")]
        end
        n1[/"logical-testevent-nearline-to-offline<br/>kind: FlinkSessionJob<br/>engine: Flink"/]
        n4[/"logical-testevent-nearline-to-online<br/>kind: FlinkSessionJob<br/>engine: Flink"/]
        n6{"logical-testevent-offline-trigger<br/>(paused)"}
    end
    n2 --> n1
    n1 --> n3
    n2 --> n4
    n4 --> n5
    n3 -.-> n6
    n6 -.-> n5

Loading

Command:

create or replace materialized view ADS."PAGE_VIEWS$reverse-field-order" as select MEMBER_URN as MEMBER_URN, CAMPAIGN_URN as PAGE_URN from ADS.AD_CLICKS;
create or replace materialized view ADS.MULTI as select first_name, last_name from ads.page_views natural join PROFILE.MEMBERS;

Result:

!graph ADS.MULTI

flowchart LR
    subgraph n0["Materialized View"]
        n1[/"ads-multi<br/>kind: FlinkSessionJob<br/>engine: Flink"/]
    end
    n2[("ADS.PAGE_VIEWS")]
    n3[("PROFILE.MEMBERS")]
    n4[("ADS.MULTI")]
    n2 --> n1
    n3 --> n1
    n1 --> n4

Loading

!graph ADS.PAGE_VIEWS

flowchart LR
    n0[("ADS.PAGE_VIEWS")]
    n1[/"ads-pageviews-reverse-field-order<br/>kind: FlinkSessionJob<br/>engine: Flink"/]
    n2[("ADS.AD_CLICKS")]
    n3[/"ads-multi<br/>kind: FlinkSessionJob<br/>engine: Flink"/]
    n4[("PROFILE.MEMBERS")]
    n5[("ADS.MULTI")]
    n2 --> n1
    n1 --> n0
    n0 --> n3
    n4 --> n3
    n3 --> n5

Loading

[ryannedolan]

Can we see example input/output in the PR description? GH will render mermaid!

[github-actions]

Code Coverage

Overall Project	84.45% -0.95%	🟢
Files changed	87.38%	🟢

File	Coverage
CronHumanizer.java	100%	🟢
LogicalTableNames.java	100%	🟢
GraphEdge.java	98.28% -1.72%	🟢
LogicalTableSchema.java	97.72%	🟢
GraphNode.java	97.5% -2.5%	🟢
HoptimatorJdbcSchema.java	95.74% -0.43%	🟢
MermaidRenderer.java	93.61% -6.39%	🟢
K8sApi.java	92.76% -6.9%	❌
GraphTarget.java	86.89% -13.11%	🟢
GraphService.java	86.48% -13.52%	🟢
PipelineGraph.java	86% -14%	🟢
K8sGraphProvider.java	83.33% -16.67%	🟢
PipelineGraphBuilder.java	82.52% -17.48%	🟢
LogicalTableDeployer.java	76.08%	🟢

[jogrogan]

Can we see example input/output in the PR description? GH will render mermaid!

Done!

[ryannedolan]

Really neat. Wondering why we need the table/view/logical subcommand. Seems like we could auto-detect that?

[jogrogan]

Really neat. Wondering why we need the table/view/logical subcommand. Seems like we could auto-detect that?

This is what I am actively working on! claude did some weird stuff I am unwinding

[srnand]

coool!