Pluggable graph interface for Hoptimator topology + Mermaid renderer

deploy/samples/logicaldb.yaml

@@ -21,5 +21,5 @@ metadata:
   name: logical-offline
 spec:
   url: jdbc:logical://nearline=ads-database;offline=ads-catalog-database
-  schema: LOGICAL_OFFLINE
+  schema: LOGICAL-OFFLINE
   dialect: Calcite

docs/extending/index.md

@@ -6,17 +6,19 @@ both — pick the layer that matches what you're doing.
 
 ## Pick the right surface
 
-| You want to…                                                                                  | What you'll write                                                  |
-| --------------------------------------------------------------------------------------------- | ------------------------------------------------------------------ |
-| Connect a new external system to the catalog (Kafka, Venice, MySQL, your-system).             | A JDBC adapter + `TableTemplate` / `JobTemplate`. See [Data sources](data-sources.md). |
-| Send Hoptimator-generated specs somewhere other than Kubernetes.                              | A `Deployer` + `DeployerProvider`. See [Deployers](deployers.md).   |
-| Reject SQL or YAML that's invalid in your environment before it deploys.                      | A `Validator` + `ValidatorProvider`. See [Validators](validators.md). |
-| Pull configuration values from somewhere other than `hoptimator-configmap`.                   | A `ConfigProvider`. See [Config providers](config-providers.md).    |
-| Customize what gets deployed for an existing system.                                          | Just a `TableTemplate` or `JobTemplate` — no Java needed. See [Templates and configuration](../kubernetes/templates.md). |
+| You want to…                                                                                        | What you'll write                                                  |
+|-----------------------------------------------------------------------------------------------------| ------------------------------------------------------------------ |
+| Connect a new external system to the catalog (Kafka, Venice, MySQL, your-system).                   | A JDBC adapter + `TableTemplate` / `JobTemplate`. See [Data sources](data-sources.md). |
+| Send Hoptimator-generated specs somewhere other than Kubernetes.                                    | A `Deployer` + `DeployerProvider`. See [Deployers](deployers.md).   |
+| Reject SQL or YAML that's invalid in your environment before it deploys.                            | A `Validator` + `ValidatorProvider`. See [Validators](validators.md). |
+| Pull configuration values from somewhere other than `hoptimator-configmap`.                         | A `ConfigProvider`. See [Config providers](config-providers.md).    |
+| Build a dependency graph from some backing store (e.g. K8s).                                        | A `GraphProvider`. The K8s-backed default ships in `hoptimator-k8s`. |
+| Render the dependency graph in a format other than Mermaid (DOT, JSON, an interactive web view, …). | A `GraphRenderer`. The Mermaid default ships in `hoptimator-graph`. |
+| Customize what gets deployed for an existing system.                                                | Just a `TableTemplate` or `JobTemplate` — no Java needed. See [Templates and configuration](../kubernetes/templates.md). |
 
 ## How extensions are loaded
 
-All four extension points are loaded via Java's `ServiceLoader`. To register
+All extension points are loaded via Java's `ServiceLoader`. To register
 an implementation, drop a service file under
 `src/main/resources/META-INF/services/` named after the SPI interface:
 
@@ -26,6 +28,8 @@ META-INF/services/com.linkedin.hoptimator.ValidatorProvider
 META-INF/services/com.linkedin.hoptimator.ConfigProvider
 META-INF/services/com.linkedin.hoptimator.ConnectorProvider
 META-INF/services/com.linkedin.hoptimator.CatalogProvider
+META-INF/services/com.linkedin.hoptimator.graph.GraphProvider
+META-INF/services/com.linkedin.hoptimator.graph.GraphRenderer
 ```
 
 Each file contains the fully qualified class name(s) of your
@@ -66,6 +70,22 @@ mutation, and the SQL/YAML is rejected if a validator returns errors.
 Common uses: naming conventions, schema compatibility, ACL checks. See
 [Validators](validators.md).
 
+### "I want to visualize what's deployed differently"
+
+The `!graph` CLI command (see
+[SQL CLI → !graph](../user-guide/sql-cli.md#graph-identifier---depth-n))
+goes through two SPIs: `GraphProvider` builds the typed
+`PipelineGraph` from some backing store, and `GraphRenderer` serializes
+it to a string. The bundled defaults are a K8s-backed
+`K8sGraphProvider` (in `hoptimator-k8s`) and a Mermaid `MermaidRenderer`
+(in `hoptimator-graph`).
+
+Add a `GraphRenderer` to support a new output format (e.g. DOT for
+graphviz, a JSON shape for a web UI). Add a `GraphProvider` if the
+pipeline state lives somewhere other than Kubernetes — the K8s
+implementation is the reference. Both register via `META-INF/services`
+like every other SPI here.
+
 ## Register, then test
 
 After dropping a service file and a class, the standard verification path

docs/getting-started/architecture.md

@@ -142,24 +142,25 @@ the pipeline.
 
 The repo is split into focused modules. The ones you'll touch most often:
 
-| Module                            | Role                                                                 |
-| --------------------------------- | -------------------------------------------------------------------- |
-| `hoptimator-api`                  | The interfaces. `Deployer`, `Engine`, `Connector`, `View`, etc.      |
-| `hoptimator-jdbc`                 | Calcite-based JDBC driver. Catalog, parser, planner integration.    |
-| `hoptimator-jdbc-driver`          | Lightweight wrapper that exposes the driver to standard JDBC code.   |
-| `hoptimator-cli`                  | The `./hoptimator` SQL CLI (sqlline + custom commands).              |
-| `hoptimator-mcp-server`           | MCP server that wraps the JDBC driver for AI agents and IDEs.        |
-| `hoptimator-util`                 | Planner rules, deployment service, template engine.                  |
-| `hoptimator-k8s`                  | Default Deployers, the catalog/operator glue, all CRDs.              |
-| `hoptimator-operator`             | The reconciler loop and its main entry point.                        |
-| `hoptimator-flink-runner`         | The runtime that executes Flink SQL jobs produced by the planner.    |
-| `hoptimator-flink-adapter`        | Flink-side adapter for catalog awareness.                            |
-| `hoptimator-kafka` / `-kafka-controller` | Kafka catalog and controller integration.                     |
-| `hoptimator-venice`               | Venice catalog adapter.                                              |
-| `hoptimator-mysql`                | MySQL catalog adapter.                                               |
-| `hoptimator-logical`              | LogicalTable support — one logical entity, multiple physical tiers.  |
-| `hoptimator-demodb`               | In-memory demo source used by the quickstart.                        |
-| `hoptimator-avro`                 | Avro schema utilities used by the catalog/connectors.                |
+| Module                                   | Role                                                                |
+|------------------------------------------|---------------------------------------------------------------------|
+| `hoptimator-api`                         | The interfaces. `Deployer`, `Engine`, `Connector`, `View`, etc.     |
+| `hoptimator-jdbc`                        | Calcite-based JDBC driver. Catalog, parser, planner integration.    |
+| `hoptimator-jdbc-driver`                 | Lightweight wrapper that exposes the driver to standard JDBC code.  |
+| `hoptimator-cli`                         | The `./hoptimator` SQL CLI (sqlline + custom commands).             |
+| `hoptimator-mcp-server`                  | MCP server that wraps the JDBC driver for AI agents and IDEs.       |
+| `hoptimator-util`                        | Planner rules, deployment service, template engine.                 |
+| `hoptimator-k8s`                         | Default Deployers, the catalog/operator glue, all CRDs.             |
+| `hoptimator-operator`                    | The reconciler loop and its main entry point.                       |
+| `hoptimator-flink-runner`                | The runtime that executes Flink SQL jobs produced by the planner.   |
+| `hoptimator-flink-adapter`               | Flink-side adapter for catalog awareness.                           |
+| `hoptimator-kafka` / `-kafka-controller` | Kafka catalog and controller integration.                           |
+| `hoptimator-venice`                      | Venice catalog adapter.                                             |
+| `hoptimator-mysql`                       | MySQL catalog adapter.                                              |
+| `hoptimator-logical`                     | LogicalTable support — one logical entity, multiple physical tiers. |
+| `hoptimator-graph`                       | Pipeline graph renderers. Ships the Mermaid backend for `!graph`.   |
+| `hoptimator-demodb`                      | In-memory demo source used by the quickstart.                       |
+| `hoptimator-avro`                        | Avro schema utilities used by the catalog/connectors.               |
 
 A handful of modules (`hoptimator-catalog`, `hoptimator-models`,
 `hoptimator-planner`) are in the tree but marked for deletion; new
@@ -179,6 +180,11 @@ contributions should not target them.
   `DeployerProvider`. Kubernetes is the default but not a hard requirement;
   the bundled `K8sSourceDeployer` and `K8sJobDeployer` are themselves
   examples. Anything that knows how to materialize a spec will do.
+- **A new visualization format or graph backend**: implement `GraphRenderer`
+  for a new output format (DOT, JSON, an interactive HTML view, …) or
+  `GraphProvider` to build the dependency graph from somewhere other than
+  Kubernetes. The bundled Mermaid renderer (`hoptimator-graph`) and K8s
+  provider (`hoptimator-k8s`) are reference impls.
 - **Different cluster configuration**: usually a `ConfigProvider` change
   rather than code.
 

docs/getting-started/concepts.md

@@ -99,7 +99,7 @@ right bootstrap servers and topic name. Connectors are produced by the
 catalog adapter for each database, embedded in the YAML that
 [TableTemplates](#tabletemplates-and-jobtemplates) and
 [JobTemplates](#tabletemplates-and-jobtemplates) emit, and can be customized
-via [hints](#hints).
+via [hints](#configuration-and-hints).
 
 Connectors do not require an `Engine` to function. The typical flow is:
 Hoptimator generates a `FlinkSessionJob` (or similar) with the connector

docs/index.md

@@ -23,7 +23,8 @@ Start with **[Getting started](getting-started/index.md)**:
 See the **[User guide](user-guide/index.md)**:
 
 - [SQL CLI](user-guide/sql-cli.md) — sqlline-based interactive shell with
-  `!pipeline`, `!specify`, `!resolve` for inspecting plans before they deploy.
+  `!pipeline`, `!specify`, `!resolve` for inspecting plans before they
+  deploy, and `!graph` for visualizing what's already running.
 - [JDBC driver](user-guide/jdbc.md) — `jdbc:hoptimator://` for Java apps,
   with full connection-property reference.
 - [MCP server](user-guide/mcp-server.md) — Model Context Protocol server
@@ -64,6 +65,8 @@ See **[Extending Hoptimator](extending/index.md)**:
   via `Validator` and `ValidatorProvider`.
 - [Config providers](extending/config-providers.md) — custom
   `ConfigProvider` SPI.
+- [Pipeline graph SPIs](extending/index.md) — `GraphProvider` (alternate backing store) and
+  `GraphRenderer` (alternate output format).
 - [Templates and configuration](kubernetes/templates.md) — authoring
   `TableTemplate` and `JobTemplate` (lives in the Kubernetes guide).
 

docs/user-guide/sql-cli.md

@@ -33,8 +33,8 @@ See [JDBC driver](jdbc.md) for the full URL syntax.
 ## Built-in commands
 
 Standard sqlline commands all work (`!help`, `!quit`, `!run`, `!record`,
-…) along with the catalog-introspection ones below. Hoptimator adds the
-last four for inspecting plans and pipelines.
+…) along with the catalog-introspection ones below. Hoptimator also adds
+commands to inspect plans, pipelines, and the deployed graph.
 
 | Command       | What it does                                                                  |
 | ------------- | ----------------------------------------------------------------------------- |
@@ -44,9 +44,11 @@ last four for inspecting plans and pipelines.
 | `!resolve`    | Print the schema and source/sink connector configs Hoptimator would use for a table. |
 | `!pipeline`   | Print the auto-generated pipeline SQL for a SELECT or CREATE MATERIALIZED VIEW statement. |
 | `!specify`    | Print every Kubernetes spec the statement would deploy. The dry-run for `CREATE MATERIALIZED VIEW`. |
+| `!graph`      | Render the deployed dependency graph rooted at an identifier as a Mermaid diagram. |
 
-`!resolve`, `!pipeline`, and `!specify` do not modify any state. Use them to
-sanity-check a plan before you let the JDBC driver actually deploy it.
+`!resolve`, `!pipeline`, `!specify`, and `!graph` do not modify any state.
+Use them to sanity-check a plan before you let the JDBC driver actually
+deploy it (and to inspect what's already running).
 
 ### `!resolve <schema.table>`
 
@@ -115,6 +117,61 @@ If you'd `kubectl apply` the output, you'd get the same result as actually
 running the `CREATE MATERIALIZED VIEW`. This is the safest way to review what
 a statement will do before you run it.
 
+### `!graph <identifier> [--depth N]`
+
+```sql
+0: Hoptimator> !graph ADS.AUDIENCE
+flowchart LR
+    subgraph n0["Materialized View"]
+        n1[/"ads-audience
+kind: SqlJob
+engine: Flink
+mode: Streaming"/]
+end
+    n2[("ADS.PAGE_VIEWS")]
+    n3[("PROFILE.MEMBERS")]
+    n4[("ADS.AUDIENCE")]
+    n2 --> n1
+    n3 --> n1
+    n1 --> n4
+```
+Rendered:
+```mermaid
+flowchart LR
+    subgraph n0["Materialized View"]
+        n1[/"ads-audience
+kind: SqlJob
+engine: Flink
+mode: Streaming"/]
+    end
+    n2[("ADS.PAGE_VIEWS")]
+    n3[("PROFILE.MEMBERS")]
+    n4[("ADS.AUDIENCE")]
+    n2 --> n1
+    n3 --> n1
+    n1 --> n4
+```
+
+Renders the deployed dependency graph rooted at `<identifier>` as Mermaid.
+Identifier resolution  runs against Calcite's catalog, so the same names you use in SQL work here:
+
+- A materialized view (`ADS.AUDIENCE`) renders the view's compiled pipeline
+  with its direct sources and sink.
+- A logical table (`LOGICAL.events`) renders the inter-tier pipelines,
+  any owned triggers, and the per-tier physical resources grouped into
+  tier subgraphs.
+- A physical resource (`KAFKA.events`) traverses the depends-on dependency
+  index up to `--depth` hops in each direction — pipelines that read or
+  write it, and recursively their other endpoints.
+
+`--depth N` only applies to physical-resource targets; view and logical-table
+graphs are intentionally single-hop ("what this view does," not the full
+upstream chain). For the chain, run `!graph` on a source identifier.
+
+Rendering backends are pluggable. Mermaid is the default and the only one
+shipped today; additional renderers can register via the
+`GraphRenderer` SPI — see [Extending Hoptimator](../extending/index.md).
+
 ## Running SQL
 
 Hoptimator supports the SQL surface described in

hoptimator-api/src/main/java/com/linkedin/hoptimator/graph/GraphEdge.java

@@ -0,0 +1,64 @@
+package com.linkedin.hoptimator.graph;
+
+import java.util.Objects;
+
+
+/**
+ * A directed edge in a pipeline visualization graph. Edges are equal when their endpoints and
+ * type match — two edges of different types between the same nodes are distinct (e.g. an
+ * {@code ownerOf} relationship coexists with a {@code dependsOnSink} relationship).
+ */
+public final class GraphEdge {
+
+  public enum Type {
+    /** {@code metadata.ownerReferences} cascade — drives subgraph membership, not arrows. */
+    OWNER_OF,
+    /** Resource → pipeline (or job) edge derived from the {@code depends-on} annotation. */
+    DEPENDS_ON_SOURCE,
+    /** Pipeline (or job) → resource edge derived from the {@code depends-on} annotation. */
+    DEPENDS_ON_SINK,
+    /** Trigger → job/pipeline; rendered as a dotted line. */
+    TRIGGERS
+  }
+
+  private final GraphNode from;
+  private final GraphNode to;
+  private final Type type;
+
+  public GraphEdge(GraphNode from, GraphNode to, Type type) {
+    this.from = Objects.requireNonNull(from, "from");
+    this.to = Objects.requireNonNull(to, "to");
+    this.type = Objects.requireNonNull(type, "type");
+  }
+
+  public GraphNode from() {
+    return from;
+  }
+
+  public GraphNode to() {
+    return to;
+  }
+
+  public Type type() {
+    return type;
+  }
+
+  @Override
+  public boolean equals(Object o) {
+    if (!(o instanceof GraphEdge)) {
+      return false;
+    }
+    GraphEdge other = (GraphEdge) o;
+    return type == other.type && from.equals(other.from) && to.equals(other.to);
+  }
+
+  @Override
+  public int hashCode() {
+    return Objects.hash(from, to, type);
+  }
+
+  @Override
+  public String toString() {
+    return from.id() + " --" + type + "--> " + to.id();
+  }
+}