Provide timeseries facade for Ditto feature properties

[thjaeckle]

Motivation

Ditto is often used in IoT landscapes together with timeseries databases. Common integration patterns include:

• Ditto ingesting data into a TS database (e.g., whenever a temperature value changes)
• Custom message handlers fetching historical data and returning it to callers

However, this integration requires significant manual effort. Ditto does not provide a native API for timeseries data, which complicates architectures and requires users to build custom solutions.

This issue proposes a new Ditto microservice called "Timeseries" that provides a unified facade for timeseries databases, similar to how Ditto's Search service provides a unified search API.

---

Concept Summary

A new Ditto microservice that automatically captures Thing property changes over time and provides a unified query API for historical timeseries data.

What It Does

Capability	Description
Automatic Ingestion	Property changes are automatically written to a timeseries database
Declarative Config	WoT ThingModel ditto:timeseries annotation defines which properties to track
Query API	RESTful API for retrieving historical data with aggregations
Access Control	Configurable policy permission (READ_TS or READ) controls who can access timeseries data
Pluggable Backend	Default MongoDB Time Series adapter; extensible for IoTDB, TimescaleDB, etc.

Architecture Overview

                                      Ditto Cluster
┌──────────────────────────────────────────────────────────────────────────────┐
│                                                                              │
│   ┌─────────────┐                                    ┌─────────────────────┐ │
│   │   Gateway   │──────────┐              ┌─────────│    Connectivity     │ │
│   │   Service   │          │    Query     │         │      Service        │ │
│   │ [TS Routes] │          │              │         │ [TS Message Handler]│ │
│   └─────────────┘          │              │         └─────────────────────┘ │
│                            ▼              ▼                                 │
│   ┌──────────┐       ┌───────────────────────┐                              │
│   │ Policies │◀──────│     Timeseries        │                              │
│   │ Service  │ fetch │       Service         │                              │
│   │          │policy │                       │                              │
│   └──────────┘ for   │ - Subscribes to       │                              │
│                perm.   things.ts-events:     │                              │
│                      │ - Enforces permission │                              │
│                      │ - Writes to TS DB     │                              │
│                      └───────────▲───────────┘                              │
│                                  │                                          │
│                                  │ Pub/Sub (things.ts-events: topic)        │
│                                  │                                          │
│                           ┌──────┴──────┐                                   │
│                           │   Things    │                                   │
│                           │   Service   │                                   │
│                           │[TS Publisher│                                   │
│                           └─────────────┘                                   │
│                                                                             │
└─────────────────────────────────────────────────────────────────────────────┘
                                   │
                                   ▼
                         ┌─────────────────┐
                         │  TS Database    │
                         │  (MongoDB TS,   │
                         │   IoTDB, etc.)  │
                         └─────────────────┘

Data flow:

1. Device updates a Thing property → Things service persists the event
2. If property has ditto:timeseries with ingest: "ALL" in WoT model → publish to Timeseries service via pub/sub
3. Timeseries service writes data point to timeseries database
4. Clients query historical data via Gateway → Timeseries service → TS database

---

HTTP API Overview

Important: The API uses a separate root /api/2/timeseries/ to avoid path collision with Thing JSON structure (see HTTP API design rationale).

Endpoint	Description
GET /api/2/timeseries/things/{thingId}/features/{featureId}/properties/{path}	Get timeseries for a single property
GET /api/2/timeseries/things/{thingId}/features/{featureId}/properties	Get timeseries for all TS-enabled properties
GET /api/2/timeseries/things/{thingId}/attributes/{path}	Get timeseries for an attribute
POST /api/2/timeseries/things/{thingId}	Batch query with multiple paths/aggregations
POST /api/2/timeseries/query	Cross-thing aggregation query (fleet analytics)

Common query parameters:

• from, to — Time range (ISO 8601 or relative like now-24h)
• step — Downsampling interval (1m, 5m, 1h, 1d)
• agg — Aggregation function (avg, min, max, sum, count, first, last, derivative, rate, integral, stddev, percentile)
• percentile — Percentile value (0-100), required when agg=percentile
• timeFormat — Timestamp format in response: iso (default, ISO 8601 string) or ms (milliseconds since epoch)

Ditto Protocol Topic Paths

The timeseries API integrates with the Ditto Protocol for WebSocket and Connectivity access:

Operation	Topic Path
Single-thing query	<ns>/<name>/things/twin/timeseries/retrieve
Cross-thing query	_/_/things/twin/timeseries/query

---

WoT ThingModel Configuration

Properties to track are declared via the ditto:timeseries WoT ThingModel extension:

{
  "@context": [
    "https://www.w3.org/2022/wot/td/v1.1",
    {"ditto": "https://ditto.eclipseprojects.io/wot/ditto-extension#"}
  ],
  "properties": {
    "temperature": {
      "type": "number",
      "unit": "cel",
      "ditto:timeseries": {
        "ingest": "ALL",
        "tags": {
          "attributes/building": "{{ thing-json:attributes/building }}"
        }
      }
    }
  }
}

ditto:timeseries Object Structure

Key	Type	Required	Description
ingest	string	Yes	Controls data ingestion: "ALL" or "NONE"
tags	object	No	Tags for grouping/filtering in aggregation queries

ingest values:

• "ALL" — Ingest all value changes into the timeseries database
• "NONE" — Disable ingestion (useful for temporarily pausing without removing config)

Future extension: The ingest field may be extended with conditional ingestion options (e.g., ingest only when value exceeds threshold).

Note on retention: Data retention (TTL) is configured at the timeseries database level (e.g., MongoDB collection TTL), not per-property. This ensures consistency across different TS backends.

---

Policy Integration

The permission required to access timeseries data is configurable:

Configuration Value	Behavior
READ_TS (default)	Users need explicit READ_TS permission (fine-grained access control)
READ	Users with READ permission can also read timeseries data (simplified management)

Policy example with READ_TS:

{
  "policyId": "org.eclipse.ditto:my-policy",
  "entries": {
    "owner": {
      "subjects": {"nginx:admin": {"type": "nginx basic auth"}},
      "resources": {
        "thing:/features/environment/properties/temperature": {
          "grant": ["READ", "WRITE", "READ_TS"],
          "revoke": []
        }
      }
    }
  }
}

---

Key Design Decisions

• Separate from event sourcing — TS data is optimized for time-range queries, not revision history
• Policy-controlled — Configurable permission (READ_TS or READ) controls timeseries access
• MongoDB default — Zero additional infrastructure using existing Ditto MongoDB with Time Series Collections
• Extensible — Adapter interface for dedicated TS databases (IoTDB, TimescaleDB, InfluxDB)
• Loose coupling — Things service publishes standard ThingEvents; Timeseries service transforms them
• Unified annotation — Single ditto:timeseries object groups all TS configuration for a property

---

Implementation Phases

1. Phase 1 (MVP): Service skeleton, MongoDB adapter, basic GET endpoints, policy integration, ditto:timeseries with ingest only
2. Phase 2: Aggregations, downsampling, batch queries
3. Phase 3: Cross-thing aggregation queries with RQL filters, full ditto:timeseries.tags support
4. Phase 4: Connectivity integration, Ditto Protocol support, additional adapters

---

Related Resources

• Detailed design document: (internal concept document)
• Similar pattern: Ditto Search service (CQRS projection of Thing events)