Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,2 +1,10 @@
**/__pycache__/
**/.venv/
*.egg-info/

# Local secrets / credentials — never commit
.env
.env.local
*.env
.secrets*
.ataccama.env
154 changes: 154 additions & 0 deletions converters/ataccama/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,154 @@
# Ataccama OSI Converter

Importer from [Ataccama ONE](https://docs.ataccama.com/ataccama-one-agentic/latest/rest-api/rest-api-overview.html)
catalog metadata to the [OSI (Open Semantic Interchange)](https://github.com/open-semantic-interchange/OSI)
semantic model specification.

Given one or more Ataccama **catalog items** (selected by URN), the converter produces
a single OSI semantic model: each catalog item becomes a dataset, each catalog attribute
becomes a field, business terms are carried across as `ai_context`, and
**data-quality scores** plus other governance metadata are attached as `ATACCAMA`
custom extensions.

## Scope & direction

- **Direction:** Ataccama → OSI (import only). See [Limitations](#limitations) for why
export (OSI → Ataccama) is not provided.
- **Input:** explicit catalog-item URNs. Ataccama catalogs are large (tens of thousands
of items, many of which are BI artifacts rather than tables), so the converter is
intentionally scoped to the items you name rather than crawling everything.
- **Data access:** a live REST client that authenticates via OAuth2 client-credentials
(tokens are short-lived and refreshed automatically) and follows the catalog API's
cursor pagination.

## Configuration

The client reads connection settings from environment variables (or a `--env-file`):

| Variable | Description |
|---|---|
| `ATACCAMA_BASE_URL` | API root, e.g. `https://<host>/api` |
| `ATACCAMA_TOKEN_URL` | OAuth2 token endpoint (Keycloak `.../protocol/openid-connect/token`) |
| `ATACCAMA_CLIENT_ID` | service-account client id |
| `ATACCAMA_CLIENT_SECRET` | service-account client secret |

> Never commit credentials. Keep them in a git-ignored env file.

## Usage

```bash
# CLI
ataccama-to-osi --env-file .ataccama.env \
--urn urn:ata:<tenant>:catalog:catalog-item:<id> \
--urn urn:ata:<tenant>:catalog:catalog-item:<id> \
--model-name my_model \
--output my_model.osi.yaml

# data-quality results are fetched by default; use --no-dq to skip them
ataccama-to-osi --env-file .ataccama.env --urn <urn> --no-dq -o my_model.osi.yaml

# opt-in: append DQ warnings to ai_context for datasets Ataccama flags as below quality
ataccama-to-osi --env-file .ataccama.env --urn <urn> --dq-ai-warnings -o my_model.osi.yaml

# or supply a file of URNs (one per line)
ataccama-to-osi --env-file .ataccama.env --urns-file items.txt -o my_model.osi.yaml
```

```python
# Library
from ataccama_osi import AtaccamaClient, ataccama_to_osi
import yaml

client = AtaccamaClient(base_url="https://<host>/api", token_url="...",
client_id="...", client_secret="...")
bundles = [client.fetch_bundle(urn) for urn in urns]
document = ataccama_to_osi(bundles, model_name="my_model")
print(yaml.dump(document, sort_keys=False))
```

## Development

```bash
uv sync --group dev # or: python -m venv .venv && pip install -e . pytest jsonschema
uv run pytest
```

Tests run fully offline against a recorded catalog fixture
(`tests/fixtures/ataccama_bundles.json`) and validate the output against the OSI schema.

## Concept mapping

| Ataccama ONE (Catalog API) | OSI Semantic Model |
|---|---|
| `CatalogItem` | `dataset` |
| `CatalogItem.name` | `dataset.name` (de-duplicated if repeated) |
| `CatalogItem.locations` + name | `dataset.source` (best-effort dotted namespace) |
| `CatalogItem.description` (rich text) | `dataset.description` (flattened to plain text) |
| assigned `Term`s | `dataset.ai_context` (`synonyms` + `instructions`) |
| `CatalogAttribute` | `field` |
| `CatalogAttribute.name` | `field.name` + quoted `ANSI_SQL` identifier expression |
| `CatalogAttribute.dataType` ∈ {DATE, DATETIME, TIMESTAMP, TIME} | `field.dimension.is_time = true` |
| `CatalogAttribute.description` / `comment` | `field.description` |
| attribute `Term`s | `field.ai_context` |
| DQ `overallQuality` + `dimensionResults` + findings + monitor threshold | dataset `custom_extensions` (`vendor_name: ATACCAMA`) → `data.dq` (`passed`, `failed`, `pass_rate_pct`, `threshold_pct`, `below_threshold`, `active_findings`, `dimensions[]`, `results_link`) |
| DQ per-attribute `overallQuality` | field `custom_extensions` (`vendor_name: ATACCAMA`) → `data.dq` (`passed`, `failed`, `pass_rate_pct`) |
| URNs, `dataType`, `columnType`, connection/source/stewardship/monitor | `custom_extensions` (`vendor_name: ATACCAMA`) → `data` |

### Data quality

Data-quality results (from the DQ API's latest processing on each item's primary
monitor) are fetched by default and attached to the `ATACCAMA` extension — overall
and per-dimension quality on the dataset, and per-column quality on each field. Pass
`--no-dq` to skip the DQ calls.

Notes:
- `pass_rate_pct` is Ataccama's overall quality (`overallQuality`) expressed as a
percentage: `passed / (passed + failed)`. The DQ API returns only the pass/fail
**counts**, not a preformatted percentage, so the converter formats it — the counts
themselves are Ataccama's own, consistent with what the platform reports.
- `threshold_pct` / `below_threshold` come from the monitor's configured overall
threshold (`overallDqThresholds`, fetched from the monitor-config endpoint) — i.e.
the same pass/fail bar shown in Ataccama. They are omitted when the monitor has no
overall threshold configured.
- `active_findings` is always reported (`0` = no open data-quality findings), so the
clean state is an explicit signal rather than an omission.
- Dimensions with no evaluated records in the processing are omitted.
- Items without a published monitor/results simply carry no `dq` block (best-effort).

#### AI warnings (opt-in)

By default the DQ block is descriptive data only. With `--dq-ai-warnings`, a plain-language
warning is appended to `ai_context.instructions` for any dataset Ataccama flags as having
data quality issues (data quality below configured threshold or active findings), so a
downstream AI/BI tool reading the OSI model (not just the vendor extension) is told to
treat the data with caution. Example:

> *Data-quality warning: 60.0% of quality checks passed on the latest run (below the
> configured 75% quality threshold). Verify this data before using it for analysis or
> automated decisions.*

Warnings are appended to any existing (term-derived) instructions, never replacing them,
and require DQ (so not usable with `--no-dq`).

## Limitations

The Ataccama Catalog API is a **catalog / governance / data-quality** surface, while OSI
is an **analytics semantic model**. Some OSI constructs therefore have no source today:

- **Metrics** — Ataccama has no analytics metrics; none are emitted. (Data-quality
scores are attached as `ATACCAMA` extensions, not as OSI metrics.)
- **Data Trust Score / Index** — not exposed by the public APIs (no precomputed
score field exists), so it is not emitted. The converter instead carries the DQ
results and monitor threshold that the APIs do provide.
- **Relationships / `primary_key` / `unique_keys`** — the Catalog API does not expose
foreign or primary keys on catalog items, so these are omitted.
- **`source` string** — the API exposes only a folder hierarchy (`locations`) and
`originPath`, not `database.schema.table`. `source` is a best-effort dotted namespace;
the authoritative connection/source URNs are preserved in `custom_extensions`.
- **Expressions are physical** — attributes map to plain quoted column identifiers
(`ANSI_SQL` only); there are no computed/multi-dialect expressions.
- **Export (OSI → Ataccama)** is not provided: the public Catalog API can only `PATCH`
`description`/stewardship/aliases on existing items and cannot create catalog items,
attributes, or relationships.
- Items with no catalogued attributes (e.g. some dashboards/reports) produce a dataset
with no `fields`.
44 changes: 44 additions & 0 deletions converters/ataccama/pyproject.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
[project]
name = "ataccama-osi"
version = "0.2.0.dev0"
description = "Importer from Ataccama ONE catalog metadata to the OSI semantic model"
authors = [{name = "Ataccama", email = "support@ataccama.com"}]
readme = "README.md"
requires-python = ">=3.12"
dependencies = [
"pyyaml>=6.0.1",
"requests>=2.31.0",
]

[project.scripts]
ataccama-to-osi = "ataccama_osi.cli:main"

[dependency-groups]
dev = [
"ruff>=0.11.5",
"pytest>=8.3.5",
"pytest-cov>=6.0.0",
]

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[tool.hatch.build]
include = ["src/ataccama_osi/**/*"]
ignore-vcs = true

[tool.hatch.build.targets.wheel]
packages = ["src/ataccama_osi"]

[tool.ruff]
line-length = 120
target-version = "py312"

[tool.ruff.lint]
select = ["E", "F", "I", "N", "UP", "W", "C4", "PIE", "SIM"]
ignore = ["E501"]

[tool.pytest.ini_options]
testpaths = ["tests"]
pythonpath = ["src"]
14 changes: 14 additions & 0 deletions converters/ataccama/src/ataccama_osi/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
"""Ataccama ONE -> OSI semantic model importer."""

from ataccama_osi.ataccama_to_osi import ataccama_to_osi
from ataccama_osi.client import AtaccamaClient
from ataccama_osi.models import CatalogAttribute, CatalogItem, CatalogItemBundle, Term

__all__ = [
"AtaccamaClient",
"CatalogAttribute",
"CatalogItem",
"CatalogItemBundle",
"Term",
"ataccama_to_osi",
]
Loading