Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
95 changes: 48 additions & 47 deletions core-spec/expression_language.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Ossie Proposal: Expression Language

**Current Status:** Proposed Final
**Current Status:** Proposed Final

**Working Group**

Expand All @@ -14,7 +14,7 @@

There are two layers in Ossie that need an expression language:

* **Ontology layer.** This layer maps onto the ontology layer which sits above the logical layer. It maps more closely to modelling languages like OWL, [(Py)Rel](https://docs.relational.ai) from RelationalAI, and [Legend](https://legend.finos.org) from Goldman Sachs
* **Ontology layer.** This layer maps onto the ontology layer which sits above the logical layer. It maps more closely to modelling languages like OWL, [(Py)Rel](https://docs.relational.ai) from RelationalAI, and [Legend](https://legend.finos.org) from Goldman Sachs
* **Logical layer.** This layer maps directly to the databases and physical layer. It maps closely to traditional BI semantic models.

This proposal is only targeted at the Logical Layer. It would be nice if the Ontological layer could re-use the same expression language, but that will be treated as a separate proposal.
Expand All @@ -25,22 +25,22 @@ We expect there will be extensions to this language to cover concepts such as su

### Design Principles

1. **Portability**: Core functions work identically across all implementations
2. **Familiarity**: Based on widely-adopted SQL syntax and semantics
3. **Analytical Focus**: Prioritizes functions commonly used in BI and analytics
1. **Portability**: Core functions work identically across all implementations
2. **Familiarity**: Based on widely-adopted SQL syntax and semantics
3. **Analytical Focus**: Prioritizes functions commonly used in BI and analytics
4. **Extensibility**: Vendor dialects can extend beyond the core

### Changes to YAML

1) Create a new dialect in the Ossie spec: Ossie\_SQL\_2026, which refers to this language specification.
1) Create a new dialect in the Ossie spec: Ossie\_SQL\_2026, which refers to this language specification.
2) Make Ossie\_SQL\_2026 the default dialect if one is not chosen.

### Standards Reference

The core language is based on **ANSI SQL:2003 Core** (ISO/IEC 9075-2:2003), selected for its:

- Wide adoption across major databases (Snowflake, Databricks, PostgreSQL, BigQuery)
- Well-defined semantics
- Wide adoption across major databases (Snowflake, Databricks, PostgreSQL, BigQuery)
- Well-defined semantics
- Support for modern analytical features (window functions, CTEs)

### Namespacing and Identifier Resolution
Expand All @@ -57,7 +57,7 @@ All identifiers MUST be valid names and follow ANSI SQL naming, with the size li

Regular identifiers (unquoted) should be case insensitive. For example, an identifier id is regular, so it would match with Id or iD. Comparing quoted and non-quoted identifiers is DB specific, so for best portability it is best to use simple identifiers.

The quote character for the Ossie dialect will follow ANSI SQL and support the double quote character (“). This means that if an expression is in a field expression or as an identifier in the YAML, this will be the expected quoting. However, there are some databases that use other escape characters. Working with these have the option of either creating expressions using their dialect or having the Ossie document written in the Ossie dialect, but then having the SQL Interface queried in the local dialect. The SQL Interface will be defined in a different document.
The quote character for the Ossie dialect will follow ANSI SQL and support the double quote character (“). This means that if an expression is in a field expression or as an identifier in the YAML, this will be the expected quoting. However, there are some databases that use other escape characters. Working with these have the option of either creating expressions using their dialect or having the Ossie document written in the Ossie dialect, but then having the SQL Interface queried in the local dialect. The SQL Interface will be defined in a different document.

#### Comparison Table

Expand All @@ -70,7 +70,7 @@ The quote character for the Ossie dialect will follow ANSI SQL and support the d

Sometimes, we may refer to a **normalized identifier**. This is a form the identifiers can be put in, so they can be matched easily and matches can be made with case-sensitive, exact matching. For **normalized identifiers**:

* Regular identifiers are upper cased
* Regular identifiers are upper cased
* Quoted identifiers have their quotes stripped and any escaped characters are unescaped

#### Name Spaces
Expand Down Expand Up @@ -100,7 +100,7 @@ Ossie expressions support the following SQL constructs within any expression:
| Scalar functions | See function categories below |
| Parentheses | Expression grouping |

###
###

### Not Supported in Expressions

Expand All @@ -119,13 +119,14 @@ Ossie expressions support the following SQL constructs within any expression:

Standard SQL operator precedence applies (highest to lowest):

1. Parentheses `()`
2. Unary operators: `+`, `-`, `NOT`
3. Multiplication/Division: `*`, `/`, `%`
4. Addition/Subtraction: `+`, `-`
5. Comparison: `=`, `<>`, `<`, `>`, `<=`, `>=`, `LIKE`, `IN`, `BETWEEN`, `IS NULL`
6`AND`
7`OR`
1. Parentheses `()`
2. Unary operators: `+`, `-`
3. Multiplication/Division: `*`, `/`, `%`
4. Addition/Subtraction/Concatenation: `+`, `-`, `||`
5. Comparison: `=`, `<>`, `<`, `>`, `<=`, `>=`, `LIKE`, `IN`, `BETWEEN`, `IS NULL`
6. `NOT`
7. `AND`
8. `OR`

---

Expand Down Expand Up @@ -180,7 +181,7 @@ APPROX_COUNT_DISTINCT(customer_id)
-- Approximate median
APPROX_PERCENTILE(amount, 0.5)

-- Approximate 95th percentile
-- Approximate 95th percentile
APPROX_PERCENTILE(response_time, 0.95)
```

Expand All @@ -197,7 +198,7 @@ APPROX_PERCENTILE(response_time, 0.95)

### Conditional Aggregations (REQUIRED)

SUM / COUNT aggregation functions support `DISTINCT.`
SUM / COUNT aggregation functions support `DISTINCT.`
All aggregations should support filtered aggregation:

```sql
Expand Down Expand Up @@ -262,8 +263,8 @@ DATE_PART('day', date_expr)

Supported date parts for `EXTRACT` and `DATE_PART`:

- `YEAR`, `QUARTER`, `MONTH`, `WEEK`, `DAY`
- `DAYOFWEEK`, `DAYOFYEAR`
- `YEAR`, `QUARTER`, `MONTH`, `WEEK`, `DAY`
- `DAYOFWEEK`, `DAYOFYEAR`
- `HOUR`, `MINUTE`, `SECOND`, `MILLISECOND`

### Date Truncation (REQUIRED)
Expand Down Expand Up @@ -321,13 +322,13 @@ model and behave identically across engines, making them the portable default.

Parsing with an explicit format string relies on a datetime format model whose token
vocabulary differs across engines (Oracle/`TO_CHAR`-style, `strftime` `%`-codes, and
Java/LDML patterns are all in use).
Java/LDML patterns are all in use).

For portability, we are looking to restrict the `format` argument to
the portable core format tokens defined in Date Formatting below, and prefer the
single-argument, ISO-8601 forms above where possible.
single-argument, ISO-8601 forms above where possible.

**Since, this differs so widely across databases, consider this experimental for now.**
**Since, this differs so widely across databases, consider this experimental for now.**

| Function | Syntax | Description |
| :---- | :---- | :---- |
Expand All @@ -341,9 +342,9 @@ single-argument, ISO-8601 forms above where possible.
| `TO_CHAR` | `TO_CHAR(date_expr, format)` | Format date as string |

Ossie defines a portable core of format tokens: the tokens that can be expressed in
every major engine's datetime format model.
every major engine's datetime format model.

This feature is experimental. Implementations choosing to support these should support the following tokens for the
This feature is experimental. Implementations choosing to support these should support the following tokens for the
`format` argument of `TO_CHAR`. The `strftime` and Java/LDML columns below are informative,
provided to aid translation.

Expand Down Expand Up @@ -412,7 +413,7 @@ a dialect extension.

Pattern wildcards for `LIKE`:

- `%` \- Match any sequence of characters
- `%` \- Match any sequence of characters
- `_` \- Match any single character

### Regular Expressions (RECOMMENDED)
Expand Down Expand Up @@ -536,8 +537,8 @@ function_name(args) OVER (

Frame clause options:

- `ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW`
- `ROWS BETWEEN n PRECEDING AND n FOLLOWING`
- `ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW`
- `ROWS BETWEEN n PRECEDING AND n FOLLOWING`
- `RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW`

### Ranking Functions (REQUIRED)
Expand Down Expand Up @@ -591,13 +592,13 @@ CAST(expression AS target_type)

Supported target types:

- `VARCHAR` / `STRING` \- Character string
- `INTEGER` / `INT` / `BIGINT` \- Integer
- `DECIMAL` / `NUMERIC` \- Fixed-point decimal
- `FLOAT` / `DOUBLE` \- Floating-point
- `BOOLEAN` \- Boolean
- `DATE` \- Date
- `TIMESTAMP` \- Timestamp
- `VARCHAR` / `STRING` \- Character string
- `INTEGER` / `INT` / `BIGINT` \- Integer
- `DECIMAL` / `NUMERIC` \- Fixed-point decimal
- `FLOAT` / `DOUBLE` \- Floating-point
- `BOOLEAN` \- Boolean
- `DATE` \- Date
- `TIMESTAMP` \- Timestamp
- `TIME` \- Time

### TRY\_CAST (RECOMMENDED)
Expand All @@ -624,7 +625,7 @@ a IS DISTINCT FROM b -- TRUE if one is NULL and other isn't

Ossie implementations MAY support additional functions through dialect-specific extensions. When using dialect extensions, the expression must specify the dialect.

The Ossie dialect should always be supported. Other dialects MAY be ignored. There is no guarantee that all different dialects for an expression will act the same, so implementations should be consistent with their dialect handling. This means that if an Ossie model has an expression written in two dialects, the implementation should deterministically choose which dialect to use.
The Ossie dialect should always be supported. Other dialects MAY be ignored. There is no guarantee that all different dialects for an expression will act the same, so implementations should be consistent with their dialect handling. This means that if an Ossie model has an expression written in two dialects, the implementation should deterministically choose which dialect to use.

### Declaring Dialect-Specific Expressions

Expand All @@ -650,11 +651,11 @@ expression:
| Current timestamp | `CURRENT_TIMESTAMP` | `CURRENT_TIMESTAMP()` | `CURRENT_TIMESTAMP()` | `CURRENT_TIMESTAMP()` | `CURRENT_TIMESTAMP` |
| Substring | `SUBSTRING(s, start, len)` | `SUBSTR(s, start, len)` | `SUBSTR(s, start, len)` | `SUBSTRING(s, start, len)` | `SUBSTRING(s, start, len)` |

###
###

### Dialect-Specific Extensions

Vendors may expose their own feature through extensions, however the default for Ossie should be to pass unknown values through.:
Vendors may expose their own feature through extensions, however the default for Ossie should be to pass unknown values through.:
---

## Cross-Reference: Tool Mappings
Expand Down Expand Up @@ -751,11 +752,11 @@ Implementations MAY support additional functions through dialect extensions. The

## References

- [SQL:2003 Standard](https://www.iso.org/standard/34132.html) (ISO/IEC 9075-2:2003)
- [Tableau Functions Reference](https://help.tableau.com/current/pro/desktop/en-us/functions.htm)
- [Looker Studio Function List](https://support.google.com/looker-studio/table/6379764)
- [DAX Function Reference](https://learn.microsoft.com/en-us/dax/dax-function-reference)
- [Snowflake SQL Reference](https://docs.snowflake.com/en/sql-reference-functions)
- [BigQuery Standard SQL Reference](https://cloud.google.com/bigquery/docs/reference/standard-sql/functions-and-operators)
- [Databricks SQL Functions](https://docs.databricks.com/sql/language-manual/sql-ref-functions.html)
- [SQL:2003 Standard](https://www.iso.org/standard/34132.html) (ISO/IEC 9075-2:2003)
- [Tableau Functions Reference](https://help.tableau.com/current/pro/desktop/en-us/functions.htm)
- [Looker Studio Function List](https://support.google.com/looker-studio/table/6379764)
- [DAX Function Reference](https://learn.microsoft.com/en-us/dax/dax-function-reference)
- [Snowflake SQL Reference](https://docs.snowflake.com/en/sql-reference-functions)
- [BigQuery Standard SQL Reference](https://cloud.google.com/bigquery/docs/reference/standard-sql/functions-and-operators)
- [Databricks SQL Functions](https://docs.databricks.com/sql/language-manual/sql-ref-functions.html)
- [PostgreSQL Functions](https://www.postgresql.org/docs/current/functions.html)
23 changes: 23 additions & 0 deletions mypy.ini
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# Shared strict mypy rules for this repo's Python packages.
#
# mypy has no `extends`/`include` directive for config files, and a
# package's own pyproject.toml can't declaratively "point at" another
# config file the way ruff's `extend` does -- mypy only ever reads one
# config file per run, chosen by `--config-file` or fixed discovery in the
# current working directory (it doesn't search upward like ruff/eslint).
#
# So each package invokes mypy with this file explicit, e.g. from
# python/ossie-sql/:
#
# uv run mypy --config-file ../../mypy.ini src tests
#
# Target paths are passed on the command line (not listed here as `files =`)
# so they're always resolved relative to the invoking package's own
# directory, regardless of where this shared file lives.
[mypy]
python_version = 3.11
strict = True

[mypy-sqlglot.*]
# sqlglot ships inline types but not a py.typed marker on all versions.
ignore_missing_imports = True
21 changes: 21 additions & 0 deletions python/ossie-sql/.pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
repos:
- repo: local
hooks:
- id: ossie-sql-ruff-check
name: ossie-sql ruff check
entry: bash -c 'cd python/ossie-sql && uv run ruff check --fix src tests'
language: system
files: ^python/ossie-sql/.*\.py$
pass_filenames: false
- id: ossie-sql-ruff-format
name: ossie-sql ruff format
entry: bash -c 'cd python/ossie-sql && uv run ruff format src tests'
language: system
files: ^python/ossie-sql/.*\.py$
pass_filenames: false
- id: ossie-sql-mypy
name: ossie-sql mypy (strict)
entry: bash -c 'cd python/ossie-sql && uv run mypy --config-file ../../mypy.ini src tests'
language: system
files: ^python/ossie-sql/.*\.py$
pass_filenames: false
13 changes: 13 additions & 0 deletions python/ossie-sql/Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
.PHONY: format lint typecheck test

format:
uv run ruff format src tests

lint:
uv run ruff check src tests

typecheck:
uv run mypy --config-file ../../mypy.ini src tests

test:
uv run pytest
57 changes: 57 additions & 0 deletions python/ossie-sql/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
# apache-ossie-sql

A [SQLGlot](https://github.com/tobymao/sqlglot) dialect implementing the
Ossie expression language defined in
[`core-spec/expression_language.md`](../../../core-spec/expression_language.md)
("Ossie_SQL_2026").

This package covers only the expression grammar: a custom SQLGlot `Dialect`
(tokenizer/parser/generator) so `sqlglot.parse_one(sql, read="ossie")` parses
and round-trips the spec's SQL subset (aggregate/window/date/string/math/
conditional functions, typed literals, `CASE`, `CAST`/`TRY_CAST`, etc.), plus
a validator that rejects the constructs the spec explicitly disallows
(`SELECT`/`FROM`/`JOIN`, `GROUP BY`, `WHERE`, subqueries, CTEs, set
operations, DDL/DML). Wiring the dialect into the Ossie YAML model (the
spec's "Changes to YAML" section) is out of scope here.

## Development

This package uses [`uv`](https://docs.astral.sh/uv/) for dependency
management.

```bash
uv sync

# Run the test suite
uv run pytest

# Format code (auto-fixes in place)
uv run ruff format src tests

# Check formatting without modifying files
uv run ruff format --check src tests

# Lint
uv run ruff check src tests

# Type-check (strict; rules come from the shared repo-root mypy.ini)
uv run mypy --config-file ../../mypy.ini src tests
```

Or via the `Makefile`:

```bash
make format
make lint
make typecheck
make test
```

### Enforcing formatting/lint/type-checking locally

Install the pre-commit hooks scoped to this package so `ruff format`,
`ruff check`, and `mypy` run automatically before each commit:

```bash
uv run pre-commit install -c python/ossie-sql/.pre-commit-config.yaml
```
31 changes: 31 additions & 0 deletions python/ossie-sql/pyproject.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[project]
name = "apache-ossie-sql"
version = "0.1.0"
description = "SQLGlot dialect for the Ossie expression language (core-spec/expression_language.md)"
license = { text = "Apache-2.0" }
readme = "README.md"
requires-python = ">=3.11"
dependencies = [
"sqlglot>=25.0",
]

[dependency-groups]
dev = [
"pytest>=8.0",
"mypy>=1.10",
"ruff>=0.11",
]

[tool.hatch.build.targets.wheel]
packages = ["src/ossie_sql"]

[tool.ruff]
extend = "../../ruff.toml"

[tool.pytest.ini_options]
testpaths = ["tests"]
pythonpath = ["src"]
Loading