Request: first-class representation of file-backed datasets (e.g. Parquet) in Dataset.source (or adjacent fields)

[jonbjo]

Summary
Today the OSI core spec describes Dataset.source as a reference to an underlying physical table/view (e.g. database.schema.table) or query. That model fits warehouse-centric semantics well but does not describe file- or object-store-backed relations (Parquet, directories/globs, file: / s3: / gs: URIs, etc.), which are common in analytical and lakehouse workflows.

This issue asks whether the working group has already discussed file-backed / columnar-file sources, and if so, what direction was considered. If not, we would like to explore first-class (core or well-documented optional profile) support so documents are portable without overloading source or relying only on opaque custom_extensions.

Motivation / use cases

• Local development and tests: point a semantic dataset at Parquet on disk without a catalog.
• Object storage: datasets backed by s3://, gs://, or HTTPS without a SQL warehouse name.
• Interoperability: engines that compile OSI to Substrait / SQL often need a stable logical name for the dataset while physical scan targets are resolved separately; the spec could make that split explicit.
• Consistency: avoids each implementation inventing incompatible conventions inside COMMON custom extensions.

Current spec (as we read it)

• Core source is a single string, with examples in database.schema.table form.
• Extensibility exists via custom_extensions / Vendor, which works but does not give a shared, validated shape for “this dataset is Parquet at these locations.”

Questions for the working group

1. Has file-backed / columnar-file (especially Parquet) been discussed? If yes, pointers to notes or prior issues would be appreciated.
2. Is the intent that source remain strictly warehouse-style, with all file semantics in extensions, or is there appetite to extend the core model?
3. Would the WG prefer:
   A) Extending Dataset with optional structured fields (e.g. physical / storage with kind, locations[], format), while keeping source as the logical relation id for queries; or
   B) A documented optional profile (separate schema or appendix) for “file datasets” without changing minimal core; or
   C) Strengthening guidance + JSON schema for a specific COMMON (or new vendor) extension payload so all implementations converge.

Sketch of a possible shape (for discussion only)
Not a proposal to adopt verbatim, only to illustrate the gap:

• name: logical dataset id (unchanged).
• source: stable logical relation name used in generated SQL/Substrait (e.g. qualified name), or explicit split between “logical id” and “physical” if the WG wants that separation.
• Optional storage descriptor: e.g. format parquet, one or more URIs (files, globs, prefixes), optional partition / read hints.

Happy to help refine this with implementer feedback (e.g. from tools that emit OSI and target Arrow/DataFusion/Iceberg).