feat: TM1 12.6.1 Arrow/Parquet/Flight datasources + write_dataframe(use_parquet/use_arrow)

[skido-cw]

Summary

Two related changes for the new TM1 v12, build 12.6.1 columnar data sources (Apache Arrow IPC, Parquet, Arrow Flight):

1. Model the new TI datasource types on Process — author/read processes whose DataSource.Type is ARROW / PARQUET / FLIGHT (previously these emitted an empty "DataSource": {}).
2. cells.write_dataframe(..., use_parquet=True | use_arrow=True) — a pure-Python columnar blob-write path: encode the DataFrame to a Parquet file / Arrow-IPC stream and load it through an ARROW/PARQUET TI datasource instead of the CSV/ASCII blob.

Both commits are verified end-to-end against a live PA 12.6.1 server.

1 — Process datasource modelling

• Process gains four datasource_flight_* scalars (constructor, from_dict, properties) and two _construct_body_as_dict branches:
  • ARROW/PARQUET: file name or http(s) URL via dataSourceNameForServer/Client, plus optional jsonRootPointer/jsonVariableMapping for nested columns; no ascii* keys.
  • FLIGHT: flightLocation / flightDescriptorType / flightDescriptor / flightAuth.
• Branches match Type case-insensitively because the server canonicalizes it to title-case on read-back (Arrow/Parquet/Flight) — otherwise from_dict(get(...)) would round-trip to an empty DataSource.
• ProcessService.get/get_all select the four flight fields.
• datasource_type stays a free string (no Type whitelist) and no version gating is added — ProcessDataSource is an OData open type validated by the engine at execute time.

Field names and the title-case canonicalization were verified against the live 12.6.1 server (the flight* names persist and round-trip; the dataSourceFlight* names are silently dropped).

2 — Columnar write_dataframe

• New keyword params use_arrow / use_parquet / parquet_compression on cells.write_dataframe.
• _encode_dataframe_columnar (pyarrow): DataFrame → Arrow-IPC stream or Parquet file; fields v1..vN (coordinates, String) + vValue (numeric column → float64 → a Numeric TI variable, no client-side CSV serialization/escaping).
• _build_columnar_blob_to_cube_process mirrors _build_blob_to_cube_process but targets the columnar datasource; _write_dataframe_through_columnar_blob uploads via FileService, runs the unbound TI, and deletes the blob. Gated to server ≥ 12.6.1.
• pyarrow is an optional extra (pip install TM1py[arrow]); a guarded import raises a clear error if use_arrow/use_parquet is used without it.

Precision note (measured live)

On the live 12.6.1 server the columnar numeric path is not bit-exact for values at the edge of double precision — read-back rounds to ~16 significant digits, whereas the ASCII/CSV path round-trips exactly. The Parquet file itself preserves the exact float64 (covered by a unit test), so the rounding is server-side in TM1's columnar ingestion. Docstrings state precision is ~float64, not guaranteed bit-exact. The benefits here are the columnar datasource path, no client-side CSV building, and Parquet upload compression — not added precision.

Testing

• Offline unit tests (no server): Process body shapes + from_dict round-trips incl. title-case Type (Tests/Process_test.py); the columnar encoder and generated TI process body (Tests/CellService_columnar_test.py). 28 tests, all pass.
• Live (TM1 12.6.1): the optional ProcessService round-trip test (gated >= 12.6.1); and the write_dataframe(use_parquet/use_arrow) path validated via a self-cleaning probe.
• black --check . and ruff check . pass.

Backward compatibility

No changes to existing datasource branches (ASCII/None/ODBC/TM1CubeView/TM1DimensionSubset/JSON) or to the default write_dataframe path; the new behavior is opt-in.

🤖 Generated with Claude Code