feat(caches): Add PyIceberg-based cache for Apache Iceberg format#970

Open

Aaron ("AJ") Steers (aaronsteers) wants to merge 23 commits into

mainfrom

devin/1769653838-iceberg-cache

Commits on Jan 29, 2026

feat(caches): Add PyIceberg-based cache for Apache Iceberg format

devin-ai-integration[bot]
and
Aaron ("AJ") Steers (aaronsteers)
committed
fix: Address CodeRabbit feedback for IcebergCache

devin-ai-integration[bot]
and
Aaron ("AJ") Steers (aaronsteers)
committed
fix: Make _drop_temp_table a no-op to prevent data loss

devin-ai-integration[bot]
and
Aaron ("AJ") Steers (aaronsteers)
committed
fix: Set type_converter_class on IcebergProcessor

devin-ai-integration[bot]
and
Aaron ("AJ") Steers (aaronsteers)
committed
feat: Add example script for IcebergCache with nested records

devin-ai-integration[bot]
and
Aaron ("AJ") Steers (aaronsteers)
committed
test: Update test to reflect schema_name='main' for SQLite compatibility

devin-ai-integration[bot]
and
Aaron ("AJ") Steers (aaronsteers)
committed
fix: Add get_string_type and get_json_type methods to IcebergTypeConverter

devin-ai-integration[bot]
and
Aaron ("AJ") Steers (aaronsteers)
committed
fix: Fix loop counter variable shadowing in example script

devin-ai-integration[bot]
and
Aaron ("AJ") Steers (aaronsteers)
committed
feat(caches): Add sophisticated object typing configuration for IcebergCache

devin-ai-integration[bot]
and
Aaron ("AJ") Steers (aaronsteers)
committed
fix: Use consistent field ID counter to avoid collisions in Iceberg schemas

devin-ai-integration[bot]
and
Aaron ("AJ") Steers (aaronsteers)
committed
fix: Use as_json_strings mode in example script for compatibility

devin-ai-integration[bot]
and
Aaron ("AJ") Steers (aaronsteers)
committed
fix: Serialize schemaless arrays/objects in nested_types mode

devin-ai-integration[bot]
and
Aaron ("AJ") Steers (aaronsteers)
committed
style: Fix ruff formatting

devin-ai-integration[bot]
and
Aaron ("AJ") Steers (aaronsteers)
committed
fix: Add timestamp conversion and improve recursive data converter

devin-ai-integration[bot]
and
Aaron ("AJ") Steers (aaronsteers)
committed
fix: Convert IcebergTypeConverter field ID counter to instance variable

devin-ai-integration[bot]
and
Aaron ("AJ") Steers (aaronsteers)
committed
fix: Handle string boolean values correctly in convert_value_to_arrow_type

devin-ai-integration[bot]
and
Aaron ("AJ") Steers (aaronsteers)
committed
fix: Remove unreachable None check in boolean handling

devin-ai-integration[bot]
and
Aaron ("AJ") Steers (aaronsteers)
committed
fix: Reset field ID counter at start of schema generation

devin-ai-integration[bot]
and
Aaron ("AJ") Steers (aaronsteers)
committed
docs: Add experimental notice to Iceberg cache module and class docstrings

devin-ai-integration[bot]
and
Aaron ("AJ") Steers (aaronsteers)
committed
docs: Add type handling configuration section to public module docstring

devin-ai-integration[bot]
and
Aaron ("AJ") Steers (aaronsteers)
committed

Commits on Jan 30, 2026

tidying
Aaron ("AJ") Steers (aaronsteers)
authored

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(caches): Add PyIceberg-based cache for Apache Iceberg format#970

feat(caches): Add PyIceberg-based cache for Apache Iceberg format#970
Aaron ("AJ") Steers (aaronsteers) wants to merge 23 commits into
mainfrom
devin/1769653838-iceberg-cache

Commits on Jan 29, 2026

feat(caches): Add PyIceberg-based cache for Apache Iceberg format

fix: Address CodeRabbit feedback for IcebergCache

fix: Make _drop_temp_table a no-op to prevent data loss

fix: Set type_converter_class on IcebergProcessor

feat: Add example script for IcebergCache with nested records

test: Update test to reflect schema_name='main' for SQLite compatibility

fix: Add get_string_type and get_json_type methods to IcebergTypeConverter

fix: Fix loop counter variable shadowing in example script

feat(caches): Add sophisticated object typing configuration for IcebergCache

fix: Use consistent field ID counter to avoid collisions in Iceberg schemas

fix: Use as_json_strings mode in example script for compatibility

fix: Serialize schemaless arrays/objects in nested_types mode

style: Fix ruff formatting

fix: Add timestamp conversion and improve recursive data converter

fix: Convert IcebergTypeConverter field ID counter to instance variable

fix: Handle string boolean values correctly in convert_value_to_arrow_type

fix: Remove unreachable None check in boolean handling

fix: Reset field ID counter at start of schema generation

docs: Add experimental notice to Iceberg cache module and class docstrings

docs: Add type handling configuration section to public module docstring

Commits on Jan 30, 2026

tidying

Commits on Jan 31, 2026

feat: Implement merge-on-read using PyIceberg upsert for incremental syncs

fix: Create directories for string local paths in _get_warehouse_path