perf: Improve record and schema flattening performance

[edgarrmondragon]

• perf: Module-level regex compiled pattern
• perf: Module-level frozenset({"null","object","array"})
• perf: Avoid unnecessary key and schema copies
• perf: Construct flattened field key incrementally as a string instead of a list not worth it

Summary by Sourcery

Improve performance of key flattening and JSON dumping by hoisting reusable constructs to module level.

Enhancements:

• Use a precompiled module-level lowercase regex for key reduction during flattening.
• Use a shared module-level frozenset for null/object/array type comparisons in JSON dumping logic.

Summary by Sourcery

Optimize schema and record flattening to reduce overhead and improve performance.

Enhancements:

• Hoist the lowercase regex pattern and null/object/array type set to module-level constants for reuse.
• Flatten schema and records using an incremental key-prefix string rather than repeatedly rebuilding key lists, while preserving duplicate-key detection semantics.
• Avoid unnecessary deep copies when flattening schemas by building the output schema from the input node directly.
• Replace sort-and-group duplicate key detection with an O(n) set-based check for flattened schema items.

Tests:

• Update snapshot JSONL outputs for mapped and flattened streams to reflect the revised flattening behavior.

[sourcery-ai]

Reviewer's Guide

Optimizes schema and record flattening performance by hoisting reusable constructs to module scope, switching from list-based to string-based key prefix handling with a fallback path for long keys, avoiding in-place schema mutations, and replacing sort-and-group duplicate detection with linear set-based checks while preserving behavior and snapshots.

Sequence diagram for record flattening with parent_prefix and JSON dump check

sequenceDiagram
    participant Caller
    participant flatten_record
    participant _flatten_record
    participant flatten_key
    participant _should_jsondump_value

    Caller->>flatten_record: flatten_record(record, flattened_schema, separator, max_level, max_key_length)
    flatten_record->>_flatten_record: _flatten_record(record_node=record, flattened_schema=flattened_schema, parent_prefix="", separator, level=0, max_level, max_key_length)
    loop for each k, v in record_node.items()
        _flatten_record->>_flatten_record: Compute new_key = parent_prefix + k
        _flatten_record->>_flatten_record: [check len(new_key) >= max_key_length]
        alt long_key
            _flatten_record->>_flatten_record: Reconstruct parent_keys from parent_prefix
            _flatten_record->>flatten_key: flatten_key(k, parent_keys, separator, max_key_length)
            flatten_key-->>_flatten_record: abbreviated key
        end
        _flatten_record->>_flatten_record: [decide recurse vs. append based on type and flattened_schema]
        _flatten_record->>_flatten_record: [build child parent_prefix = parent_prefix + k + separator on recursion]
    end
    _flatten_record-->>Caller: flattened_record
    Caller->>_should_jsondump_value: _should_jsondump_value(key, value, flattened_schema)
    _should_jsondump_value->>_should_jsondump_value: Check frozenset(flattened_schema[key]["type"]) == _NULL_OBJECT_ARRAY_TYPES
    _should_jsondump_value-->>Caller: bool

Loading

Flow diagram for incremental key flattening with fallback and duplicate detection

flowchart TD
    A["_flatten_schema start"] --> B["Iterate schema_node.properties"]
    B --> C["Compute new_key = parent_prefix + field_name"]
    C --> D{"len(new_key) >= max_key_length"}
    D -- "no" --> E["Use new_key as-is"]
    D -- "yes" --> F["Reconstruct parent_keys from parent_prefix"]
    F --> G["Call flatten_key(field_name, parent_keys, separator, max_key_length)"]
    G --> H["Set new_key from flatten_key result"]
    E --> I{"field_schema.type indicates nested object and level < max_level"}
    H --> I
    I -- "yes" --> J["Compute next_prefix = parent_prefix + field_name + separator"]
    J --> K["Recurse _flatten_schema(field_schema, parent_prefix=next_prefix, ...)"]
    I -- "no" --> L{"first_element.type in composite"}
    L -- "string" --> M["Append (new_key, {**first_element, type=['null', 'string']})"]
    L -- "array" --> N["Append (new_key, {**first_element, type=['null', 'array']})"]
    L -- "object" --> O["Append (new_key, {**first_element, type=['null', 'object']})"]
    L -- "other/typeless" --> P["Append (new_key, {type=['null', 'string']})"]
    P --> Q["After loop: check duplicates with seen set"]
    M --> Q
    N --> Q
    O --> Q
    Q --> R{"key in seen"}
    R -- "yes" --> S["Raise ValueError(duplicate column name)"]
    R -- "no" --> T["Add key to seen"]
    T --> U["Return dict(items)"]

Loading

File-Level Changes

Change	Details	Files
Hoist reusable constructs to module scope for reuse across flatten operations.	Introduce a precompiled module-level lowercase regex used by flatten_key for key reduction. Add a module-level frozenset for null/object/array type comparisons in JSON dump decision logic.	singer_sdk/helpers/_flattening.py
Refactor schema flattening to use string prefixes instead of parent key lists and avoid deep copies and in-place mutations.	Change flatten_schema to build a shallow schema copy excluding properties and pass the original schema plus an empty parent_prefix into _flatten_schema. Rewrite _flatten_schema to accept parent_prefix strings instead of parent_keys lists, constructing new_key via prefix concatenation. Add a slow-path fallback in _flatten_schema that reconstructs parent key lists from parent_prefix and invokes flatten_key when max_key_length is exceeded. Adjust recursive calls to _flatten_schema to build the next_prefix using the original field_name so nested behavior matches the previous implementation. Replace nullable type handling to create new field schema dicts with updated type lists instead of mutating the first composite element in-place. Replace sort-and-group duplicate key detection with an O(n) seen-set check over generated items before returning the dict.	singer_sdk/helpers/_flattening.py
Refactor record flattening to mirror the schema changes using string prefixes, while preserving key-abbreviation behavior and JSON-dump decisions.	Update flatten_record to call _flatten_record with parent_prefix string, separator, and initial level instead of parent_key list. Rewrite _flatten_record to derive new_key via parent_prefix concatenation and only reconstruct parent key lists plus call flatten_key when max_key_length is exceeded. Pass parent_prefix + key + separator into recursive _flatten_record calls using the original key so nested prefixes reconstruct as in the old implementation. Update _should_jsondump_value to compare flattened schema types using the module-level frozenset constant instead of constructing a new set each time.	singer_sdk/helpers/_flattening.py
Regenerate flattening-related mapped stream snapshots to validate behavior remains unchanged under the optimized implementation.	Update mapped_stream flatten_all snapshot JSONL outputs. Update mapped_stream flatten_all_with_dot_separator snapshot JSONL outputs. Update mapped_stream flatten_depth_1 snapshot JSONL outputs. Update mapped_stream map_and_flatten snapshot JSONL outputs.	tests/snapshots/mapped_stream/flatten_all.jsonl tests/snapshots/mapped_stream/flatten_all_with_dot_separator.jsonl tests/snapshots/mapped_stream/flatten_depth_1.jsonl tests/snapshots/mapped_stream/map_and_flatten.jsonl

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.21%. Comparing base (24237d2) to head (3b25ec7).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3687      +/-   ##
==========================================
- Coverage   94.21%   94.21%   -0.01%     
==========================================
  Files          73       73              
  Lines        6208     6203       -5     
  Branches      763      761       -2     
==========================================
- Hits         5849     5844       -5     
  Misses        267      267              
  Partials       92       92              

Flag	Coverage Δ
core	83.04% <100.00%> (-0.02%)	⬇️
end-to-end	76.07% <12.50%> (+0.04%)	⬆️
optional-components	44.88% <12.50%> (+0.02%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[codspeed-hq]

Merging this PR will improve performance by ×2.5

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 4 improved benchmarks
✅ 10 untouched benchmarks

Performance Changes

	Benchmark	BASE	HEAD	Efficiency
⚡	test_bench_flatten_schema_deep	4,472.9 µs	775.8 µs	×5.8
⚡	test_bench_flatten_schema_wide	1,031.5 µs	289.5 µs	×3.6
⚡	test_bench_flatten_key_short	35.5 µs	25.1 µs	+41.4%
⚡	test_bench_flatten_record_wide	357.5 µs	263 µs	+35.95%

Tip

Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing perf/flattening (3b25ec7) with main (24237d2)}

[edgarrmondragon]

@sourcery-ai review

[sourcery-ai]

Hey - I've left some high level feedback:

• The change from sorting items to returning them in insertion order in _flatten_schema may affect downstream consumers relying on deterministic sorted key order; consider either preserving sort or explicitly documenting this behavioral change.
• The new parent_prefix approach assumes the prefix always ends with the separator when non-empty; it may be worth adding a small invariant check or helper to construct/validate prefixes to avoid subtle bugs if future changes accidentally omit the trailing separator.

Prompt for AI Agents

Please address the comments from this code review:

## Overall Comments
- The change from sorting items to returning them in insertion order in `_flatten_schema` may affect downstream consumers relying on deterministic sorted key order; consider either preserving sort or explicitly documenting this behavioral change.
- The new `parent_prefix` approach assumes the prefix always ends with the separator when non-empty; it may be worth adding a small invariant check or helper to construct/validate prefixes to avoid subtle bugs if future changes accidentally omit the trailing separator.

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

_{Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.}

[sourcery-ai]

Hey - I've found 1 issue, and left some high level feedback:

• The "rare slow path" logic for reconstructing parent keys from parent_prefix is duplicated between _flatten_schema and _flatten_record; consider extracting this into a small helper to keep the separator handling and edge cases centralized.
• In _flatten_schema, sep_len = len(separator) is computed for every call even though it is invariant within the function; you can move this out of the loop or compute it lazily only when the slow path is taken to avoid unnecessary work.

Prompt for AI Agents

Please address the comments from this code review:

## Overall Comments
- The "rare slow path" logic for reconstructing parent keys from `parent_prefix` is duplicated between `_flatten_schema` and `_flatten_record`; consider extracting this into a small helper to keep the separator handling and edge cases centralized.
- In `_flatten_schema`, `sep_len = len(separator)` is computed for every call even though it is invariant within the function; you can move this out of the loop or compute it lazily only when the slow path is taken to avoid unnecessary work.

## Individual Comments

### Comment 1
<location path="singer_sdk/helpers/_flattening.py" line_range="360" />
<code_context>
     if "properties" not in schema_node:
         return {}

+    sep_len = len(separator)
     for field_name, field_schema in schema_node["properties"].items():
-        new_key = flatten_key(
</code_context>
<issue_to_address>
**issue (bug_risk):** Reconstructing parent keys breaks when separator is an empty string.

This optimization assumes a non-empty `separator`. When `separator == ""`, `sep_len` is 0, `parent_prefix[:-sep_len]` is unchanged, and calling `.split("")` raises `ValueError`. The previous `parent_keys` approach worked regardless of separator length. Please either explicitly handle empty separators (e.g., reject them or skip this optimization) or adjust the reconstruction logic so it safely supports an empty separator.
</issue_to_address>

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

_{Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.}

[read-the-docs-community]

Documentation build overview

📚 Meltano SDK | 🛠️ Build #33380695 | 📁 Comparing 3b25ec7 against latest (8845045)

  🔍 Preview build  

1 file changed

± stream_maps.html