Skip to content

Flatten KoboToolbox matrix and repeat-group responses#256

Merged
rudokemper merged 7 commits into
rk/save-csv-to-diskfrom
248/flatten-kobotoolbox-matrix-v2
Jul 15, 2026
Merged

Flatten KoboToolbox matrix and repeat-group responses#256
rudokemper merged 7 commits into
rk/save-csv-to-diskfrom
248/flatten-kobotoolbox-matrix-v2

Conversation

@rudokemper

@rudokemper rudokemper commented Jul 1, 2026

Copy link
Copy Markdown
Member

Goal

Closes #248

Note to the reviewer: +340 of the line difference is just fixtures and conftest stuff. The actual code changes are +67 lines, and +90 are tests.

What I changed and why

  • Added flatten_kobotoolbox_submission() in the Kobo connector script and call it from transform_kobotoolbox_form_data() so repeat groups and matrix questions become wide {group}/{index}/{leaf} keys instead of one JSON cell. Existing reverse_properties_separated_by="/" then maps those to flat SQL columns. See README and docstrings for more info.

How I convinced myself this is right

  • In terms of the "why do this", by working with one of our highest tier partners who wanted to analyze matrix results and was not able to work with a JSON blob dumped in one cell. We had never encountered matrix questions in a Kobo questionnaire before this.
  • In terms of "how I know this works", besides tests, I tried it in production with real data. The result is as described in the README.

What I'm not doing here

  • Apply this to ODK (also uses XLSForm), but I did leave clear TODOs to track that future need if it arises.

LLM use disclosure

As I was working with a partner in country and didn't have a lot of time, I used Claude Opus 4.7 in planning mode to scheme this work out, and had it implement it too. Then I made various edits to ensure the code and its documentation is legible to me. I screened the code carefully to make sure it is valid and maintainable, making a few improvements here and there.

Comment thread f/connectors/kobotoolbox/kobotoolbox_responses.py Outdated
return None


def flatten_kobotoolbox_submission(submission):

@conservationtimothy conservationtimothy Jul 14, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible for Kobo submissions to contain deeply nested repeat groups (e.g. a repeat inside another repeat)? The current flattening logic only handles one level of slash-keyed rows, so I was wondering whether nested structures are something we need to account for, or whether Kobo's output shape guarantees this isn't needed.

Note: I saw the "if/when ODK flattening is required" TODO, so if this is more relevant to ODK rather than Kobo, feel free to ignore.

Not a blocker since I saw this working in practice, just curious about the expected payload shape.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great catch. Thanks for pointing this out. I tried it out with a real Kobo form and discovered that yes, you can have repeats inside of repeats, and we needed to implement recursive flattening to handle this. 980e25f

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this

@conservationtimothy conservationtimothy left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved but don't know if @IamJeffG might wanna give this the once over.

@rudokemper

Copy link
Copy Markdown
Member Author

Thanks. When you have time, please check out the PR for the downstream branch this merges info also.

@IamJeffG

Copy link
Copy Markdown
Contributor

No need for extra approval from me on this one! thanks.

@rudokemper rudokemper merged commit 3d91ed6 into rk/save-csv-to-disk Jul 15, 2026
1 check passed
@rudokemper rudokemper deleted the 248/flatten-kobotoolbox-matrix-v2 branch July 15, 2026 16:19
rudokemper added a commit that referenced this pull request Jul 15, 2026
* Serialize CSV cells when saving to disk

* Improve the CSV to Postgres script

* For the survey scripts, save to CSV

* Simplify docstring

* guard csv-write and db-ingest step behind an empty-data check

* isolate the pyODK token cache per test

* Flatten KoboToolbox matrix and repeat-group responses (#256)

* Start with fixtures and assertions

* Add flattening logic

* Leave TODO in ODK script

* Documentation

* Cleanup

* Improve readability of slash keyed rows fn

* Recursive flattening
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flatten KoboToolbox matrix and repeat-group responses instead of dumping them as JSON blobs

3 participants