Flatten KoboToolbox matrix and repeat-group responses#256
Conversation
| return None | ||
|
|
||
|
|
||
| def flatten_kobotoolbox_submission(submission): |
There was a problem hiding this comment.
Is it possible for Kobo submissions to contain deeply nested repeat groups (e.g. a repeat inside another repeat)? The current flattening logic only handles one level of slash-keyed rows, so I was wondering whether nested structures are something we need to account for, or whether Kobo's output shape guarantees this isn't needed.
Note: I saw the "if/when ODK flattening is required" TODO, so if this is more relevant to ODK rather than Kobo, feel free to ignore.
Not a blocker since I saw this working in practice, just curious about the expected payload shape.
There was a problem hiding this comment.
Great catch. Thanks for pointing this out. I tried it out with a real Kobo form and discovered that yes, you can have repeats inside of repeats, and we needed to implement recursive flattening to handle this. 980e25f
There was a problem hiding this comment.
Thanks for this
conservationtimothy
left a comment
There was a problem hiding this comment.
Approved but don't know if @IamJeffG might wanna give this the once over.
|
Thanks. When you have time, please check out the PR for the downstream branch this merges info also. |
|
No need for extra approval from me on this one! thanks. |
* Serialize CSV cells when saving to disk * Improve the CSV to Postgres script * For the survey scripts, save to CSV * Simplify docstring * guard csv-write and db-ingest step behind an empty-data check * isolate the pyODK token cache per test * Flatten KoboToolbox matrix and repeat-group responses (#256) * Start with fixtures and assertions * Add flattening logic * Leave TODO in ODK script * Documentation * Cleanup * Improve readability of slash keyed rows fn * Recursive flattening
Goal
Closes #248
Note to the reviewer: +340 of the line difference is just fixtures and
confteststuff. The actual code changes are +67 lines, and +90 are tests.What I changed and why
flatten_kobotoolbox_submission()in the Kobo connector script and call it fromtransform_kobotoolbox_form_data()so repeat groups and matrix questions become wide{group}/{index}/{leaf}keys instead of one JSON cell. Existingreverse_properties_separated_by="/"then maps those to flat SQL columns. See README and docstrings for more info.How I convinced myself this is right
What I'm not doing here
LLM use disclosure
As I was working with a partner in country and didn't have a lot of time, I used Claude Opus 4.7 in planning mode to scheme this work out, and had it implement it too. Then I made various edits to ensure the code and its documentation is legible to me. I screened the code carefully to make sure it is valid and maintainable, making a few improvements here and there.