Skip to content

feat: par_filter_map_collect#46

Open
claudioap wants to merge 1 commit intob-r-u:mainfrom
claudioap:par_filter_map
Open

feat: par_filter_map_collect#46
claudioap wants to merge 1 commit intob-r-u:mainfrom
claudioap:par_filter_map

Conversation

@claudioap
Copy link
Copy Markdown

I have a usecase that consists of bootstrapping a database with a subset of the planet. This subset is somewhat sparse and would not cause memory problems if it was to be eagerly allocated. I imagine other people to have similar requirements: basically being able to run a filter_map on the data.

One can do this in series with the supplied functions, however, not in parallel, which is a requirement for anyone who's doing frequent runs on big dumps.

I generalized my approach in this PR. Albeit more granular than what #42 is offering, this is slightly less elegant as it assumes that you want your blobs decoded and that you want to run through OsmData, which might not be true.

I could not get flatten() to let errors out and the workaround not a lazy iterator. This approach can easily fill someone's RAM if the filters happen to be too broad. Nonetheless, I've been reliably using it for a while now and just had the time to upstream it. Hope it serves someone else.

Cheers

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant