Refactor store-backing calculator function#708
Draft
yousefmoazzam wants to merge 5 commits intomainfrom
Draft
Conversation
The padded shape of the next section's input chunk was being used, which is incorrect from the perspective of the order of operations that occur when setting up a section's source and sink in `_setup_source_sink()` in the task runner. Instead, it should be the padded shape of the current section's input chunk.
…culation This is purely for the sake of making the order of the chunk size calculations in `determine_store_backing()` match the order of the allocations that would occur during the setup + processing of a section.
For the last section: - the input chunk's backing (RAM or hdf5 file) is purely determined by the output of the previous section - the output chunk is not written anywhere (the dummy sink simply discards the blocks given to it) Furthermore, the dummy sink doesn't take a type of backing as a parameter (and if it did, it's unclear what it could do with that information, given that the dummy sink simply discards the input data). Therefore, there's no need to have any calculation of the backing of the store for the last section's writing.
Due to the previous commit, there is now only one dataset store backing calculator function that needs a reduction operation performed, so there is no need to have a generic way to produce a decorator that can perform the reduction on the result of different functions.
yousefmoazzam
force-pushed
the
refactor-determine-store-backing
branch
from
f5a9001 to
ac1e96e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The main changes are:
determine_store_backing()functionChecklist
user-release-notelabel in order to include this PR in the "NotableChanges for Users" section in release notes