Sped up the creation and filling of datasets#30
Open
clayton-ho wants to merge 3 commits intoHaeffnerLab:masterfrom
Open
Sped up the creation and filling of datasets#30clayton-ho wants to merge 3 commits intoHaeffnerLab:masterfrom
clayton-ho wants to merge 3 commits intoHaeffnerLab:masterfrom
Conversation
Author
|
I realized that I was stupid and forgot to account for live datasets, in which case the Dataset object will not be able to accommodate the increasing dataset. |
Author
These have been fixed in the latest commit, which somewhat speeds up getData since we now create the data arrays upon instantiation of a Dataset object so we never have to check for the existence of data arrays. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Sped up the creation and filling of datasets (in Dataset.py), making the RSG over an order of magnitude faster for large datasets.
This requires the datavault server to have a setting "shape" which returns the shape of the dataset.
Instead of initializing an empty numpy array and appending successive data grabs from the datavault, we now use the datavault's shape setting to preallocate a numpy array and fill it with data.
Additionally, the size of each data grab has been increased from 100 to 1000 since most of the overhead arises from filling the dataset.