[WIP] Add an interface for converting ODOP runs into benchmark records#90
Draft
martymichal wants to merge 7 commits intomainfrom
Draft
[WIP] Add an interface for converting ODOP runs into benchmark records#90martymichal wants to merge 7 commits intomainfrom
martymichal wants to merge 7 commits intomainfrom
Conversation
ODOP keeps track of individual runs. The idea is that after the end of a run the final results can be saved aside, compressed and also annotated if needed for benchmarking or archival purposes. A benchmark record should hold the following information: - the name of the ODOP run - the used ODOP con fig - metrics captured during the run using OdopObs - total execution time - runtime environment This adds only the basic class with initial loading logic. The conversion of ODOP runs into benchmark records will be added in a follow-up commit.
During a run ODOP collects a number of metrics used not just for scheduling. They are vital when wanting to benchmark the effect of ODOP on an application. Currently OdobObs uses Tinyflux[0] library for storing time-series metrics. These are stored per-node in a multi-node set-up. For more convenient post-processing, it seems better to have all the metrics from all nodes in the same time series. This is merge is done using a heap queue. In case the amount of metrics is already quite high, the recommendation for tinyflux is to split a database into multiple files to keep the query speeds reasonable[1]. [0] https://github.com/citrusvanilla/tinyflux [1] https://tinyflux.readthedocs.io/en/latest/tips.html#dealing-with-growing-datasets
martymichal
added
the
benchmarking
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Initial work on adding a benchmarking interface for tracking ODOP runs for benchmarking purposes. See commits for more details on the work.
Includes #89