Proposed redesign of the output structure provided to users.#3111
Proposed redesign of the output structure provided to users.#3111morrowcj wants to merge 5 commits into
Conversation
|
Comments from @KFosterReed (June 5 2026) and my responses:
This functionality is preserved in the structure I've envisioned, but I'm also interested in hearing the dev team's thoughts.
I was unaware of that. We can change the document to indicate a mechanical change to this feature rather than an entirely new feature. |
|
Current Coverage: 99% Mypy errors on tidy_out_design branch: 1134 |
|
🚨 Please update the changelog. This PR cannot be merged until |
|
Current Coverage: % Mypy errors on tidy_out_design branch: 1134 |
|
🚨 Please update the changelog. This PR cannot be merged until |
ew3361zh
left a comment
ew3361zh
left a comment
There was a problem hiding this comment.
Hey Clay, this is really interesting and I think there are a lot of things that make sense as far as making the data more palatable and usable for scientists/researchers/other collaborators. I agree it makes sense to think about restructuring it and I like the principals of tidy data being applied here.
I think I'm at the stage where a demonstration would be helpful - I realize you added a lot of code examples but I think it would be helpful to get on the same page through a presentation if you're up for it. Maybe just in the dev team to start.
Most of my questions center around:
- definitions: what is an observation? a domain? etc.
- how it compares efficiency wise with both building and outputting a full RuFaS simulation's worth of data? how efficient would the data manipulation tools be and how difficult might it be to integrate this new system into our existing graphing and reporting frameworks? Do those still have a role in post processing?
- do we need to change variable names at all to ensure distinct entries?
- what about more complex pieces of data dictionaries and nested dictionaries of data currently being added to OM? would we need to flatten the data more universally as it's being sent to OM?
- do we get rid of the exclude info maps flag? or is there some accommodation for not having this contextual data?
Thank you for writing this up - lots of great ideas but lots to digest and think through and I figured the best thing to do at this point was to get the conversation moving.
| RuFaS should provide tabular data to users that conforms to the "tidy data" paradigm, with consistent structure: | ||
|
|
||
| - rows represent individual observations (a specific domain at a specific time). Exactly what constitutes an RuFaS | ||
| observation is debatable, but it should *not* represent the entire farm at a moment in time. |
There was a problem hiding this comment.
Yea I feel like this definition is important to defining the requirement. Maybe it would be helpful here to offer some examples of different sides of the debate you've been weighing for what we might consider constituting a RuFaS observation.
| - Columns represent individual variables (attributes of the observation). This includes grouping information like which | ||
| field or pen an observation was made from. | ||
|
|
||
| - Column labels should also not contain information about the variable like code address (module name, caller function, |
There was a problem hiding this comment.
How does this work for variables that are the same but across different parts of the model - nitrogen for example is observed in manure in multiple forms, in the crop and soil module in a bunch of forms, in the animal module in multiple forms. If we're currently distinguishing these by their contexts, how do we keep them distinct without that context? Or is the context derived from the information in the column?
There was a problem hiding this comment.
If measured on the same observational unit, names need to be different (organic_nitrogen, inorganic_nitrogen, etc.). If not, then the grouping variables (like domain) differentiate them.
| [Pandas Integration](#pandas-integration)). Measurements recorded outside of the main simulation should use | ||
| negative time steps if they are time series (i.e., the burn-in period during herd initialization) or an explicit | ||
| missing value (`numpy.nan`). |
There was a problem hiding this comment.
I feel like this could get confusing and I'm not sure makes sense to try and shoehorn simulation day into it. Maybe there's a different way to denote measurements recorded outside the main simulation.
There was a problem hiding this comment.
Yeah, I think you're right. There's probably a better way to do this. My thought was that, for herd initialization, those data are time series that occur prior to the simulation, so would have measurements from days before day 0. I still think those values should have negative time, indicating how many days before the simulation start they were measured.
| - Implement a check when a measurement is recorded to ensure that it isn't going to overwrite an existing record. | ||
| There can only ever be one measurement per $address \times group \times time$, and if there is a conflict, than | ||
| another grouping variable -- which collectively define the observation -- needs to be added for that measurement. |
There was a problem hiding this comment.
I think depending on the form, this seems like it could potentially be pretty expensive. I'm still trying to envision what a full-simulation version of the current outputs would look like under this new format but in my mind it seems a lot larger than what we currently create with the output pool.
There was a problem hiding this comment.
I don't think it needs to be much larger, per my example. but I might be missing something. Definitely worth testing/discussing.
| function mimics how RuFaS methods record measurements internally. Note that some of the variables aren't observed at | ||
| every time step. | ||
|
|
||
| 4. `frame_pool` constructs a data frame from each element in the data pool. |
There was a problem hiding this comment.
So is the idea that we would still build and maintain an output pool either in a similar fashion to what we're currently doing or in an updated version but then build the dataframe after the simulation is over? Or is the dataframe being built as the simulation is happening?
There was a problem hiding this comment.
The first one! Store data basically the same as we do now, and convert it to a data frame when serving it to users (or when performing operations that data frames are well-suited for like aggregation).
| return tidied | ||
| ``` | ||
|
|
||
| To demonstrate how it works, we will use the following `cases` (a list of info maps), displayed as a data frame table |
There was a problem hiding this comment.
I wonder if you might be able to do this in a presentation to the dev team? I think that would be helpful to see and make sure we're all on the same page.
| It should be noted that while the tidy data frame objects are much smaller for this case, the relative performance | ||
| also depends upon the number of columns in the data (which are less efficient than rows). Additionally, some | ||
| operations require internally copying objects. During those operations, memory usage may be 2x (or more) the object | ||
| size. |
There was a problem hiding this comment.
Can you say more about this? I think one big concern I have overall is ensuring the efficiency of a new data storage and output system and tying those to the core RuFaS simulation. The memory usage seems potentially concerning.
There was a problem hiding this comment.
Yes, I'm not sure about how this works at scale, TBH. But I do think that there are ways to mitigate any memory problems, like only loading the parts of the data that you're manipulating into memory - much like how RuFaS already supports "chunkification".
|
@ew3361zh, this is excellent feedback thank you for being so thorough in your consideration. First I'll say, that I was planning to give a short presentation about tidy output data at our next all team meeting, but I may need to tweak the specifics of that presentation based on your feedback. Below are some of my initial responses to your main points:
An observation is a discrete unit of something for which you are gathering data. This unit differs with the type of data: If you are measuring traits of individual cows through time, the observation is a single cow at a single time point. If you are measuring properties about pens that house the cows, the observation is a single pen at a single time point. "Domain" is less concrete, it was just my suggestion for grouping data by kind and indicating the observational unit. For example, individual cow weight and food consumption are both measurements taken for the same observational unit and are similar in kind, so we might chose to group them as "cow body condition" or even more broadly "cow measurements". By contrast, pen size and bedding material may only be measured once and don't pertain to individual cows, so could be grouped into a "pen info" domain. We'd need to decide what was most practical, logical, and/or efficient.
I'm not 100% sure. I know that storage is very efficient and I think that transformations are optimized to be as efficient as possible, but we'd probably need to do some testing. The example that I included shows that building the data could be incredibly similar to how we currently do it. Since these tools are so commonly used, I am pretty confident that there are solutions to any of the problems that we'd run into.
I don't think so. Even if two variables have identical names, they can be distinguished by their grouping variables. This is a key reason I've suggested adding a "domain". For example, we might have a variable called "nitrogen" that tracks the N amount in each soil layer and another called "nitrogen" that tracks the amount of N in manure. The former would belong to the "soil nutrients" domain (and would also have "field id", "soil layer", etc. grouping variables) while the latter would be in the "manure composition" domain.
I think we'd want to flatten them.
My inclination is yes, but I don't know enough about this use case. The way I envision it, the info map data would be quite small in memory, since it never gets duplicated. So we'd have 1 (piece of each) info map stored for each variable class that gets added to the pool. The primary purpose of the info map is to tell the OM which variable to append a measurement to. For example our output pool might look like the following, where the key for each variable is a tuple containing the info map information (of the form {
"pen info": { # first domain
("Animal", "_collect_pen_info" , "pen_size"): x # first variable,
},
"cow measurements": { # second domain
("Animal", "_measure_body_condition", "body_weight"): y, # second variable
("Animal", "_record_feed_consumed", "feed_consumed"): z # third variable
}
} |
2 participants
The sole purpose of this PR is to provide an avenue for developers to assess and comment on the proposed design doc (tidy-rufas-output_design-doc.qmd and tidy-rufas-output_design-doc.html). It is not intended to be merged as-is.
Context and reviewer instructions
For reproducibility, I have included all files necessary to recreate my output in the
tidy_output_designfolder, but only the following files are important for users to review:The main formatted design document is tidy-rufas-output_design-doc.html. This is the main document that reviewers should read, but comments should be added as github comments to the source file tidy-rufasoutput_design-doc.qmd.
A secondary (optional) document, based on feedback from @YijingGong, outlines a solution for organizing data collected at different levels. The formatted version of this document is animal-domain-data-example.html and the source file where comments should be added is animal-domain-data-example.qmd.
All remaining files were used to generate the html outputs. The ideal way to view the documents is to download the html files and open them in a browser. Comments can be left on the source qmd as part of a review (from the "Files Changed" tab of this PR).
Relevant to #2487