docs: 📝 add design docs for the request file#65
Conversation
lwjohnst86
left a comment
There was a problem hiding this comment.
Great start 😁 here are some initial changes.
| ### Non-goals | ||
|
|
||
| - Propagate is not intended to be a general-purpose query engine. The initial | ||
| request format should not support joins, grouping, arbitrary functions, random | ||
| sampling, or row selection by position. | ||
| - We also don't support shorthands such as `*` since that would make it less | ||
| explicit which columns are used from just looking at the request file without | ||
| also seeing the data package metadata. |
There was a problem hiding this comment.
These should be included at the location where they are relevant. Design docs are about the what and the why, and these are why things, which should be treated as important as the rest of the text.
| <!-- TODO What does an empty list of columns and rows mean? --> | ||
| <!-- We discussed elsewhere that one purpose of this file is for someone to get an overview of which data is used in a specific study. On the one side, that could benefit from having to specify all the columns if the study used all of them and that an empty list means nothing rather than everything. However, having an empty list mean "everything" would make cases were less than all columns were selected since this would change from empty to a list. --> |
There was a problem hiding this comment.
| <!-- TODO What does an empty list of columns and rows mean? --> | |
| <!-- We discussed elsewhere that one purpose of this file is for someone to get an overview of which data is used in a specific study. On the one side, that could benefit from having to specify all the columns if the study used all of them and that an empty list means nothing rather than everything. However, having an empty list mean "everything" would make cases were less than all columns were selected since this would change from empty to a list. --> |
Addressed in my edit above.
There was a problem hiding this comment.
So, should this comment should be removed now?
| - 2012-01-01 | ||
| - 2012-12-31 |
There was a problem hiding this comment.
This one is tricky, as it would require that we sort or something after reading the values.
There was a problem hiding this comment.
If we don't trust the order, we could name the start and end points
There was a problem hiding this comment.
Yeah, in #3 (comment) I suggested to use min and max. There I also made a difference between comparing against a single value or a list of values. But then I thought it might be cleaner with the same key name everywhere. I'm open to either, but I don't think sorting is needed when mapping from the UI since there would be an order there.
One more thought is that we can get rid of between altogether and replace it with two conditions, eg. instead of the example above, we could require the following:
- column: col3
operator: "<="
value: 2012-01-01
- column: col3
operator: ">="
value: 2012-12-31I don't think that would be an inconvenience when specified in a UI. It also makes it more explicit whether the end points are inclusive or not, which I don't think is clear to someone using BETWEEN unless they have previous SQL knowledge.
martonvago
left a comment
There was a problem hiding this comment.
Very nice, I added my thoughts 😁
| - Row filters use supported SQL-like syntax. | ||
| - Only allowed SQL operations and shorthands are used. | ||
|
|
||
| While most of these check are likely easier to test directly in the yaml file, |
There was a problem hiding this comment.
| While most of these check are likely easier to test directly in the yaml file, | |
| While most of these checks are likely easier to test directly in the yaml file, |
| matter when editing manually and we allow more expressiveness here than, which | ||
| doesn't seem to be a common use case based on |
There was a problem hiding this comment.
Sentence broke down here a bit 😅
| - 2012-01-01 | ||
| - 2012-12-31 |
There was a problem hiding this comment.
If we don't trust the order, we could name the start and end points
| <!-- We discussed elsewhere that one purpose of this file is for someone to get an overview of which data is used in a specific study. On the one side, that could benefit from having to specify all the columns if the study used all of them and that an empty list means nothing rather than everything. However, having an empty list mean "everything" would make cases were less than all columns were selected since this would change from empty to a list. --> | ||
| <!-- For rows, it seems like the natural is that empty = all rows. So we can also consider if we want "empty" to mean the same thing (ie "all") both for columns and rows. --> | ||
|
|
||
| #### Row condition alternatives |
There was a problem hiding this comment.
Good discussion about this tricky bit!
My two cents:
I think that request.yaml is supposed to be a text-based representation of the request form in the UI, so it should follow the UI's structure as closely as possible. This is all the more true because request.yaml can still be edited by the user, so it's still an interface of sorts (unlike the payload sent from frontend to backend in a classic web app). The UI is structured like a syntax tree, which lends itself to quite a nice textual representation in YAML. Using an example from the web interface docs:
filters:
growth_records:
and:
- is_null:
column: location_country
not: true
- or:
- equal:
column: growth_stage
value: flowering
- greater_than:
column: observation_date
value: 2020-01-01Or the more verbose:
filters:
growth_records:
operator: and
children:
- column: location_country
operator: is null
not: true
- operator: or
children:
- column: growth_stage
operator: "="
value: flowering
- column: observation_date
operator: ">"
value: 2020-01-01This structure is a modified version of your alternative 2 that also includes grouping column conditions. The naming and exact structure can be refined of course.
I like this because keeping the structure aligned with the interface allows complexity (creating the SQL query) to be pushed further down into the core bits of Propagate. This strikes me as both safer and simpler.
Safer because, this way, the user cannot introduce arbitrary SQL-like expressions through the request file. Of course, we would defend against this in alternative 1 by parsing and checking the condition, but that is still riskier than sidestepping the issue altogether.
Simpler because we go
filter tree in UI > filter tree in YAML > filter tree in Rust > SQL
and not
filter tree in UI > SQL-like text in YAML > parsed SQL-like structure in Rust > SQL.
There was a problem hiding this comment.
Hmm, great ideas/thoughts here! My brain comprehends the second, more verbose option here, but I can see how either could work 🤔
There was a problem hiding this comment.
Yes, I also like this. I also prefer the second one because it is more explicit.
We could use words like group and condition, as I've done in the web.qmd - or at least align between the two (web and request YAML)
There was a problem hiding this comment.
I'm also in favor of keeping the yaml file structure as close as possible to how information is represented in the graphical UI! Agree with you that it becomes both safer and simpler for us, including simpler to check that only the allowed subset of SQL that we want to support is used.
I think the two suggestion above are quite similar to some of the expanded syntax suggestions for alternative two from this issue comment #3 (comment), which I think would become the following for the example above:
rows/filters:
- resource: growth_records
all:
- column: location_country
operator: is null
not: true
- any:
- column: growth_stage
operator: "="
value: flowering
- column: observation_date
operator: ">"
value: 2020-01-01I'm happy with all three options we have here in this discussion, and we essentially need to decide what level of verbosity we want. I think the suggestion I have from previously is somewhere between the two above and I have a preference for one of the two more verbose options we have in this discussion. I can comment on a few things in terms of why I picked them, but I'm open to landing on any of the above suggestions (or a combination of them):
- I prefer to specify the
resourcekey (versus just the resource name as the key) to make it more explicit what that value represents. - My preference for which logic operators/words to use when combining conditions/filters is
all/anywhen it's outside/wrapping the conditions andand/orwhen it is separating the conditions as an operator, ie I likeany(X, Y)and I likeX or Y. So, in our case I would preferall/anybut I'm good either way. - I prefer not to specify the
operatorkey twice at two different levels of nesting since it might be confusing that the same key name would take different operator values depending on the nesting level, ie onlyand/orat the resource level, but then all condition operators when it is on the inner level. I'm ok including it if we call this key something else at the outer level (e.g.match: any,logic: and, or similar). I'm also ok omitting it as in my example above since there are only two possible values. - If we wanted to improve the direct experience of editing the file directly, we could consider
allas the default and not require it to be written out. That would also lead to less nesting, but it is of course more explicit to write it out. Maybe we can think about how this will be represented in the UI as well to see what would be the closer mapping to that. - If we keep the most verbose option as in @martonvago's second options, then I prefer if we replace
childrenwith something likeconditions. - As for the outermost key name, I also suggested
filter,where, and evenrow_filterspreviously in the brainstorm issue, but @lwjohnst86 mentioned that he wanted it to berowsfor symmetry withcolumnseven if these are row filters rather than row names. I do find it more natural to refer to them as "filters" in my mind, but I'm good either way here; I think they have different advantages.
There was a problem hiding this comment.
Ah, I missed the second half of the comments on #3, sorry! I personally don't have a strong preference between what you suggested there and the examples I put here. So on all points above I'm very happy to go with your preference. I'm also leaning towards filters as a name for this, but I don't mind rows if that's what we want.
There was a problem hiding this comment.
I'm fine with your suggestions @joelostblom. These types of comments should be put into the doc (they are design decisions). Can you modify and add it to it?
As for 6. technically they are row "keeps", as the condition keeps rows, it doesn't filter them (filter means remove, not keep). Just a small semantic comment 😛
There was a problem hiding this comment.
Small - maybe misplaced - comment: "filtering" is both about keeping something and removing something else, so it's ambiguous in that way.
signekb
left a comment
There was a problem hiding this comment.
Nice work, Joel! This is a difficult one.
I've added some thoughts and suggestions:
| precisely, the following checks are made: | ||
|
|
||
| - Required sections and keys are present. | ||
| - The referenced data package contains the correct id and name. |
There was a problem hiding this comment.
| - The referenced data package contains the correct id and name. | |
| - The referenced data package has the same ID and name. |
Maybe?
There was a problem hiding this comment.
I added a slight rewrite of this
| subset of data from a data package. The file is created by a Requester through | ||
| the web application and is used by an Owner to check and generate the requested |
There was a problem hiding this comment.
You could link to the definition of the user types here 🔗
Co-authored-by: Luke W. Johnston <lwjohnst86@users.noreply.github.com>
Co-authored-by: martonvago <57952344+martonvago@users.noreply.github.com> Co-authored-by: Signe Kirk Brødbæk <40836345+signekb@users.noreply.github.com>
Co-authored-by: Luke W. Johnston <lwjohnst86@users.noreply.github.com>
Co-authored-by: Luke W. Johnston <lwjohnst86@users.noreply.github.com>
Co-authored-by: Luke W. Johnston <lwjohnst86@users.noreply.github.com>
Co-authored-by: Luke W. Johnston <lwjohnst86@users.noreply.github.com>
Co-authored-by: Luke W. Johnston <lwjohnst86@users.noreply.github.com>
joelostblom
left a comment
There was a problem hiding this comment.
Thanks for the detailed comments everyone! I had a go at incorporated almost all of it and adding to the discussion around the filter filters. Please have another look when you get a chance.
| The request file contains four top-level sections: | ||
|
|
||
| - `request`: Information about this specific request. | ||
| - `requester`: Information about the person and project requesting the data. |
There was a problem hiding this comment.
My bad, I forget that we updated this to not be nested during our call. As above I will add the elaborated sections below instead of here.
| either with the same resource multiple times or different ones. | ||
| - `columns`: A list of columns to include. Features such as selecting a range of | ||
| columns are handled via the interface and expanded to explicit column lists in | ||
| the request file. |
There was a problem hiding this comment.
I added a modified version of this
| columns are handled via the interface and expanded to explicit column lists in | ||
| the request file. | ||
| - `rows`: Describes which rows to include in the subset. Can reference columns | ||
| that are not part of the subset. |
There was a problem hiding this comment.
I added a modified version of this
| <!-- We discussed elsewhere that one purpose of this file is for someone to get an overview of which data is used in a specific study. On the one side, that could benefit from having to specify all the columns if the study used all of them and that an empty list means nothing rather than everything. However, having an empty list mean "everything" would make cases were less than all columns were selected since this would change from empty to a list. --> | ||
| <!-- For rows, it seems like the natural is that empty = all rows. So we can also consider if we want "empty" to mean the same thing (ie "all") both for columns and rows. --> | ||
|
|
||
| #### Row condition alternatives |
There was a problem hiding this comment.
I'm also in favor of keeping the yaml file structure as close as possible to how information is represented in the graphical UI! Agree with you that it becomes both safer and simpler for us, including simpler to check that only the allowed subset of SQL that we want to support is used.
I think the two suggestion above are quite similar to some of the expanded syntax suggestions for alternative two from this issue comment #3 (comment), which I think would become the following for the example above:
rows/filters:
- resource: growth_records
all:
- column: location_country
operator: is null
not: true
- any:
- column: growth_stage
operator: "="
value: flowering
- column: observation_date
operator: ">"
value: 2020-01-01I'm happy with all three options we have here in this discussion, and we essentially need to decide what level of verbosity we want. I think the suggestion I have from previously is somewhere between the two above and I have a preference for one of the two more verbose options we have in this discussion. I can comment on a few things in terms of why I picked them, but I'm open to landing on any of the above suggestions (or a combination of them):
- I prefer to specify the
resourcekey (versus just the resource name as the key) to make it more explicit what that value represents. - My preference for which logic operators/words to use when combining conditions/filters is
all/anywhen it's outside/wrapping the conditions andand/orwhen it is separating the conditions as an operator, ie I likeany(X, Y)and I likeX or Y. So, in our case I would preferall/anybut I'm good either way. - I prefer not to specify the
operatorkey twice at two different levels of nesting since it might be confusing that the same key name would take different operator values depending on the nesting level, ie onlyand/orat the resource level, but then all condition operators when it is on the inner level. I'm ok including it if we call this key something else at the outer level (e.g.match: any,logic: and, or similar). I'm also ok omitting it as in my example above since there are only two possible values. - If we wanted to improve the direct experience of editing the file directly, we could consider
allas the default and not require it to be written out. That would also lead to less nesting, but it is of course more explicit to write it out. Maybe we can think about how this will be represented in the UI as well to see what would be the closer mapping to that. - If we keep the most verbose option as in @martonvago's second options, then I prefer if we replace
childrenwith something likeconditions. - As for the outermost key name, I also suggested
filter,where, and evenrow_filterspreviously in the brainstorm issue, but @lwjohnst86 mentioned that he wanted it to berowsfor symmetry withcolumnseven if these are row filters rather than row names. I do find it more natural to refer to them as "filters" in my mind, but I'm good either way here; I think they have different advantages.
| - Missing values: `IS NULL`, `IS NOT NULL` | ||
| - E.g. `"end_date IS NULL"` | ||
| - Negation: `NOT` | ||
| - E.g. `NOT =`, `NOT IN`, `NOT BETWEEN` |
There was a problem hiding this comment.
One slightly annoying thing is the NOT is a prefix to negating all operators except IS NULL where it is advised to use IS NOT NULL rather than NOT IS NULL because of how null comparisons work in sql.
Also it does seem like the most common way to compare inequality for numericals are <>, second most common is !=, so ours is a bit less standard, but we also don't have to do the most standard sql thing since editing the file is not a priority use case
martonvago
left a comment
There was a problem hiding this comment.
Just some typos, otherwise this is taking shape very nicely 🎉
| <!-- We discussed elsewhere that one purpose of this file is for someone to get an overview of which data is used in a specific study. On the one side, that could benefit from having to specify all the columns if the study used all of them and that an empty list means nothing rather than everything. However, having an empty list mean "everything" would make cases were less than all columns were selected since this would change from empty to a list. --> | ||
| <!-- For rows, it seems like the natural is that empty = all rows. So we can also consider if we want "empty" to mean the same thing (ie "all") both for columns and rows. --> | ||
|
|
||
| #### Row condition alternatives |
There was a problem hiding this comment.
Ah, I missed the second half of the comments on #3, sorry! I personally don't have a strong preference between what you suggested there and the examples I put here. So on all points above I'm very happy to go with your preference. I'm also leaning towards filters as a name for this, but I don't mind rows if that's what we want.
Co-authored-by: martonvago <57952344+martonvago@users.noreply.github.com>
|
Thanks for noticing the typos; I applied all your suggestions! |
| which facilitates referring to a specific iteration of a request, e.g. in | ||
| dialog between the owner and requester. Precision to the second should be | ||
| sufficient. | ||
| - Note that this field should not be updated via the user interface, it should |
There was a problem hiding this comment.
I've added a suggestion trying to incorporate this.
| subsets can be specified in the same request by repeating the `resource` key. | ||
| This can be done either with the same resource multiple times or with | ||
| different resources for each. | ||
| - `columns`: The columns to include. Each list item must correspond to an exact |
There was a problem hiding this comment.
So, we do not include a shorthand for saying "include all columns from this resource"?
There was a problem hiding this comment.
@lwjohnst86 mentioned that he prefers if all columns that are part of the data subset are mentioned explicitly in the file (which I think is part of why we also don't have exclusions, ranges, etc). I believe the reasoning is that a person should be able to see which variables are used in the project when opening request.yaml manually without knowing about the content of the referenced data package (but I will @lwjohnst86 fill in if I misremember).
There was a problem hiding this comment.
I guess it could be that if columns is missing (like rows), it interprets that as all columns? Like the all rows for no rows?
There was a problem hiding this comment.
Yes, that was one of the suggestions in my previous comment in the file, so that there would be symmetry between the meaning of an empty value for rows and columns. It would also make it very clear when e.g. all columns except one was selected.
I thought you wanted all columns listed explicitly for the reasons I mentioned above, but if that is no longer the case, then we just need to decide whether it is clear enough that a request that looks like this, requests all the columns and rows from resource1:
subsets:
- resource: resource1
- resource: resource2
columns:
- col1
rows:
- col1 > 50I think it probably is clear enough, and we can write that "...if you don't want to request all the data in a resource, you can select specific columns and rows to include in the data subset..."
There was a problem hiding this comment.
I really like the idea of there being symmetry between empty values from rows and columns both meaning selecting all. To me, your example above is clear 👍
There was a problem hiding this comment.
I think with @martonvago comment about mirroring the web app/tui interface and how it would actually be implememted, it makes sense that no columns selected means all columns requested from the resource (which maybe? mimics how we would do it in the app..?). So yea, let's do that, as @signekb says, it also has some symmetry.
| <!-- TODO What does an empty list of columns and rows mean? --> | ||
| <!-- We discussed elsewhere that one purpose of this file is for someone to get an overview of which data is used in a specific study. On the one side, that could benefit from having to specify all the columns if the study used all of them and that an empty list means nothing rather than everything. However, having an empty list mean "everything" would make cases were less than all columns were selected since this would change from empty to a list. --> |
There was a problem hiding this comment.
So, should this comment should be removed now?
Co-authored-by: Signe Kirk Brødbæk <40836345+signekb@users.noreply.github.com>
joelostblom
left a comment
There was a problem hiding this comment.
Thanks for the comments, updated!
|
|
||
| The `data-package` section identifies the data package the request applies to. | ||
| This information will be automatically populated from the data package metadata | ||
| when creating a request via the web application. |
There was a problem hiding this comment.
Good point! I added a line about that as a possible use case.
| subsets can be specified in the same request by repeating the `resource` key. | ||
| This can be done either with the same resource multiple times or with | ||
| different resources for each. | ||
| - `columns`: The columns to include. Each list item must correspond to an exact |
There was a problem hiding this comment.
@lwjohnst86 mentioned that he prefers if all columns that are part of the data subset are mentioned explicitly in the file (which I think is part of why we also don't have exclusions, ranges, etc). I believe the reasoning is that a person should be able to see which variables are used in the project when opening request.yaml manually without knowing about the content of the referenced data package (but I will @lwjohnst86 fill in if I misremember).
| This can be done either with the same resource multiple times or with | ||
| different resources for each. |
There was a problem hiding this comment.
I think it is still up for discussion whether we prefer to support that or not. I mentioned the following in #65 (comment), but that comment is not easy to find now:
I also do think we should be explicit about the possibility to either repeat the same resource or referring to different ones within the same request. Unless you are saying that you don't want a requester to be able to repeat a resource and all data from a single resource needs to be extracted to a single subset in a single query. I'm ok with that too, but if that's the intention I think it would be beneficial to write that explicitly.
As a direct answer to your question; I would imagine a scenario where someone is interested in looking into column A for a subset of the population and then column B for another subset. If the data was very large or a column very heavy (e.g. images or geo-coordinates), it could be impractical if we forced the user to pull column A and B for the union of two subset (e.g. they needed to pull down images for 10 rows as an example and we force them to pull 1 million images because that's how big the subset for the other column is).
| requested is aligned with what it is intended to be used for, which is a legal | ||
| requirement (e.g. in GDPR). | ||
|
|
||
| - `name`: A short machine-readable project name, using lowercase letters and |
There was a problem hiding this comment.
That's a good consideration! I think it depends on what this name field is meant to be used for. We previously discussed that this should be machine readable and used for identification purposes. That's quite different from the description of the name key for the data package below and to me this description suggests that project.name is not (primarily) meant for human consumption and therefore does not need to be short (and maybe even benefits from being longer since it makes it more unique).
However, if it is meant for human consumption, then I agree with you that it would benefit from being shorter than the title. I think either works and I also think we don't need name: Machines can create an id from the title and humans will create an abbreviation of the title themselves as well to use as a short name. So I'm happy to implement according to anyone feeling more strongly about this than I do.
lwjohnst86
left a comment
There was a problem hiding this comment.
It's coming along nicely
| - Only allowed SQL operations and shorthands are used. | ||
|
|
||
| While most of these check are likely easier to test directly in the YAML file, | ||
| the subset queries filters can also be converted to SQL and checked with |
There was a problem hiding this comment.
| the subset queries filters can also be converted to SQL and checked with | |
| the subset queries can also be converted to SQL and checked with |
| date-modified: DATE | ||
| date-created: DATE |
There was a problem hiding this comment.
| date-modified: DATE | |
| date-created: DATE | |
| datetime-modified: DATETIME | |
| datetime-created: DATETIME |
| iteration: NUMBER | ||
| ``` | ||
|
|
||
| - `date-modified`: When the request was last modified. This can act as an ID |
There was a problem hiding this comment.
| - `date-modified`: When the request was last modified. This can act as an ID | |
| - `datetime-modified`: When the request was last modified. This can act as an ID |
|
|
||
| - `resource`: Specifies the data resource to create the subset from. Multiple | ||
| subsets can be specified in the same request by repeating the `resource` key. | ||
| This can be done either with the same resource multiple times or with |
There was a problem hiding this comment.
| This can be done either with the same resource multiple times or with | |
| This can be done with |
| This can be done either with the same resource multiple times or with | ||
| different resources for each. |
There was a problem hiding this comment.
I'm not completely sure where this belongs, maybe in this file, but probably should be clarified in the requirements.
For GDPR purposes (and other legal/privacy regulations), as well as for statisticaly reasons, any subsetting by rows in any resource will lead to the observational units (participants if human) being the same across all resources. For example, if in survey a condition is Gender == "Man", and in another resource biometrics that doesn't have a condition, the participant IDs from the subsetted survey will be applied to the biometrics. So that all resources contain the same set of participants. This is a legal thing, as much as a statistical thing. It's to ensure that only the necessary data is given to a researcher that needs it. The principle of "least permissions" but in data context 😛
So, in your example, that won't be allowed to request from the same resource more than once. But actually, your example highlights that they won't be given everything when they only need a specific amount (e.g. the ask for 10 rows and only get images for those 10 rows).
| subsets can be specified in the same request by repeating the `resource` key. | ||
| This can be done either with the same resource multiple times or with | ||
| different resources for each. | ||
| - `columns`: The columns to include. Each list item must correspond to an exact |
There was a problem hiding this comment.
I think with @martonvago comment about mirroring the web app/tui interface and how it would actually be implememted, it makes sense that no columns selected means all columns requested from the resource (which maybe? mimics how we would do it in the app..?). So yea, let's do that, as @signekb says, it also has some symmetry.
| <!-- We discussed elsewhere that one purpose of this file is for someone to get an overview of which data is used in a specific study. On the one side, that could benefit from having to specify all the columns if the study used all of them and that an empty list means nothing rather than everything. However, having an empty list mean "everything" would make cases were less than all columns were selected since this would change from empty to a list. --> | ||
| <!-- For rows, it seems like the natural is that empty = all rows. So we can also consider if we want "empty" to mean the same thing (ie "all") both for columns and rows. --> | ||
|
|
||
| #### Row condition alternatives |
There was a problem hiding this comment.
I'm fine with your suggestions @joelostblom. These types of comments should be put into the doc (they are design decisions). Can you modify and add it to it?
As for 6. technically they are row "keeps", as the condition keeps rows, it doesn't filter them (filter means remove, not keep). Just a small semantic comment 😛
| Propagate is not intended to be a general-purpose query engine. The initial | ||
| request format should not support joins, grouping, arbitrary functions, random | ||
| sampling, or row selection by position. We also don't support shorthands such as `*` since that would make it less | ||
| explicit which columns are used from just looking at the request file without |
There was a problem hiding this comment.
this paragraph should be at the top as a callout block.
Description
Sorry this took longer than planned.
Note that there is some overlap here with the web design doc in terms of the language we have designed for making the requests.
Also see inline
TODOcomments for some questions that would benefit from discussing which way to take forward.Closes #10, closes #3
Needs a thorough review.
Checklist
just run-all