Skip to content

docs: 📝 add design docs for the request file#65

Open
joelostblom wants to merge 26 commits into
mainfrom
docs/instruction-design
Open

docs: 📝 add design docs for the request file#65
joelostblom wants to merge 26 commits into
mainfrom
docs/instruction-design

Conversation

@joelostblom

Copy link
Copy Markdown
Contributor

Description

Sorry this took longer than planned.

Note that there is some overlap here with the web design doc in terms of the language we have designed for making the requests.

Also see inline TODO comments for some questions that would benefit from discussing which way to take forward.

Closes #10, closes #3

Needs a thorough review.

Checklist

  • Ran just run-all

@joelostblom joelostblom marked this pull request as ready for review July 7, 2026 10:11
@joelostblom joelostblom requested a review from a team as a code owner July 7, 2026 10:11
@joelostblom joelostblom moved this from In Progress to In Review in Product development Jul 7, 2026
Comment thread docs/design/interface/cli.qmd

@lwjohnst86 lwjohnst86 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great start 😁 here are some initial changes.

Comment thread docs/design/interface/requests.qmd Outdated
Comment thread docs/design/interface/requests.qmd Outdated
Comment on lines +274 to +281
### Non-goals

- Propagate is not intended to be a general-purpose query engine. The initial
request format should not support joins, grouping, arbitrary functions, random
sampling, or row selection by position.
- We also don't support shorthands such as `*` since that would make it less
explicit which columns are used from just looking at the request file without
also seeing the data package metadata.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These should be included at the location where they are relevant. Design docs are about the what and the why, and these are why things, which should be treated as important as the rest of the text.

Comment thread docs/design/interface/cli.qmd Outdated
Comment thread docs/design/interface/cli.qmd Outdated
Comment thread docs/design/interface/requests.qmd Outdated
Comment thread docs/design/interface/requests.qmd Outdated
Comment thread docs/design/interface/requests.qmd Outdated
Comment on lines +130 to +131
<!-- TODO What does an empty list of columns and rows mean? -->
<!-- We discussed elsewhere that one purpose of this file is for someone to get an overview of which data is used in a specific study. On the one side, that could benefit from having to specify all the columns if the study used all of them and that an empty list means nothing rather than everything. However, having an empty list mean "everything" would make cases were less than all columns were selected since this would change from empty to a list. -->

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
<!-- TODO What does an empty list of columns and rows mean? -->
<!-- We discussed elsewhere that one purpose of this file is for someone to get an overview of which data is used in a specific study. On the one side, that could benefit from having to specify all the columns if the study used all of them and that an empty list means nothing rather than everything. However, having an empty list mean "everything" would make cases were less than all columns were selected since this would change from empty to a list. -->

Addressed in my edit above.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, should this comment should be removed now?

Comment thread docs/design/interface/requests.qmd
Comment thread docs/design/interface/requests.qmd
Comment on lines +216 to +217
- 2012-01-01
- 2012-12-31

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This one is tricky, as it would require that we sort or something after reading the values.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we don't trust the order, we could name the start and end points

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, in #3 (comment) I suggested to use min and max. There I also made a difference between comparing against a single value or a list of values. But then I thought it might be cleaner with the same key name everywhere. I'm open to either, but I don't think sorting is needed when mapping from the UI since there would be an order there.

One more thought is that we can get rid of between altogether and replace it with two conditions, eg. instead of the example above, we could require the following:

  - column: col3
    operator: "<="
    value: 2012-01-01
  - column: col3
    operator: ">="
    value: 2012-12-31

I don't think that would be an inconvenience when specified in a UI. It also makes it more explicit whether the end points are inclusive or not, which I don't think is clear to someone using BETWEEN unless they have previous SQL knowledge.

@github-project-automation github-project-automation Bot moved this from In Review to In Progress in Product development Jul 7, 2026

@martonvago martonvago left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nice, I added my thoughts 😁

Comment thread docs/design/interface/cli.qmd Outdated
Comment thread docs/design/interface/cli.qmd Outdated
- Row filters use supported SQL-like syntax.
- Only allowed SQL operations and shorthands are used.

While most of these check are likely easier to test directly in the yaml file,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
While most of these check are likely easier to test directly in the yaml file,
While most of these checks are likely easier to test directly in the yaml file,

Comment thread docs/design/interface/requests.qmd Outdated
Comment thread docs/design/interface/requests.qmd Outdated
Comment on lines +192 to +193
matter when editing manually and we allow more expressiveness here than, which
doesn't seem to be a common use case based on

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sentence broke down here a bit 😅

Comment thread docs/design/interface/requests.qmd
Comment on lines +216 to +217
- 2012-01-01
- 2012-12-31

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we don't trust the order, we could name the start and end points

<!-- We discussed elsewhere that one purpose of this file is for someone to get an overview of which data is used in a specific study. On the one side, that could benefit from having to specify all the columns if the study used all of them and that an empty list means nothing rather than everything. However, having an empty list mean "everything" would make cases were less than all columns were selected since this would change from empty to a list. -->
<!-- For rows, it seems like the natural is that empty = all rows. So we can also consider if we want "empty" to mean the same thing (ie "all") both for columns and rows. -->

#### Row condition alternatives

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good discussion about this tricky bit!

My two cents:
I think that request.yaml is supposed to be a text-based representation of the request form in the UI, so it should follow the UI's structure as closely as possible. This is all the more true because request.yaml can still be edited by the user, so it's still an interface of sorts (unlike the payload sent from frontend to backend in a classic web app). The UI is structured like a syntax tree, which lends itself to quite a nice textual representation in YAML. Using an example from the web interface docs:

filters:
  growth_records:
    and:
      - is_null:
          column: location_country
          not: true
      - or:
          - equal:
              column: growth_stage
              value: flowering
          - greater_than:
              column: observation_date
              value: 2020-01-01

Or the more verbose:

filters:
  growth_records:
    operator: and
    children:
      - column: location_country
        operator: is null
        not: true

      - operator: or
        children:
          - column: growth_stage
            operator: "="
            value: flowering

          - column: observation_date
            operator: ">"
            value: 2020-01-01

This structure is a modified version of your alternative 2 that also includes grouping column conditions. The naming and exact structure can be refined of course.

I like this because keeping the structure aligned with the interface allows complexity (creating the SQL query) to be pushed further down into the core bits of Propagate. This strikes me as both safer and simpler.

Safer because, this way, the user cannot introduce arbitrary SQL-like expressions through the request file. Of course, we would defend against this in alternative 1 by parsing and checking the condition, but that is still riskier than sidestepping the issue altogether.

Simpler because we go
filter tree in UI > filter tree in YAML > filter tree in Rust > SQL
and not
filter tree in UI > SQL-like text in YAML > parsed SQL-like structure in Rust > SQL.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, great ideas/thoughts here! My brain comprehends the second, more verbose option here, but I can see how either could work 🤔

@signekb signekb Jul 8, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I also like this. I also prefer the second one because it is more explicit.

We could use words like group and condition, as I've done in the web.qmd - or at least align between the two (web and request YAML)

@joelostblom joelostblom Jul 13, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm also in favor of keeping the yaml file structure as close as possible to how information is represented in the graphical UI! Agree with you that it becomes both safer and simpler for us, including simpler to check that only the allowed subset of SQL that we want to support is used.

I think the two suggestion above are quite similar to some of the expanded syntax suggestions for alternative two from this issue comment #3 (comment), which I think would become the following for the example above:

rows/filters:
  - resource: growth_records
    all:
      - column: location_country
        operator: is null
        not: true
      - any:
        - column: growth_stage
          operator: "="
          value: flowering
        - column: observation_date
          operator: ">"
          value: 2020-01-01

I'm happy with all three options we have here in this discussion, and we essentially need to decide what level of verbosity we want. I think the suggestion I have from previously is somewhere between the two above and I have a preference for one of the two more verbose options we have in this discussion. I can comment on a few things in terms of why I picked them, but I'm open to landing on any of the above suggestions (or a combination of them):

  1. I prefer to specify the resource key (versus just the resource name as the key) to make it more explicit what that value represents.
  2. My preference for which logic operators/words to use when combining conditions/filters is all/any when it's outside/wrapping the conditions and and/or when it is separating the conditions as an operator, ie I like any(X, Y) and I like X or Y. So, in our case I would prefer all/any but I'm good either way.
  3. I prefer not to specify the operator key twice at two different levels of nesting since it might be confusing that the same key name would take different operator values depending on the nesting level, ie only and/or at the resource level, but then all condition operators when it is on the inner level. I'm ok including it if we call this key something else at the outer level (e.g. match: any, logic: and, or similar). I'm also ok omitting it as in my example above since there are only two possible values.
  4. If we wanted to improve the direct experience of editing the file directly, we could consider all as the default and not require it to be written out. That would also lead to less nesting, but it is of course more explicit to write it out. Maybe we can think about how this will be represented in the UI as well to see what would be the closer mapping to that.
  5. If we keep the most verbose option as in @martonvago's second options, then I prefer if we replace children with something like conditions.
  6. As for the outermost key name, I also suggested filter, where, and even row_filters previously in the brainstorm issue, but @lwjohnst86 mentioned that he wanted it to be rows for symmetry with columns even if these are row filters rather than row names. I do find it more natural to refer to them as "filters" in my mind, but I'm good either way here; I think they have different advantages.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I missed the second half of the comments on #3, sorry! I personally don't have a strong preference between what you suggested there and the examples I put here. So on all points above I'm very happy to go with your preference. I'm also leaning towards filters as a name for this, but I don't mind rows if that's what we want.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm fine with your suggestions @joelostblom. These types of comments should be put into the doc (they are design decisions). Can you modify and add it to it?

As for 6. technically they are row "keeps", as the condition keeps rows, it doesn't filter them (filter means remove, not keep). Just a small semantic comment 😛

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small - maybe misplaced - comment: "filtering" is both about keeping something and removing something else, so it's ambiguous in that way.

@signekb signekb left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work, Joel! This is a difficult one.
I've added some thoughts and suggestions:

Comment thread docs/design/interface/cli.qmd Outdated
precisely, the following checks are made:

- Required sections and keys are present.
- The referenced data package contains the correct id and name.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- The referenced data package contains the correct id and name.
- The referenced data package has the same ID and name.

Maybe?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added a slight rewrite of this

Comment thread docs/design/interface/cli.qmd Outdated
Comment thread docs/design/interface/cli.qmd Outdated
Comment thread docs/design/interface/requests.qmd Outdated
Comment thread docs/design/interface/requests.qmd Outdated
Comment on lines +6 to +7
subset of data from a data package. The file is created by a Requester through
the web application and is used by an Owner to check and generate the requested

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You could link to the definition of the user types here 🔗

Comment thread docs/design/interface/requests.qmd Outdated
Comment thread docs/design/interface/requests.qmd Outdated
Comment thread docs/design/interface/requests.qmd
Comment thread docs/design/interface/requests.qmd
Comment thread docs/design/interface/requests.qmd Outdated
joelostblom and others added 9 commits July 9, 2026 16:51
Co-authored-by: Luke W. Johnston <lwjohnst86@users.noreply.github.com>
Co-authored-by: martonvago <57952344+martonvago@users.noreply.github.com>
Co-authored-by: Signe Kirk Brødbæk <40836345+signekb@users.noreply.github.com>
Co-authored-by: Luke W. Johnston <lwjohnst86@users.noreply.github.com>
Co-authored-by: Luke W. Johnston <lwjohnst86@users.noreply.github.com>
Co-authored-by: Luke W. Johnston <lwjohnst86@users.noreply.github.com>
Co-authored-by: Luke W. Johnston <lwjohnst86@users.noreply.github.com>
joelostblom and others added 2 commits July 13, 2026 11:49
Co-authored-by: Luke W. Johnston <lwjohnst86@users.noreply.github.com>

@joelostblom joelostblom left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the detailed comments everyone! I had a go at incorporated almost all of it and adding to the discussion around the filter filters. Please have another look when you get a chance.

Comment thread docs/design/interface/requests.qmd
Comment thread docs/design/interface/requests.qmd
Comment thread docs/design/interface/requests.qmd Outdated
The request file contains four top-level sections:

- `request`: Information about this specific request.
- `requester`: Information about the person and project requesting the data.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My bad, I forget that we updated this to not be nested during our call. As above I will add the elaborated sections below instead of here.

Comment thread docs/design/interface/requests.qmd Outdated
Comment thread docs/design/interface/requests.qmd
Comment thread docs/design/interface/requests.qmd Outdated
either with the same resource multiple times or different ones.
- `columns`: A list of columns to include. Features such as selecting a range of
columns are handled via the interface and expanded to explicit column lists in
the request file.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added a modified version of this

Comment thread docs/design/interface/requests.qmd Outdated
columns are handled via the interface and expanded to explicit column lists in
the request file.
- `rows`: Describes which rows to include in the subset. Can reference columns
that are not part of the subset.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added a modified version of this

Comment thread docs/design/interface/requests.qmd Outdated
Comment thread docs/design/interface/requests.qmd
<!-- We discussed elsewhere that one purpose of this file is for someone to get an overview of which data is used in a specific study. On the one side, that could benefit from having to specify all the columns if the study used all of them and that an empty list means nothing rather than everything. However, having an empty list mean "everything" would make cases were less than all columns were selected since this would change from empty to a list. -->
<!-- For rows, it seems like the natural is that empty = all rows. So we can also consider if we want "empty" to mean the same thing (ie "all") both for columns and rows. -->

#### Row condition alternatives

@joelostblom joelostblom Jul 13, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm also in favor of keeping the yaml file structure as close as possible to how information is represented in the graphical UI! Agree with you that it becomes both safer and simpler for us, including simpler to check that only the allowed subset of SQL that we want to support is used.

I think the two suggestion above are quite similar to some of the expanded syntax suggestions for alternative two from this issue comment #3 (comment), which I think would become the following for the example above:

rows/filters:
  - resource: growth_records
    all:
      - column: location_country
        operator: is null
        not: true
      - any:
        - column: growth_stage
          operator: "="
          value: flowering
        - column: observation_date
          operator: ">"
          value: 2020-01-01

I'm happy with all three options we have here in this discussion, and we essentially need to decide what level of verbosity we want. I think the suggestion I have from previously is somewhere between the two above and I have a preference for one of the two more verbose options we have in this discussion. I can comment on a few things in terms of why I picked them, but I'm open to landing on any of the above suggestions (or a combination of them):

  1. I prefer to specify the resource key (versus just the resource name as the key) to make it more explicit what that value represents.
  2. My preference for which logic operators/words to use when combining conditions/filters is all/any when it's outside/wrapping the conditions and and/or when it is separating the conditions as an operator, ie I like any(X, Y) and I like X or Y. So, in our case I would prefer all/any but I'm good either way.
  3. I prefer not to specify the operator key twice at two different levels of nesting since it might be confusing that the same key name would take different operator values depending on the nesting level, ie only and/or at the resource level, but then all condition operators when it is on the inner level. I'm ok including it if we call this key something else at the outer level (e.g. match: any, logic: and, or similar). I'm also ok omitting it as in my example above since there are only two possible values.
  4. If we wanted to improve the direct experience of editing the file directly, we could consider all as the default and not require it to be written out. That would also lead to less nesting, but it is of course more explicit to write it out. Maybe we can think about how this will be represented in the UI as well to see what would be the closer mapping to that.
  5. If we keep the most verbose option as in @martonvago's second options, then I prefer if we replace children with something like conditions.
  6. As for the outermost key name, I also suggested filter, where, and even row_filters previously in the brainstorm issue, but @lwjohnst86 mentioned that he wanted it to be rows for symmetry with columns even if these are row filters rather than row names. I do find it more natural to refer to them as "filters" in my mind, but I'm good either way here; I think they have different advantages.

@joelostblom joelostblom moved this from In Progress to In Review in Product development Jul 13, 2026
- Missing values: `IS NULL`, `IS NOT NULL`
- E.g. `"end_date IS NULL"`
- Negation: `NOT`
- E.g. `NOT =`, `NOT IN`, `NOT BETWEEN`

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One slightly annoying thing is the NOT is a prefix to negating all operators except IS NULL where it is advised to use IS NOT NULL rather than NOT IS NULL because of how null comparisons work in sql.

Also it does seem like the most common way to compare inequality for numericals are <>, second most common is !=, so ours is a bit less standard, but we also don't have to do the most standard sql thing since editing the file is not a priority use case

@martonvago martonvago left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just some typos, otherwise this is taking shape very nicely 🎉

Comment thread docs/design/interface/cli.qmd Outdated
Comment thread docs/design/interface/requests.qmd Outdated
Comment thread docs/design/interface/requests.qmd Outdated
Comment thread docs/design/interface/requests.qmd Outdated
<!-- We discussed elsewhere that one purpose of this file is for someone to get an overview of which data is used in a specific study. On the one side, that could benefit from having to specify all the columns if the study used all of them and that an empty list means nothing rather than everything. However, having an empty list mean "everything" would make cases were less than all columns were selected since this would change from empty to a list. -->
<!-- For rows, it seems like the natural is that empty = all rows. So we can also consider if we want "empty" to mean the same thing (ie "all") both for columns and rows. -->

#### Row condition alternatives

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I missed the second half of the comments on #3, sorry! I personally don't have a strong preference between what you suggested there and the examples I put here. So on all points above I'm very happy to go with your preference. I'm also leaning towards filters as a name for this, but I don't mind rows if that's what we want.

Comment thread docs/design/interface/requests.qmd Outdated
@github-project-automation github-project-automation Bot moved this from In Review to In Progress in Product development Jul 14, 2026
Co-authored-by: martonvago <57952344+martonvago@users.noreply.github.com>
@joelostblom joelostblom requested a review from martonvago July 14, 2026 13:43
@joelostblom joelostblom moved this from In Progress to In Review in Product development Jul 14, 2026
@joelostblom

Copy link
Copy Markdown
Contributor Author

Thanks for noticing the typos; I applied all your suggestions!

@signekb signekb left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good, this is moving along nicely 🚀
Edit: Damn, accidentally clicked "comment" instead of "request changes". Sorry about that.

Comment thread docs/design/interface/requests.qmd Outdated
Comment thread docs/design/interface/requests.qmd Outdated
Comment thread docs/design/interface/requests.qmd Outdated
Comment thread docs/design/interface/requests.qmd Outdated
which facilitates referring to a specific iteration of a request, e.g. in
dialog between the owner and requester. Precision to the second should be
sufficient.
- Note that this field should not be updated via the user interface, it should

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've added a suggestion trying to incorporate this.

Comment thread docs/design/interface/requests.qmd
Comment thread docs/design/interface/requests.qmd Outdated
subsets can be specified in the same request by repeating the `resource` key.
This can be done either with the same resource multiple times or with
different resources for each.
- `columns`: The columns to include. Each list item must correspond to an exact

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, we do not include a shorthand for saying "include all columns from this resource"?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lwjohnst86 mentioned that he prefers if all columns that are part of the data subset are mentioned explicitly in the file (which I think is part of why we also don't have exclusions, ranges, etc). I believe the reasoning is that a person should be able to see which variables are used in the project when opening request.yaml manually without knowing about the content of the referenced data package (but I will @lwjohnst86 fill in if I misremember).

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess it could be that if columns is missing (like rows), it interprets that as all columns? Like the all rows for no rows?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that was one of the suggestions in my previous comment in the file, so that there would be symmetry between the meaning of an empty value for rows and columns. It would also make it very clear when e.g. all columns except one was selected.

I thought you wanted all columns listed explicitly for the reasons I mentioned above, but if that is no longer the case, then we just need to decide whether it is clear enough that a request that looks like this, requests all the columns and rows from resource1:

subsets:
  - resource: resource1
  - resource: resource2
    columns:
      - col1
    rows:
      - col1 > 50

I think it probably is clear enough, and we can write that "...if you don't want to request all the data in a resource, you can select specific columns and rows to include in the data subset..."

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I really like the idea of there being symmetry between empty values from rows and columns both meaning selecting all. To me, your example above is clear 👍

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think with @martonvago comment about mirroring the web app/tui interface and how it would actually be implememted, it makes sense that no columns selected means all columns requested from the resource (which maybe? mimics how we would do it in the app..?). So yea, let's do that, as @signekb says, it also has some symmetry.

Comment thread docs/design/interface/requests.qmd Outdated
Comment thread docs/design/interface/requests.qmd Outdated
Comment thread docs/design/interface/requests.qmd Outdated
Comment on lines +130 to +131
<!-- TODO What does an empty list of columns and rows mean? -->
<!-- We discussed elsewhere that one purpose of this file is for someone to get an overview of which data is used in a specific study. On the one side, that could benefit from having to specify all the columns if the study used all of them and that an empty list means nothing rather than everything. However, having an empty list mean "everything" would make cases were less than all columns were selected since this would change from empty to a list. -->

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, should this comment should be removed now?

joelostblom and others added 3 commits July 15, 2026 06:57
Co-authored-by: Signe Kirk Brødbæk <40836345+signekb@users.noreply.github.com>

@joelostblom joelostblom left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the comments, updated!

Comment thread docs/design/interface/requests.qmd Outdated

The `data-package` section identifies the data package the request applies to.
This information will be automatically populated from the data package metadata
when creating a request via the web application.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point! I added a line about that as a possible use case.

subsets can be specified in the same request by repeating the `resource` key.
This can be done either with the same resource multiple times or with
different resources for each.
- `columns`: The columns to include. Each list item must correspond to an exact

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lwjohnst86 mentioned that he prefers if all columns that are part of the data subset are mentioned explicitly in the file (which I think is part of why we also don't have exclusions, ranges, etc). I believe the reasoning is that a person should be able to see which variables are used in the project when opening request.yaml manually without knowing about the content of the referenced data package (but I will @lwjohnst86 fill in if I misremember).

Comment on lines +153 to +154
This can be done either with the same resource multiple times or with
different resources for each.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it is still up for discussion whether we prefer to support that or not. I mentioned the following in #65 (comment), but that comment is not easy to find now:

I also do think we should be explicit about the possibility to either repeat the same resource or referring to different ones within the same request. Unless you are saying that you don't want a requester to be able to repeat a resource and all data from a single resource needs to be extracted to a single subset in a single query. I'm ok with that too, but if that's the intention I think it would be beneficial to write that explicitly.

As a direct answer to your question; I would imagine a scenario where someone is interested in looking into column A for a subset of the population and then column B for another subset. If the data was very large or a column very heavy (e.g. images or geo-coordinates), it could be impractical if we forced the user to pull column A and B for the union of two subset (e.g. they needed to pull down images for 10 rows as an example and we force them to pull 1 million images because that's how big the subset for the other column is).

requested is aligned with what it is intended to be used for, which is a legal
requirement (e.g. in GDPR).

- `name`: A short machine-readable project name, using lowercase letters and

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a good consideration! I think it depends on what this name field is meant to be used for. We previously discussed that this should be machine readable and used for identification purposes. That's quite different from the description of the name key for the data package below and to me this description suggests that project.name is not (primarily) meant for human consumption and therefore does not need to be short (and maybe even benefits from being longer since it makes it more unique).

However, if it is meant for human consumption, then I agree with you that it would benefit from being shorter than the title. I think either works and I also think we don't need name: Machines can create an id from the title and humans will create an abbreviation of the title themselves as well to use as a short name. So I'm happy to implement according to anyone feeling more strongly about this than I do.

@joelostblom joelostblom requested a review from signekb July 15, 2026 05:42

@lwjohnst86 lwjohnst86 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's coming along nicely ☺️

- Only allowed SQL operations and shorthands are used.

While most of these check are likely easier to test directly in the YAML file,
the subset queries filters can also be converted to SQL and checked with

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
the subset queries filters can also be converted to SQL and checked with
the subset queries can also be converted to SQL and checked with

Comment thread docs/design/interface/requests.qmd
Comment thread docs/design/interface/requests.qmd
Comment on lines +44 to +45
date-modified: DATE
date-created: DATE

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
date-modified: DATE
date-created: DATE
datetime-modified: DATETIME
datetime-created: DATETIME

iteration: NUMBER
```

- `date-modified`: When the request was last modified. This can act as an ID

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- `date-modified`: When the request was last modified. This can act as an ID
- `datetime-modified`: When the request was last modified. This can act as an ID


- `resource`: Specifies the data resource to create the subset from. Multiple
subsets can be specified in the same request by repeating the `resource` key.
This can be done either with the same resource multiple times or with

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
This can be done either with the same resource multiple times or with
This can be done with

Comment on lines +153 to +154
This can be done either with the same resource multiple times or with
different resources for each.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not completely sure where this belongs, maybe in this file, but probably should be clarified in the requirements.

For GDPR purposes (and other legal/privacy regulations), as well as for statisticaly reasons, any subsetting by rows in any resource will lead to the observational units (participants if human) being the same across all resources. For example, if in survey a condition is Gender == "Man", and in another resource biometrics that doesn't have a condition, the participant IDs from the subsetted survey will be applied to the biometrics. So that all resources contain the same set of participants. This is a legal thing, as much as a statistical thing. It's to ensure that only the necessary data is given to a researcher that needs it. The principle of "least permissions" but in data context 😛

So, in your example, that won't be allowed to request from the same resource more than once. But actually, your example highlights that they won't be given everything when they only need a specific amount (e.g. the ask for 10 rows and only get images for those 10 rows).

subsets can be specified in the same request by repeating the `resource` key.
This can be done either with the same resource multiple times or with
different resources for each.
- `columns`: The columns to include. Each list item must correspond to an exact

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think with @martonvago comment about mirroring the web app/tui interface and how it would actually be implememted, it makes sense that no columns selected means all columns requested from the resource (which maybe? mimics how we would do it in the app..?). So yea, let's do that, as @signekb says, it also has some symmetry.

<!-- We discussed elsewhere that one purpose of this file is for someone to get an overview of which data is used in a specific study. On the one side, that could benefit from having to specify all the columns if the study used all of them and that an empty list means nothing rather than everything. However, having an empty list mean "everything" would make cases were less than all columns were selected since this would change from empty to a list. -->
<!-- For rows, it seems like the natural is that empty = all rows. So we can also consider if we want "empty" to mean the same thing (ie "all") both for columns and rows. -->

#### Row condition alternatives

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm fine with your suggestions @joelostblom. These types of comments should be put into the doc (they are design decisions). Can you modify and add it to it?

As for 6. technically they are row "keeps", as the condition keeps rows, it doesn't filter them (filter means remove, not keep). Just a small semantic comment 😛

Comment on lines +308 to +311
Propagate is not intended to be a general-purpose query engine. The initial
request format should not support joins, grouping, arbitrary functions, random
sampling, or row selection by position. We also don't support shorthands such as `*` since that would make it less
explicit which columns are used from just looking at the request file without

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this paragraph should be at the top as a callout block.

@github-project-automation github-project-automation Bot moved this from In Review to In Progress in Product development Jul 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: In Progress

Development

Successfully merging this pull request may close these issues.

Write out instruction design doc Brainstorm instruction design

4 participants