Hierarchical Filtering

[cicdguy]

hierarchical_filtering = list(
  PARAM = c("AVAL", "AVALC", "CHNG"),
  ADY = c("AVAL")
)

Provenance:

Creator: waddella

[cicdguy]

@palangaa & @npaszty & @martik32 will be happy to see that

Provenance:

Creator: waddella

[cicdguy]

happy to test when needed.

Provenance:

Creator: npaszty

[cicdguy]

What would be the behaviour?

Provenance:

Creator: kpagacz

[cicdguy]

@kpagacz

hierarchical filtering is where the parent filter controls the record set that is available to the child filter.
Using our RCD...

here's the full set
user/1100/files/ddab3980-92c9-11eb-9741-60b71f7da8f8)

here's the filtered set on SEX == "F" note that the AGE range on the filter is incorrect and should be 64 not 69. This is a very simple example hierarchical filtering is typical behavior one would expect.
user/1100/files/06333380-92ca-11eb-997b-61384107630d)

Provenance:

Creator: npaszty

[gogonzo]

I don't think hierarchical filtering is worth considering technical difficulties behind. I'd like to express my scepticism about that, and I'll be short.
Let's consider simple case where we have three filters.

• filter 1
• filter 2
  • filter 3

According to the vision set long time ago, filter-3 should be dependent on the data filtered by the filter-2. At the same moment filter-2 and filter-3 depends on unfiltered data.
It means that for filter-3 we need an extra filter operation. In the filter panel, extra filter operation requires creation of an extra filter-call before. It's not a big issue if we look at this separately, but it's another reactive-data (cached) which takes some computation power (concern shared with @nikolas-burkoff ).
In more complicated scenario filter-2 depends on filter-1, so to generate filter-3 we need to generate three filtered data

• filter 1: tmp1 <- filter(unfiltered, filter 1)
• filter 2: temp2 <- filter(tmp1, filter 2)
  • filter 3: temp3 <- filter(tmp2, filter 3)

You know what's the ugliest in the hierarchy - that to generate each temporary filter we execute only one filter condition, but when return data from filter-panel we have everything combined so that filtered <- filter(unfiltered, filter 1 & filter 2 & filter 3)

---

I suggest to close this issue and rather apply #131 and forget about the hierarchical.

@pawelru @donyunardi @lcd2yyz

[nikolas-burkoff]

@npaszty ?

Although to be fair #131 does effectively perform very similar functionality

[npaszty]

@gogonzo @nikolas-burkoff

apologies for the delay. just back from US holiday.

also not seeing that this is terribly different than #131. if the data filtering outcome is the same, basically an "and" condition, then happy to go with whatever the best approach is. the key is that subsequent filters added only display data ranges that are available as set by the previous filter.

filter0: SAFFL = "Y" --> produces dataset with only safety population records. all other subsequent filters only display data that are within the range based on this data set.
filter1: SEX = "M" --> produces dataset for only male subjects in the safety population from filter0. all other subsequent filters only display data that are within the range based on this data set.
filter2: PARAMCD = "CRP" --> produces dataset with only CRP records for male subjects in safety population from filter1. all other subsequent filters only display data that are within the range based on this data set.
filter3: range of AVAL values only includes values for male subjects in safety population with CRP from the filter2 dataset --> produces a dataset with only the selected range of CRP values.

if filter1 is removed/changed then that would affect the data ranges displayed in any still applied- filters subsequent to filter1. so filter2 above effectively becomes filter1 and the cascade of data ranges now affects subsequent filter ranges displayed.

[gogonzo]

the key is that subsequent filters added only display data ranges that are available as set by the previous filter.

all other subsequent filters only display data that are within the range based on this data set.

if filter1 is removed/changed then that would affect the data ranges displayed in any still applied- filters subsequent to filter1

@npaszty yes but also prior filters will display data ranged based on following filters:

So if you're adding filter0, filter1, ... each time new added filter will be based on the previously added filters - TRUE. But also prior filters will be updated if something is added after them. The same with removing filters - if one is removed, everything else would be updated.
It's worth to mention that selected values won't be updated - if we have a range selected it will stay on the current values but the distribution to visualize what's available in the data would change.

[npaszty]

@gogonzo

not sure about the prior filter being updated if a subsequent one is added. you won't get that situation under the hierarchical filtering concept.

in the example I provided a subsequent filter would not all of a sudden introduce SAFFL = "N" records or some other SEX or PARAMCD value.

filter 1 --> data 1 of 100 records
filter 2 --> data 2 (subset of data 1) of 75 records
filter 3 --> data 3 (subset of data 1 and data 2) of 25 records
etc.

data 3 would only have records that are available in data 2 and data 1 so in my example still SAFFL = "Y" and SEX = "M". etc. and so on down the hierarchical filter tree.

maybe I misunderstood what you meant.
not sure about the visualization description. should just include data that are in the last level of the hierarchy so data x.
maybe we need a f2f discussion. ;-)

[gogonzo]

@npaszty I understand everything. I understand the concept of the hierarchical filters.
What I'm proposing in this "Hierarchical Filtering" issue is alternative solution to the one explained by you - to just update all filters according to single filtered-data (being output of combining all filters).
So my proposition isn't exactly hierarchical filter but all filters being responsive of any other filters.

Proper hierarchical filtering has some technical pitfalls from the teal perspective

[npaszty]

@gogonzo
okay. I guess easiest is to be able to play with a simple prototype when it's ready. ;-)