[Incremental Reprocessing] [POC] Prefiltering

[rkistner]

This is a throwaway PR to do proof-of-concept pre-filtering when replicating tables with filters. This POC is for MongoDB only.

For example, if a bucket definition data query is select _id as id * from todos where user_id = bucket.user_id and archived = false, this extracts a replication filter of {$expr: {$in:["$archived",[0,false]]}.

The pre-filters only cover some basic operators, and the fallback is to replicate all data from the referenced collection.

Current limitations:

• Only data queries are filtered.
• There is some unnecessary duplication if the same collection/table appears multiple times with different filters - those are treated as separate tables, instead of one table with separate filters.
• The "SQLite" type is passed to the filter expression, but the MongoDB type is not converted. This PR currently has a workaround for booleans, but implementing a general solution here can be difficult. We'd need to do some type coercion to cater for this, which may make efficient index usage difficult.
  • It's possible that this becomes a blocker for practical usage. In that case, we may need to investigate the option to explicitly write queries in the source database syntax, instead of auto-inferring these from the synced queries.
• The filters are only applied on the snapshot queries, not on the changestream pipeline. In theory it could work on the changestream pipeline as well, but I haven't had luck with that yet.
• No way to opt-out currently. So no way to turn this off if the pre-filter behavior is broken, or if a missing index in the source database causes performance issues.

The actual query -> pre-filter computation is done as a quick hack. A production implementation should probably be based on the new compiler in #451 instead.

[changeset-bot]

⚠️ No Changeset found

Latest commit: f84f277

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[rkistner]

After experimenting a little with this, I don't think the approach of auto-computing filters from the sync queries is feasible.

The basic issue is that if we have a sync rule query with a filter WHERE value = ?, it is effectively WHERE toSqliteValue(value) = ?, where toSqliteValue performs the type conversions documented here. And unfortunately:

1. We cannot compute toSqliteValue in the MongoDB query. May technically be possible in some source databases such as Postgres.
2. Even if we could compute it in the query, that would prevent is from utilizing efficient indexes, thus negating most performance benefits we could get from pre-filtering the data.

We could instead do the inverse: do WHERE value IN sqliteToSourceValue(?). However:

1. A single SQLite value may have may different source values matching it. The exact number depends on the specific value. For example, [1] could be a text "[1]", an array [Int32(1)], or an array [Long(1)]. null may be null, undefined, MinKey, MaxKey, or potentially a date outside our supported time range.
2. In some cases even an indefinite number of source values may match, for example if the value is a date with reduced precision.
3. Implementing the exact conversions could be tricky.
4. If we consider operators other than =, this gets even more tricky.

Theoretically we could implement support for only a strict subset of values, e.g. value = 0 and value = 1. Even then, the queries could appear complicated, and it would be difficult to explain to users what is or isn't supported.

Other options

So instead, I propose that we allow the user to specify pre-filters in the source query language of the database. For example, we can have this in sync rules for mongodb:

source_tables:
  todos:
    filter: {archived: {$eq: false}}

We'd still need some work if we want the same query to work in both the change stream pipeline and in snapshot queries, but that should be doable.

This approach gives the user direct control over the query, which makes it easy to optimize (e.g. creating relevant indexes), and easy to see what is happening, without much complexity on the PowerSync side. The main disadvantage is having some duplication of logic.

For Postgres, there is also an option that does not require configuration on the PowerSync side at all:

1. Logical replication row filters filter what is passed through on the logical replication stream.
2. RLS can be used to filter rows passed through on the initial snapshot queries.
3. We could also read the row filters and apply them automatically on snapshot queries, avoiding the need for RLS (this is what Postgres itself does when doing logical replication).