Enable multiple event listers to consume for Polaris events by nandorKollar · Pull Request #3973 · apache/polaris

nandorKollar · 2026-03-11T13:43:10Z

This PR enables multiple event listeners watching for Polaris events.

Checklist

🛡️ Don't disclose security issues! (contact security@apache.org)
🔗 Clearly explained why the changes are needed, or linked related issues: Fixes #
🧪 Added/updated tests with good coverage, or manually tested (and explained how)
💡 Added comments for complex logic
🧾 Updated CHANGELOG.md (if needed)
📚 Updated documentation in site/content/in-dev/unreleased (if needed)

nandorKollar · 2026-03-11T13:50:33Z

@jbonofre @adutra @adnanhemani would you mind share your thoughts about this approach?

adutra

@nandorKollar this is look promising thanks!

I think we should avoid calling dispatch() altogether (and avoid materializing the PolarisEvent instance on the heap) if we know the event type is not needed. I would like us to consider that, if a user deactivates all listeners, they would get a performance comparable to what they would get if there were no events in Polaris at all.

...ache/polaris/service/events/jsonEventListener/aws/cloudwatch/AwsCloudWatchEventListener.java

...rvice/src/main/java/org/apache/polaris/service/events/PolarisEventListenerConfiguration.java

runtime/service/src/main/java/org/apache/polaris/service/events/PolarisEventListeners.java

jbonofre · 2026-03-15T10:48:32Z

runtime/service/src/main/java/org/apache/polaris/service/events/PolarisEventListeners.java

+  public void onStartup(@Observes StartupEvent event) {
+    String listerTypes = configuration.types().orElse("");
+    if (listerTypes.isBlank()) {
+      return;


If we don't have types here, it means we never reach type() merging bellow (line 51).

If a user only set polaris.event-listener.type (the old config) without setting types, the onStartup() method just returns directly, and no listener is registered.

It means it's a breaking change.

Imho, the type() merge logic should be before the early return, or this check should check both types and type (to be backward compatible).

Good point, thanks! This is indeed broken, fixed in 1a0aaa5

adutra · 2026-03-16T11:13:13Z

...rvice/src/main/java/org/apache/polaris/service/events/PolarisEventListenerConfiguration.java

   * identifier.
   */
-  String type();
+  Optional<String> type();


We don't need this one anymore, right?

I think we need it to read the old config: polaris.event-listener.type. It is optional to allow one to omit. The value of this config (if present), and the new one polaris.event-listener.types should be merged. We should also indicate that polaris.event-listener.type is deprecated, should not be used.

Hmm indeed our evolution guidelines considers changes to the configuration as breaking changes: https://polaris.apache.org/releases/1.3.0/evolution/

So, I think we indeed need to deprecate this config but keep it around for some time.

How can we mark it deprecated?

You should:

Annotate the method with @Deprecated

Add a note to the changelog.

We could probably add a config redirect too.

Cf.

polaris/persistence/nosql/persistence/cdi/quarkus/src/main/java/org/apache/polaris/persistence/nosql/quarkus/backend/MongoDBConfigSourceFactory.java

Line 45 in 0872093

static final String ACTIVE_PROPERTY = "quarkus.mongodb.active";

WDYT about config redirects? My code pointer above may not be an exact match for this use case, but I hope it can be a starting point 🙂

Well, I’m trying to understand how we can use that. What would be the benefit of a config redirect? The old property is already mapped to the new one, I think. Would ConfigSourceFactory make this mapping cleaner, or is there a specific scenario that wouldn’t work with the current approach?

Where is the old config mapped to the new one (I might have missed it)?

What I mean is that ideally we should treat the old polaris.event-listener.type values as a single list entry for the new polaris.event-listener.types config (if the new one is not set explicitly). The no-op old type value can be ignored (mapped to empty list).

adutra · 2026-03-16T11:20:58Z

...rvice/src/main/java/org/apache/polaris/service/events/PolarisEventListenerConfiguration.java

+   * Comma separated list of event listers, each item must be a registered {@link
+   * PolarisEventListener} identifier.
+   */
+  Optional<String> types();


Suggested change

Optional<String> types();

Optional<List<String>> types();

It's a bit odd to see an Optional<List<...> – but that's how SmallRye Config wants it 😅

Update: Optional<Set<String>> types(); is probably better.

nandorKollar · 2026-03-16T23:05:44Z

runtime/defaults/src/main/resources/application.properties


 polaris.file-io.type=default

-polaris.event-listener.type=no-op


As there's no need to set this property to no-op, should we also delete NoOpPolarisEventListener? It will be a breaking change, because anyone who still refers to no-op with the old polaris.event-listener.type (or even with the new polaris.event-listener.types) will hit the following error: No bean found for required type [interface org.apache.polaris.service.events.listeners.PolarisEventListener] and qualifiers [[@jakarta.enterprise.inject.Any(), @io.smallrye.common.annotation.Identifier(value="no-op")]]

We can mark it deprecated though.

Yes, I think we could delete the no-op listener. Regarding breaking changes, our evolution guidelines accept that: https://polaris.apache.org/releases/1.3.0/evolution/

…tomatically by smallrye

nandorKollar · 2026-03-17T07:22:32Z

@nandorKollar this is look promising thanks!

I think we should avoid calling dispatch() altogether (and avoid materializing the PolarisEvent instance on the heap) if we know the event type is not needed. I would like us to consider that, if a user deactivates all listeners, they would get a performance comparable to what they would get if there were no events in Polaris at all.

Thanks, this makes sense, however I've further questions related to this. Do you happen to know why does AwsCloudWatchEventListener filter only for a single event type? I think event listeners should accept any event type, and allow users to configure a particular event listener to listen for only a certain events, but the listener code itself should not assume that it is working only with event type xyz.

adutra · 2026-03-17T10:09:04Z

Do you happen to know why does AwsCloudWatchEventListener filter only for a single event type? I think event listeners should accept any event type, and allow users to configure a particular event listener to listen for only a certain events, but the listener code itself should not assume that it is working only with event type xyz.

Completely agree here. Listeners should NOT filter events, and especially not in a hard-coded fashion – that's a user decision. @adnanhemani may know why the CoudWatch listener was designed that way. Imho it's not very useful – it can only be useful if you are interested in after-refresh-table events, and hence, in certain table optimization use cases.

runtime/service/src/main/java/org/apache/polaris/service/events/PolarisEventListeners.java

nandorKollar · 2026-03-18T08:38:08Z

Do you happen to know why does AwsCloudWatchEventListener filter only for a single event type? I think event listeners should accept any event type, and allow users to configure a particular event listener to listen for only a certain events, but the listener code itself should not assume that it is working only with event type xyz.

Completely agree here. Listeners should NOT filter events, and especially not in a hard-coded fashion – that's a user decision. @adnanhemani may know why the CoudWatch listener was designed that way. Imho it's not very useful – it can only be useful if you are interested in after-refresh-table events, and hence, in certain table optimization use cases.

@adutra How about introducing the event listener configuration in a follow-up PR, together with the optimization you mentioned earlier (i.e., avoiding materializing the PolarisEvent)?

In that PR, we could address the following points:

Make each event listener configurable. For example, CloudWatch could be configured to receive only a subset of events, while at the same time every event is also persisted in the database.
Ensure the event dispatcher only emits events when there is a registered listener for the given event type. This was partially implemented in Create PolarisEvent only when event type is actually supported by listener #3442 — we could revive that work.
Avoid assumptions in the event listener implementation about handling only a specific event type.

What do you think?

nandorKollar · 2026-03-18T08:40:44Z

Oh and we shouldn't forget #2573 either. We need to find a way to trigger event cleanup, or even set retention period for events persisted in the database.

adutra · 2026-03-18T14:40:00Z

@adutra How about introducing the event listener configuration in a follow-up PR, together with the optimization you mentioned earlier (i.e., avoiding materializing the PolarisEvent)?

In that PR, we could address the following points:

Make each event listener configurable. For example, CloudWatch could be configured to receive only a subset of events, while at the same time every event is also persisted in the database.

Ensure the event dispatcher only emits events when there is a registered listener for the given event type. This was partially implemented in Create PolarisEvent only when event type is actually supported by listener #3442 — we could revive that work.

Avoid assumptions in the event listener implementation about handling only a specific event type.

What do you think?

I'm all in for breaking the work in many smaller PRs, but I'm a little confused: you said you would like to introduce the event listener configuration in a follow-up PR – but you also wrote you would like to make each event listener configurable in this PR. Could you clarify this small item?

nandorKollar · 2026-03-18T16:09:40Z

@adutra How about introducing the event listener configuration in a follow-up PR, together with the optimization you mentioned earlier (i.e., avoiding materializing the PolarisEvent)?
In that PR, we could address the following points:

Make each event listener configurable. For example, CloudWatch could be configured to receive only a subset of events, while at the same time every event is also persisted in the database.

Ensure the event dispatcher only emits events when there is a registered listener for the given event type. This was partially implemented in Create PolarisEvent only when event type is actually supported by listener #3442 — we could revive that work.

Avoid assumptions in the event listener implementation about handling only a specific event type.

What do you think?

I'm all in for breaking the work in many smaller PRs, but I'm a little confused: you said you would like to introduce the event listener configuration in a follow-up PR – but you also wrote you would like to make each event listener configurable in this PR. Could you clarify this small item?

I recommend to target only focus on making multiple event listeners work in this PR, and rather implement the rest in a second PR. The only configuration parameter for now is deprecating polaris.event-listener.type and adding polaris.event-listener.types. Of course if you think we should avoid event materialization in this PR, I can continue working on that in this change too.

adutra · 2026-03-18T17:31:26Z

I recommend to target only focus on making multiple event listeners work in this PR, and rather implement the rest in a second PR. The only configuration parameter for now is deprecating polaris.event-listener.type and adding polaris.event-listener.types. Of course if you think we should avoid event materialization in this PR, I can continue working on that in this change too.

That works for me. I think this PR is good to go then, but it would still be nice to add a note to the changelog about the deprecation of polaris.event-listener.type.

nandorKollar · 2026-03-18T18:41:12Z

I recommend to target only focus on making multiple event listeners work in this PR, and rather implement the rest in a second PR. The only configuration parameter for now is deprecating polaris.event-listener.type and adding polaris.event-listener.types. Of course if you think we should avoid event materialization in this PR, I can continue working on that in this change too.

That works for me. I think this PR is good to go then, but it would still be nice to add a note to the changelog about the deprecation of polaris.event-listener.type.

Should we add it only to the changelog?

EDIT: never mind, found the way to deprecate it in the doc too. :)

dimas-b

Sorry about last minute comments. The PR LGTM overall. My only suggestion is to try and add automatic redirects for the old config name.

dimas-b · 2026-03-19T23:39:20Z

runtime/service/src/main/java/org/apache/polaris/service/events/EventBusConfigurer.java

+
+public class EventBusConfigurer {
+  private static final String LOCAL_CODEC_NAME = "local";
+  private static final LocalEventBusCodec<PolarisEvent> LOCAL_CODEC =


nit: is this constant really required?

dimas-b · 2026-03-19T23:46:26Z

...rvice/src/main/java/org/apache/polaris/service/events/PolarisEventListenerConfiguration.java

   * identifier.
   */
-  String type();
+  Optional<String> type();


We could probably add a config redirect too.

Cf.

polaris/persistence/nosql/persistence/cdi/quarkus/src/main/java/org/apache/polaris/persistence/nosql/quarkus/backend/MongoDBConfigSourceFactory.java

Line 45 in 0872093

static final String ACTIVE_PROPERTY = "quarkus.mongodb.active";

dimas-b · 2026-03-19T23:47:51Z

...ervice/src/main/java/org/apache/polaris/service/events/PolarisServiceBusEventDispatcher.java

+
+  @Inject EventBus eventBus;
+
+  //  @Inject PolarisEventListeners polarisEventListeners;


nit: leftover?

dimas-b · 2026-03-19T23:50:30Z

...ice/src/test/java/org/apache/polaris/service/catalog/iceberg/AbstractIcebergCatalogTest.java

    this.catalog = initCatalog("my-catalog", ImmutableMap.of());
-    testPolarisEventListener = (TestPolarisEventListener) polarisEventListener;
+    testPolarisEventListener =
+        (TestPolarisEventListener) polarisEventListener.select(Identifier.Literal.of("test")).get();


Would it work if we had @Identifier on the polarisEventListener field without Intance<>?

Yes, looks like it works! Thanks, great recommendation!

dimas-b · 2026-03-19T23:52:38Z

...estFixtures/java/org/apache/polaris/service/events/listeners/TestPolarisEventDispatcher.java

-        "relational-jdbc",
-        "polaris.persistence.relational.jdbc.database-type",
-        "cockroachdb");
+public class TestPolarisEventDispatcher implements PolarisEventDispatcher {


nit: the Test prefix makes this class look like a JUnit test class, which it is not. How about InMemoryEventCollector?

InMemoryEventCollector suggests that it can be used outside of tests too no? This class is indented to be used only in tests though.

The name describes what the class does 🤷 Where it can be used is controlled by the location of the class in the sources tree (ATM under /test).

If we (hypothetically) wanted to expose this functionality to mainstream code, we'd only have to move the class without renaming it.

…er_service_bus # Conflicts: # runtime/service/src/main/java/org/apache/polaris/service/config/ServiceProducers.java

github-project-automation bot added this to Basic Kanban Board Mar 11, 2026

github-project-automation bot moved this to PRs In Progress in Basic Kanban Board Mar 11, 2026

jbonofre self-requested a review March 11, 2026 13:58

nandorKollar force-pushed the multiple_event_listener_service_bus branch 7 times, most recently from d0242ed to 09c5600 Compare March 11, 2026 19:59

Enable multiple event listers to consume for Polaris events

aad3caf

nandorKollar force-pushed the multiple_event_listener_service_bus branch from 09c5600 to aad3caf Compare March 11, 2026 20:13

adutra mentioned this pull request Mar 13, 2026

(Proposal) REST API for Events #3924

Open

6 tasks

adutra reviewed Mar 13, 2026

View reviewed changes

nandorKollar added 2 commits March 14, 2026 20:41

Rewrite listener creation logic + cleanup

9389c01

Backward compatibility: take old listener config into account

1c71676

nandorKollar force-pushed the multiple_event_listener_service_bus branch from 8b05fcf to 1c71676 Compare March 14, 2026 20:20

adjust doc

dd17149

jbonofre reviewed Mar 15, 2026

View reviewed changes

Fix backward compatibility

1a0aaa5

adutra reviewed Mar 16, 2026

View reviewed changes

Fix tests

c9f843b

nandorKollar force-pushed the multiple_event_listener_service_bus branch from 48b5fbd to c9f843b Compare March 16, 2026 22:43

nandorKollar commented Mar 16, 2026

View reviewed changes

Simplify parsing of types config: no need to split, it can be done au…

716f6db

…tomatically by smallrye

nandorKollar force-pushed the multiple_event_listener_service_bus branch from ee21f22 to 716f6db Compare March 16, 2026 23:13

nandorKollar marked this pull request as ready for review March 17, 2026 07:22

adutra reviewed Mar 17, 2026

View reviewed changes

runtime/service/src/main/java/org/apache/polaris/service/events/PolarisEventListeners.java Outdated Show resolved Hide resolved

nandorKollar added 2 commits March 17, 2026 12:43

Address code review comment: concat streams

4d5b172

Delete no-op event listener

9fe8851

adutra previously approved these changes Mar 18, 2026

View reviewed changes

github-project-automation bot moved this from PRs In Progress to Ready to merge in Basic Kanban Board Mar 18, 2026

Document deprecation of 'polaris.event-listener.type'

ce9addd

nandorKollar dismissed adutra’s stale review via ce9addd March 18, 2026 19:18

nandorKollar requested review from adutra and jbonofre March 19, 2026 18:28

dimas-b reviewed Mar 19, 2026

View reviewed changes

Address code review comments

b570a2e

nandorKollar force-pushed the multiple_event_listener_service_bus branch from 16170de to b570a2e Compare March 20, 2026 13:16

Merge remote-tracking branch 'origin/main' into multiple_event_listen…

0c39bd6

…er_service_bus # Conflicts: # runtime/service/src/main/java/org/apache/polaris/service/config/ServiceProducers.java


		polaris.file-io.type=default

		polaris.event-listener.type=no-op


		@Inject EventBus eventBus;

		// @Inject PolarisEventListeners polarisEventListeners;

Conversation

nandorKollar commented Mar 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

Uh oh!

nandorKollar commented Mar 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

adutra left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nandorKollar Mar 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nandorKollar commented Mar 17, 2026

Uh oh!

adutra commented Mar 17, 2026

Uh oh!

Uh oh!

nandorKollar commented Mar 18, 2026

Uh oh!

nandorKollar commented Mar 18, 2026

Uh oh!

adutra commented Mar 18, 2026

Uh oh!

nandorKollar commented Mar 18, 2026

Uh oh!

adutra commented Mar 18, 2026

Uh oh!

nandorKollar commented Mar 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dimas-b left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

nandorKollar commented Mar 11, 2026 •

edited

Loading

nandorKollar commented Mar 11, 2026 •

edited

Loading

nandorKollar Mar 16, 2026 •

edited

Loading

nandorKollar commented Mar 18, 2026 •

edited

Loading

dimas-b Mar 20, 2026 •

edited

Loading