support a staging bucket in bulk storage by isegall-da · Pull Request #6018 · canton-network/splice

isegall-da · 2026-06-17T20:34:18Z

Fixes #6162
(See #5884 for context)

Could benefit from some more unit testing of the update history part, but this PR is a behemoth as it is (sorry, couldn't break it in a very sensible way), so I'm leaving that as a followup PR (and hence not competing #6126 here) on closer look, the two pipelines share 99% of the code now, and I don't see significant coverage gaps actually.

Note that the changes are not bwd compatible, so require disabling bulk storage and deleting the data on CILR&DevNet where we enabled it (tracked in #5884)

Pull Request Checklist

Cluster Testing

If a cluster test is required, comment /cluster_test on this PR to request it, and ping someone with access to the DA-internal system to approve it.
If an upgrade test is required, comment /upgrade_test on this PR to request it, and ping someone with access to the DA-internal system to approve it.
If a hard-migration test is required (from the latest release), comment /hdm_test on this PR to request it, and ping someone with access to the DA-internal system to approve it.
If a logical synchronizer upgrade test is required (from canton-3.5), comment /lsu_test on this PR to request it, and ping someone with access to the DA-internal system to approve it.

PR Guidelines

Include any change that might be observable by our partners or affect their deployment in the release notes.
Specify fixed issues with Fixes #n, and mention issues worked on using #n
Include a screenshot for frontend-related PRs - see README or use your favorite screenshot tool

Merge Guidelines

Make the git commit message look sensible when squash-merging on GitHub (most likely: just copy your PR description).

Signed-off-by: Itai Segall <itai.segall@digitalasset.com>

…ging-2 Signed-off-by: Itai Segall <itai.segall@digitalasset.com>

Signed-off-by: Itai Segall <itai.segall@digitalasset.com>

isegall-da · 2026-07-01T13:38:44Z

@rautenrieth-da @ray-roestenburg-da I know this PR is a bit of a monster (sorry..), but can I get a review still please? 🙏

Signed-off-by: Itai Segall <itai.segall@digitalasset.com>

ray-roestenburg-da · 2026-07-02T10:05:01Z

+      automationConfig: AutomationConfig,
+      backoffClock: Clock,
+      retryProvider: RetryProvider,
+  )(implicit tracer: Tracer): PekkoRetryingService[S]


should this not return RetryingService[S]? or rename the trait to PekkoRetryableService

renamed the trait, and also the method.

ray-roestenburg-da

Thanks! looks good, I only had one minor question

ray-roestenburg-da · 2026-07-02T11:00:06Z

+      .sourceKey(key)
+      .build()
+
+    s3Client.copyObject(copyReq).asScala.map(_ => ())


Is unwrapCompletionException also useful here? (and in deleteObject)

Not sure what you mean, and in what way you think it would be useful.

ray-roestenburg-da · 2026-07-02T11:02:09Z

+          else { client.deleteObjects(deleteObjRequest).asScala }
+      } yield {
+        ()
+      }).futureValue


Looks like you are ignoring the future value?

Oh wait scalatest, so this will just fail, never mind

(maybe slightly better to map over the buckets and future sequence them and then get the futureValue on that)

ray-roestenburg-da · 2026-07-02T11:09:26Z

+    NotUsed,
+  ] =
+    Flow[(T, Seq[ObjectKeyAndChecksum])]
+      .mapAsync(parallelism = 1) { case (ts, objs) =>


nit: you could set a val in this class to 1 and pass it to all mapAsyncs here cause it's important that's always one. val noParallelism = 1 or something

I actually am not sure if all these parallelism = 1 are actually necessary, or we can tolerate parallelism. Left a TODO to revisit that. Clearly 1 is safest, so leaving as 1 for now.

ray-roestenburg-da · 2026-07-02T11:16:09Z

-        )
-        .filter(identity)
-        .take(1)
+//    // Wait for update history to initialize and for history backfilling to complete before starting bulk storage dumps


nit: Prefer no commented code, rather remove it.

whoops, for sure, this is just a leftover

Signed-off-by: Itai Segall <itai.segall@digitalasset.com>

isegall-da · 2026-07-03T01:16:44Z

(will merge on Monday, didn't yet merge the PR on cn-internal, and don't want to do that right before logging off for the long weekend)

isegall-da added 30 commits June 17, 2026 20:33

wip

993dd97

Signed-off-by: Itai Segall <itai.segall@digitalasset.com>

app compiles (not the tests yet)

263f1cd

Signed-off-by: Itai Segall <itai.segall@digitalasset.com>

.

6ffc231

Signed-off-by: Itai Segall <itai.segall@digitalasset.com>

[ci] compiles

98cfd16

Signed-off-by: Itai Segall <itai.segall@digitalasset.com>

untested pipeline implementation

14bc77b

Signed-off-by: Itai Segall <itai.segall@digitalasset.com>

now even compiles

c04f64a

Signed-off-by: Itai Segall <itai.segall@digitalasset.com>

[ci] refactor

b055165

Signed-off-by: Itai Segall <itai.segall@digitalasset.com>

[ci] cleanup

99c24c4

Signed-off-by: Itai Segall <itai.segall@digitalasset.com>

wip test

3a19412

Signed-off-by: Itai Segall <itai.segall@digitalasset.com>

[ci] slightly better test

f0870db

Signed-off-by: Itai Segall <itai.segall@digitalasset.com>

[ci] test 2

bd4630f

Signed-off-by: Itai Segall <itai.segall@digitalasset.com>

[ci] .

e63da16

Signed-off-by: Itai Segall <itai.segall@digitalasset.com>

[ci] start the services

207bedd

Signed-off-by: Itai Segall <itai.segall@digitalasset.com>

[ci] .

cdc5556

Signed-off-by: Itai Segall <itai.segall@digitalasset.com>

[ci] fix

30162f8

Signed-off-by: Itai Segall <itai.segall@digitalasset.com>

[ci] another fix

869818f

Signed-off-by: Itai Segall <itai.segall@digitalasset.com>

[ci] compiles again

3bdba06

Signed-off-by: Itai Segall <itai.segall@digitalasset.com>

[ci] start it

ac6708a

Signed-off-by: Itai Segall <itai.segall@digitalasset.com>

[ci] download from committed

e37c505

Signed-off-by: Itai Segall <itai.segall@digitalasset.com>

[ci] .

5bc6aab

Signed-off-by: Itai Segall <itai.segall@digitalasset.com>

[ci] fix unit test

08b4caa

Signed-off-by: Itai Segall <itai.segall@digitalasset.com>

[ci] .

ba36a93

Signed-off-by: Itai Segall <itai.segall@digitalasset.com>

[ci] the integration test passes

c648144

Signed-off-by: Itai Segall <itai.segall@digitalasset.com>

[ci] fix the unit test

f9aaf85

Signed-off-by: Itai Segall <itai.segall@digitalasset.com>

[ci] fixes

52594ec

Signed-off-by: Itai Segall <itai.segall@digitalasset.com>

[ci] more fixes

51fc86f

Signed-off-by: Itai Segall <itai.segall@digitalasset.com>

[ci] simplify the dependencies

9c9a88c

Signed-off-by: Itai Segall <itai.segall@digitalasset.com>

[ci] more refactoring

e1e0566

Signed-off-by: Itai Segall <itai.segall@digitalasset.com>

[ci] basic unit test works

35fadf5

Signed-off-by: Itai Segall <itai.segall@digitalasset.com>

[ci] a bit more testing

9b56573

Signed-off-by: Itai Segall <itai.segall@digitalasset.com>

isegall-da added 9 commits June 26, 2026 23:13

[ci] a bit more testing

274faff

Signed-off-by: Itai Segall <itai.segall@digitalasset.com>

[ci] cleanup

b276e37

Signed-off-by: Itai Segall <itai.segall@digitalasset.com>

[ci] more

a3f9df1

Signed-off-by: Itai Segall <itai.segall@digitalasset.com>

[ci] works

0f49687

Signed-off-by: Itai Segall <itai.segall@digitalasset.com>

[ci] more

9a45a13

Signed-off-by: Itai Segall <itai.segall@digitalasset.com>

[ci] cleanup

7f76582

Signed-off-by: Itai Segall <itai.segall@digitalasset.com>

[ci] fix update history test

ab6f23b

Signed-off-by: Itai Segall <itai.segall@digitalasset.com>

[ci] updateTestConfigForParallelRuns

d625c28

Signed-off-by: Itai Segall <itai.segall@digitalasset.com>

[ci] Merge remote-tracking branch 'origin/main' into isegall/bulk-sta…

bccb283

…ging-2 Signed-off-by: Itai Segall <itai.segall@digitalasset.com>

isegall-da requested review from rautenrieth-da and ray-roestenburg-da June 30, 2026 02:21

isegall-da changed the title ~~WIP: bulk staging~~ support a staging bucket in bulk storage Jun 30, 2026

isegall-da marked this pull request as ready for review June 30, 2026 02:22

isegall-da added 2 commits June 30, 2026 11:50

[ci] fix test assignment

baa6cf3

Signed-off-by: Itai Segall <itai.segall@digitalasset.com>

[ci] another test fix

2bff938

Signed-off-by: Itai Segall <itai.segall@digitalasset.com>

[ci] small fix

6aac37a

Signed-off-by: Itai Segall <itai.segall@digitalasset.com>

ray-roestenburg-da reviewed Jul 2, 2026

View reviewed changes

ray-roestenburg-da approved these changes Jul 2, 2026

View reviewed changes

ray-roestenburg-da reviewed Jul 2, 2026

View reviewed changes

[ci] review comments

586bc4a

Signed-off-by: Itai Segall <itai.segall@digitalasset.com>

Uh oh!

Conversation

isegall-da commented Jun 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Checklist

Cluster Testing

PR Guidelines

Merge Guidelines

Uh oh!

isegall-da commented Jul 1, 2026

Uh oh!

ray-roestenburg-da Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ray-roestenburg-da left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ray-roestenburg-da Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

isegall-da commented Jul 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

isegall-da commented Jun 17, 2026 •

edited

Loading

ray-roestenburg-da Jul 2, 2026 •

edited

Loading

ray-roestenburg-da Jul 2, 2026 •

edited

Loading