JBTM-845 Include the Infinispan backed slot store in the object store… by mmusgrov · Pull Request #204 · jbosstm/performance

mmusgrov · 2026-04-24T14:39:52Z

NARAYANA_REPO=mmusgrov
NARAYANA_BRANCH=JBTM-845
NY_BRANCH=JBTM-845
THREAD_COUNTS=1 24 240

github seems unable to support 1600 threads

marcosgopen · 2026-04-27T08:15:52Z

@mmusgrov the job is failing because the expected number or store benchmark is now 6 instead of 4:
we should update line

performance/narayana/scripts/hudson/benchmark.sh

Line 112 in 936102a

BM5="ArjunaJTA/jta com.arjuna.ats.jta.xa.performance.*StoreBenchmark.* 4"

I think this check is quite counterintuitive but to change 4 to 6 is the fastest fix.

P.S. InfinispanSlotsStoreRestartBenchmark is not executed by the script because it ends with '*StoreRestartBenchmark' instead of '*StoreBenchmark'. If you want to include that too maybe the configuration number i mentioned above needs to be changed to 7.

mmusgrov · 2026-04-27T14:13:49Z

@mmusgrov the job is failing because the expected number or store benchmark is now 6 instead of 4: we should update line

performance/narayana/scripts/hudson/benchmark.sh

Line 112 in 936102a

BM5="ArjunaJTA/jta com.arjuna.ats.jta.xa.performance.*StoreBenchmark.* 4"

I think this check is quite counterintuitive but to change 4 to 6 is the fastest fix.

P.S. InfinispanSlotsStoreRestartBenchmark is not executed by the script because it ends with '*StoreRestartBenchmark' instead of '*StoreBenchmark'. If you want to include that too maybe the configuration number i mentioned above needs to be changed to 7.

Well spotted. I went for 6 since InfinispanSlotsStoreBase is a base class for the others.

I agree this script is quite/very counterintuitive, I've tripped up over it before in other parts too.

… tests

mmusgrov · 2026-04-28T14:40:05Z

  let tc=$(wc -l < ${CSVF})
  let tc=tc-1 # subtract 1 to account for the header
-  if [ $tc != $3 ]; then
+  if [ $tc -lt $3 ]; then


I changed it to less than the expected number since that still implies at least one benchmark failed to finish

This is another example of how non-intuitive/difficult to parse the script is. I ran it on my laptop and got:

Some benchmark tests did not finish. Expected: 4 Actual: 7

As you [Marco] mentioned elsewhere the pattern for the store benchmarks:

BM5="ArjunaJTA/jta com.arjuna.ats.jta.xa.performance.*StoreBenchmark.* 6"

had been incorrect but I couldn't figure out where the value 7 came from, lol!

We used to have 4 store benchmarks, This PR is adding 3 InfiniSpan benchmarks. I think this is it.

I think BM5="ArjunaJTA/jta com.arjuna.ats.jta.xa.performance.StoreBenchmark. 7" should work, but I haven't checked.

Ah you are correct, I forgot that I also added DiskSlotsStoreBenchmark.
I put the value as 6 since I have removed the InfinispanSlotsRestartStoreBenchmark.

marcosgopen · 2026-04-29T08:32:29Z

I can see an interesting error in the infinispan restart benchmark: "Suppressed: org.infinispan.persistence.spi.PersistenceException: ISPN029025: Failed acquiring lock 'FileSystemLock{directory=null/infinispan-caches/node1/data/objectStoreCluster, file=objectStoreCluster.lck}' for SIFS"

mmusgrov · 2026-04-29T08:43:56Z

I can see an interesting error in the infinispan restart benchmark: "Suppressed: org.infinispan.persistence.spi.PersistenceException: ISPN029025: Failed acquiring lock 'FileSystemLock{directory=null/infinispan-caches/node1/data/objectStoreCluster, file=objectStoreCluster.lck}' for SIFS"

I've seen that one before, I'll investigate. I'll also modify the benchmark to see if we can fail it early (it's been running for 3 hrs.

mmusgrov · 2026-04-29T08:51:18Z

I also noticed many overlapping classes in the uber jar used for the benchmarking such as

arjunacore-7.3.4.Final-SNAPSHOT.jar define 546 overlapping classes

and also

Discovered module-info.class. Shading will break its strong encapsulation

Not necessarily introduced in this PR but I'll investigate those too.

marcosgopen · 2026-04-29T09:00:51Z

Thanks @mmusgrov

mmusgrov · 2026-04-30T07:40:03Z

I've put this on Hold while I investigate "ARJUNA016053: Could not commit transaction".

…H Action)

mmusgrov · 2026-04-30T15:47:03Z

I've put this on Hold while I investigate "ARJUNA016053: Could not commit transaction".

The failures were caused by the run that used 1600 threads which we know fails on when running on the github infrastructure (see pull/205). I fixed it by removing the run with that many threads, ie I set the env var in the PR description:

THREAD_COUNTS=1 24 240

mmusgrov · 2026-04-30T16:33:00Z

I can see an interesting error in the infinispan restart benchmark: "Suppressed: org.infinispan.persistence.spi.PersistenceException: ISPN029025: Failed acquiring lock 'FileSystemLock{directory=null/infinispan-caches/node1/data/objectStoreCluster, file=objectStoreCluster.lck}' for SIFS"

The benchmark that produces that error is InfinispanSlotsRestartStoreBenchmark (which I based on the test InfinispanSlotsRestartStoreBenchmark from the main narayana repo) and it uses write through caching backed by the filesystem and it stops and restarts an infinispan store. So when different JMH benchmark threads try to start stores backed by the same file system location it can find the lock held, ie I don't think that testing the performance of restarts, using JMH, is particularly good benchmark so I have deleted that particular benchmark.

marcosgopen

Thanks @mmusgrov

jmfinelli · 2026-05-06T12:42:36Z

It looks like the DiskSlotsStoreBenchmark is failing at 1600 threads even in my local environment. This suggests the issue is likely independent of the GitHub infrastructure.

cc @mmusgrov, @marcosgopen

marcosgopen · 2026-05-06T13:48:58Z

It looks like the DiskSlotsStoreBenchmark is failing at 1600 threads even in my local environment. This suggests the issue is likely independent of the GitHub infrastructure.

cc @mmusgrov, @marcosgopen

Well spotted @jmfinelli . The SlotStore has a numberOfSlots configuration that must be configured correctly and the default configuration should be changed when running with 1600 threads. e.g. "configBean.setNumberOfSlots(1600);" Plus the disk read/write speed highly influences the results. May I suggest to disable this benchmark in the CI testing as with 1600 threads is quite unstable and keep it for reference? WDYT @mmusgrov ?

marcosgopen · 2026-05-06T14:01:06Z

If we want to keep the DiskSlotsStoreBenchmark to test the logic behind it (not the real scenario performance) we could configure it to run in memory so that we don't hit the disk bottleneck: i.e.

JTAStoreBase.setup(SlotStoreAdaptor.class.getName());
        SlotStoreEnvironmentBean configBean = BeanPopulator.getDefaultInstance(SlotStoreEnvironmentBean.class);

        configBean.setBackingSlotsClassName(DiskSlots.class.getName());
        configBean.setNumberOfSlots(2000); // tested with 1800 threads
        configBean.setStoreDir("/tmp/SlotStore"); // in memory, no disk

mmusgrov · 2026-05-06T14:08:54Z

It looks like the DiskSlotsStoreBenchmark is failing at 1600 threads even in my local environment. This suggests the issue is likely independent of the GitHub infrastructure.
cc @mmusgrov, @marcosgopen

Well spotted @jmfinelli . The SlotStore has a numberOfSlots configuration that must be configured correctly and the default configuration should be changed when running with 1600 threads. e.g. "configBean.setNumberOfSlots(1600);" Plus the disk read/write speed highly influences the results. May I suggest to disable this benchmark in the CI testing as with 1600 threads is quite unstable and keep it for reference? WDYT @mmusgrov ?

The point of including it was to compare with the Infinispan Slot store and the other store types, not to test the absolute throughput. Depending upon what failures you saw @jmfinelli I would suggest either reducing the number of threads or tuning the number of slots since there are restrictions on that: https://github.com/jbosstm/narayana/blob/7.3.3.Final/ArjunaCore/arjuna/classes/com/arjuna/ats/internal/arjuna/objectstore/slot/SlotStoreEnvironmentBean.java#L44

jmfinelli · 2026-05-06T14:19:45Z

Basically, it fails straight after it starts. The output of my local testing is similar to what happened here, even though I only tested 1600 threads and DiskSlotsStoreBenchmark

jmfinelli · 2026-05-06T14:35:18Z

Yep, configBean.setNumberOfSlots(2000) fixes the issue (at least locally). Thanks @marcosgopen and @mmusgrov

jmfinelli · 2026-05-06T14:47:54Z

@mmusgrov Since you’re the author of the Infinispan store, you probably have the best context to fix this bug. Would you like to open the PR, or should I handle it? Just let me know what works best for you!

mmusgrov · 2026-05-06T14:53:20Z

@mmusgrov Since you’re the author of the Infinispan store, you probably have the best context to fix this bug. Would you like to open the PR, or should I handle it? Just let me know what works best for you!

I'll fix it since there may be other issues.

mmusgrov · 2026-05-08T10:21:06Z

@jmfinelli I will push a fix later today in readiness for the weekend runs.

jmfinelli · 2026-05-08T10:24:25Z

@jmfinelli I will push a fix later today in readiness for the weekend runs.

Sure, no problem at all. Narayana has been already released so nu rush at all. I'm completing the last steps before sending the announcement. Thanks Mike

mmusgrov mentioned this pull request Apr 24, 2026

JBTM-845 Datagrid ObjectStore jbosstm/narayana#2537

Merged

mmusgrov force-pushed the JBTM-845 branch 3 times, most recently from 48e385f to bb9bdb5 Compare April 27, 2026 14:10

JBTM-845 Include the Infinispan backed slot store in the object store…

3535716

… tests

mmusgrov force-pushed the JBTM-845 branch from bb9bdb5 to fea6f3b Compare April 28, 2026 14:00

mmusgrov mentioned this pull request Apr 28, 2026

JBTM-845 include infinispan store in the store quickstart jbosstm/quickstart#629

Merged

mmusgrov commented Apr 28, 2026

View reviewed changes

mmusgrov added the Hold label Apr 30, 2026

mmusgrov added 4 commits April 30, 2026 11:42

JBTM-845 Tweak script support for benchmarking LRA PR's (script and G…

086b5f8

…H Action)

JBTM-845 Make the test for incomplete benchmarks slightly less brittle

05bf324

JBTM-845 reduce shade plugin warnings

784c1f2

JBTM-845 throw an Error on benchmark failure

dd43674

mmusgrov force-pushed the JBTM-845 branch from ef27dae to dd43674 Compare April 30, 2026 10:44

mmusgrov removed the Hold label Apr 30, 2026

marcosgopen approved these changes May 4, 2026

View reviewed changes

mmusgrov merged commit 1e75aa2 into jbosstm:main May 4, 2026
4 of 6 checks passed

Uh oh!

Conversation

mmusgrov commented Apr 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

marcosgopen commented Apr 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mmusgrov commented Apr 27, 2026

Uh oh!

mmusgrov Apr 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mmusgrov Apr 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

marcosgopen Apr 28, 2026

Choose a reason for hiding this comment

Uh oh!

mmusgrov Apr 30, 2026

Choose a reason for hiding this comment

Uh oh!

marcosgopen commented Apr 29, 2026

Uh oh!

mmusgrov commented Apr 29, 2026

Uh oh!

mmusgrov commented Apr 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

marcosgopen commented Apr 29, 2026

Uh oh!

mmusgrov commented Apr 30, 2026

Uh oh!

mmusgrov commented Apr 30, 2026

Uh oh!

mmusgrov commented Apr 30, 2026

Uh oh!

marcosgopen left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jmfinelli commented May 6, 2026

Uh oh!

marcosgopen commented May 6, 2026

Uh oh!

marcosgopen commented May 6, 2026

Uh oh!

mmusgrov commented May 6, 2026

Uh oh!

jmfinelli commented May 6, 2026

Uh oh!

jmfinelli commented May 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jmfinelli commented May 6, 2026

Uh oh!

mmusgrov commented May 6, 2026

Uh oh!

mmusgrov commented May 8, 2026

Uh oh!

jmfinelli commented May 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mmusgrov commented Apr 24, 2026 •

edited

Loading

marcosgopen commented Apr 27, 2026 •

edited

Loading

mmusgrov Apr 28, 2026 •

edited

Loading

mmusgrov Apr 28, 2026 •

edited

Loading

mmusgrov commented Apr 29, 2026 •

edited

Loading

jmfinelli commented May 6, 2026 •

edited

Loading