I/O imbalance across bookies

[Jayer23]

Discussed in #24009

^{Originally posted by Jayer23 February 20, 2025}
Description

We observed an abnormal write pattern in our Pulsar cluster:

The nodes exposed in bookie metrics have little difference in read and write io. Producer has compressed write enabled.

In fact, monitoring at the node level found that io was greatly magnified, and the io differences between nodes were large. Some nodes had io less than 50MB/s, while some nodes had io greater than 450MB/s.

Journal disks on some nodes show very high write throughput (e.g., 400+ MB/s).

Environment

• Pulsar version: 3.0.7
• Storage config:
  • Journal: 3×SSD
  • Ledger：9×HDD
• Replication: 3 replicas, 2 quorum writes

Expected Behavior

Write operations should be balanced between bookies

[thetumbled]

Apache Bookkeeper do not support load balance between bookies natively.
To mitigate the defect of unbalance, you can try this pr: apache/bookkeeper#4246, or implement your own load balance feature that is suitable to your specific enviroment.

[Shawyeok]

You probably have some naughty partitions with heavy write throughput, you can check by following Prometheus query:

pulsar_throughput_in{cluster="YOUR_CLUSTER_NAME"} > 5 * 1024^2

If you discover partitions with excessive write loads, consider increasing the number of partitions for the affected topics. This will help distribute the write throughput more evenly across your bookie instances.

Notes about Pulsar message publishing:

• Each partition maintains exactly one writable ledger at a time
• With configuration ensembleSize=2, writeQuorum=2:
  • A partition with 100MB/s write throughput will generate approximately 100MB/s write load on each bookie in the ensemble
  • Bookies not in the current ensemble will not receive any writes from this ledger

[Jayer23]

You probably have some naughty partitions with heavy write throughput, you can check by following Prometheus query:

pulsar_throughput_in{cluster="YOUR_CLUSTER_NAME"} > 5 * 1024^2

If you discover partitions with excessive write loads, consider increasing the number of partitions for the affected topics. This will help distribute the write throughput more evenly across your bookie instances.

Notes about Pulsar message publishing:

• Each partition maintains exactly one writable ledger at a time

• With configuration ensembleSize=2, writeQuorum=2:

  • A partition with 100MB/s write throughput will generate approximately 100MB/s write load on each bookie in the ensemble
  • Bookies not in the current ensemble will not receive any writes from this ledger

1. There are only 3 partitions in the cluster with write rates exceeding 2MB/s
2. The producer has Lz4 compression . Does the Prometheus query threshold need to be adjusted?

[Shawyeok]

There are only 3 partitions in the cluster with write rates exceeding 2MB/s
The producer has Lz4 compression . Does the Prometheus query threshold need to be adjusted?

You need to figure out what ledgers contribute most write load on the bookies with high write throughput.

How much difference between following querys?

sum(pulsar_throughput_in{cluster="YOUR_CLUSTER_NAME"}) / 1024^2

sum(rate(bookie_WRITE_BYTES{group="YOUR_CLUSTER_NAME"}[1m])) / 1024^2

Btw, what's query behind this graph?

[Jayer23]

There are only 3 partitions in the cluster with write rates exceeding 2MB/s
The producer has Lz4 compression . Does the Prometheus query threshold need to be adjusted?

You need to figure out what ledgers contribute most write load on the bookies with high write throughput.

How much difference between following querys?

sum(pulsar_throughput_in{cluster="YOUR_CLUSTER_NAME"}) / 1024^2

sum(rate(bookie_WRITE_BYTES{group="YOUR_CLUSTER_NAME"}[1m])) / 1024^2

Btw, what's query behind this graph?

1. sum(pulsar_throughput_in{cluster="YOUR_CLUSTER_NAME"}) / 1024^2

2. sum(rate(bookie_WRITE_BYTES{group="YOUR_CLUSTER_NAME"}[1m])) / 1024^2

3. We collected host-level disk io metrics，include journal disks and ledger disks，found that io-writes of some hosts is much larger than that of others
The query is: sum(diskio.bytes_written{instance=~"sd.*"})by(host）

[Shawyeok]

Is there any other process that might be generating I/O writes in your deployment? You can check this using pidstat -d 1.

[Jayer23]

Is there any other process that might be generating I/O writes in your deployment? You can check this using pidstat -d 1.

We are sure there is no other process. So we are also confused why bookie_WRITE_BYTES metrics are quite different from diskio.bytes_written

bookie_WRITE_BYTES

diskio.bytes_written

[Shawyeok]

Does bookie_WRITE_BYTES consistent with io write metric for journal disks? You may check compaction activities on ledger disks for further investigation.

Can you provide a demo to reproduce this problem?

[Jayer23]

Does bookie_WRITE_BYTES consistent with io write metric for journal disks? You may check compaction activities on ledger disks for further investigation.

Can you provide a demo to reproduce this problem?

Does bookie_WRITE_BYTES consistent with io write metric for journal disks?
Inconsistency.

You may check compaction activities on ledger disks for further investigation.
Ok, thanks.

Can you provide a demo to reproduce this problem?
We also don't konw how to reproduce it, but we found the same problem in several clusters.

[Jayer23]

We found that the reason for this problem is that in the default configurationjournalSyncData=true,which will cause a large amount of 8B data to be written to the journal disk. Because the 4K Alignment feature of SSD causes write amplification, the problem is solved after configuring journalSyncData=false

[Shawyeok]

We found that the reason for this problem is that in the default configuration journalSyncData=true, which will cause a large amount of 8B data to be written to the journal disk.

It's interesting. By default journalAlignmentSize is 4K, it means if you write an entry (the carrier of pulsar message in bookkeeper) with small payload (small message), it will align to 4K before write to disk. It's strange there're many write with 8B data written to journal.

Btw, it seems not explaining why it occur on some nodes only. Could you share your bookkeeper.conf? Is there any differences configuration in bookkeeper.conf in your entire deployment?

[Jayer23]

We found that the reason for this problem is that in the default configuration journalSyncData=true, which will cause a large amount of 8B data to be written to the journal disk.

It's interesting. By default journalAlignmentSize is 4K, it means if you write an entry (the carrier of pulsar message in bookkeeper) with small payload (small message), it will align to 4K before write to disk. It's strange there're many write with 8B data written to journal.

Btw, it seems not explaining why it occur on some nodes only. Could you share your bookkeeper.conf? Is there any differences configuration in bookkeeper.conf in your entire deployment?

The difference is nettyMaxFrameSizeBytes=52428800, Other configurations are default