[Bug] a dead-lock case in ServiceInstanceChangeListener will cause consumer can`t use latest provider instances list

[mz0113]

Pre-check

• I am sure that all the content I provide is in English.

Search before asking

• I had searched in the issues and found no similar issues.

Apache Dubbo Component

Java SDK (apache/dubbo)

Dubbo Version

3.0.12 or 3.2.18 also have this bug
jdk1.8
windows 11

Steps to reproduce this issue

服务提供者下线时候，或者重启后变更IP情况下，有概率触发此死锁问题，表现就是消费者端感知不到提供者节点已经变化，造成no provider等情况，我看了下线程栈，nacos.publisher-com.alibaba.nacos.client.naming.event.InstancesChangeEvent 会一直wait等待获取元数据的锁，而这把锁是被元数据重试线程一直抢占，而每次元数据重试都会卡好几秒（因为provider已经下线了调不通了）。InstancesChangeEvent 没办法继续执行，就会导致nacos的事件队列发生堆积（就是那个queue），会堆积到好几千上万，消费不动事件队列了。

目前看下来是概率性的，要巧合一点，但是基本上几天会必现一次，尤其是线下测试环境因为服务变动非常频繁更加容易触发。

我看代码中，元数据重试线程每次执行重试任务都是以org.apache.dubbo.registry.client.event.listener.ServiceInstancesChangedListener#allInstances 这个字段存储的最新的提供者列表进行元数据获取的。但问题就恰好出现在这，一种虚拟死锁，allInstances 因为InstancesChangeEvent 线程抢不到锁一直卡着，无法更新为最新的提供者列表，因此allInstances 中一直有下线的提供者节点在，元数据重试线程就一直对这些节点进行重试，进一步抢占元数据锁。最终陷入一种死锁的逻辑中，再无自行恢复的可能了。

====以下是英文翻译=====

‌Service providers going offline, or changing IP addresses after restart, may probabilistically trigger this deadlock issue.‌ The symptom is that consumer-side clients fail to detect changes in provider nodes, resulting in "no provider" errors. Upon reviewing the thread dump, the nacos.publisher-com.alibaba.nacos.client.naming.event.InstancesChangeEvent thread continuously waits to acquire the metadata lock, which is monopolized by the metadata retry thread. Each metadata retry gets blocked for several seconds—since the target provider is already offline and unreachable. As a result, InstancesChangeEvent cannot proceed, causing the Nacos event queue (i.e., the internal queue) to accumulate thousands or even tens of thousands of pending events, eventually becoming unprocessable.

This issue occurs probabilistically but tends to manifest at least once every few days, especially in offline testing environments where service changes are frequent.

From code analysis, the metadata retry thread uses the org.apache.dubbo.registry.client.event.listener.ServiceInstancesChangedListener#allInstances field—supposedly holding the latest provider list—to perform metadata retrieval. However, the crux lies here: a virtual deadlock occurs. Since the InstancesChangeEvent thread cannot acquire the lock to update allInstances, the list remains stale and still contains the offline provider nodes. Consequently, the retry thread keeps attempting to access these defunct nodes, further holding the metadata lock. This creates a self-sustaining deadlock loop with no possibility of self-recovery.

What you expected to happen

do not dead-lock

Anything else

No response

Do you have a (mini) reproduction demo?

• Yes, I have a minimal reproduction demo to help resolve this issue more effectively!

Are you willing to submit a pull request to fix on your own?

• Yes I am willing to submit a pull request on my own!

Code of Conduct

• I agree to follow this project's Code of Conduct

[zrlw]

Does this issue has something to do with #16067 ?
what's your dubbo.metadata.storage-type (remote / local)?

[zrlw]

maybe you should try latest codes as the method getRemoteMetadata has changed.

3.0.12

        synchronized (metaCacheManager) {  <=== line 169 in your graph （InstancesChangeEvent thread Blocked）
            // try to load metadata from remote.
            int triedTimes = 0;
            while (triedTimes < 3) {
                metadata = MetadataUtils.getRemoteMetadata(revision, instances, metadataReport);  《=== line 173 in your graph (metadata retry thread waiting for response)

                if (metadata != MetadataInfo.EMPTY) {// succeeded
                    metadata.init();
                    break;
                } else {// failed
                    if (triedTimes > 0) {
                        if (logger.isDebugEnabled()) {
                            logger.debug("Retry the " + triedTimes + " times to get metadata for revision=" + revision);
                        }
                    }
                    triedTimes++;
                    try {
                        Thread.sleep(1000);
                    } catch (InterruptedException e) {
                    }
                }
            }

            if (metadata == MetadataInfo.EMPTY) {
                logger.error("Failed to get metadata for revision after 3 retries, revision=" + revision);
            } else {
                metaCacheManager.put(revision, metadata);
            }
        }
        return metadata;

latest codes:

        synchronized (metaCacheManager) {
            // try to load metadata from remote.
            int triedTimes = 0;
            while (triedTimes < 3) {

                metadata = MetricsEventBus.post(                                                <===  Changed to async mode
                        MetadataEvent.toSubscribeEvent(applicationModel),
                        () -> MetadataUtils.getRemoteMetadata(revision, instances, metadataReport),
                        result -> result != MetadataInfo.EMPTY);

                if (metadata != MetadataInfo.EMPTY) { // succeeded
                    metadata.init();
                    break;
                } else { // failed
                    if (triedTimes > 0) {
                        if (logger.isDebugEnabled()) {
                            logger.debug("Retry the " + triedTimes + " times to get metadata for revision=" + revision);
                        }
                    }
                    triedTimes++;
                    try {
                        Thread.sleep(1000);
                    } catch (InterruptedException e) {
                    }
                }
            }

            if (metadata == MetadataInfo.EMPTY) {
                logger.error(
                        REGISTRY_FAILED_LOAD_METADATA,
                        "",
                        "",
                        "Failed to get metadata for revision after 3 retries, revision=" + revision);
            } else {
                metaCacheManager.put(revision, metadata);
            }
        }
        return metadata;
    }

[mz0113]

Does this issue has something to do with #16067 ? what's your dubbo.metadata.storage-type (remote / local)?

it is local mode，and this issue doesn`t have relation to #16067 ;

从你发的比对代码来看，重试线程仍旧会同步阻塞等待元数据返回呀，否则的话，那个while (triedTimes < 3) 的重试循环怎么跑以及下面的metadata.init();也需要元数据结果才能执行，肯定是必须要拿到上一次的元数据查询结果再看是否需要重试。也就是这段代码本质上还是同步阻塞的，如果每次元数据需要3秒钟获取，那么这个方法要3秒*重试次数后才会return。

这个issue简单来说就是nacos服务变更通知下来后，需要进行元数据查询，然后查询失败后提交元数据重试任务，元数据重试线程把元数据锁长时间占用，导致服务变更事件线程一直拿不到锁无法更新服务列表，形成一种类似于死锁的样子（但不是真的两把锁互相死锁了），因为无法刷新服务列表，所以重试线程一直都会尝试重试获取那些已经下线的节点的元数据。但是服务列表又需要事件变更线程拿到这把锁才能刷新列表。所以就尬住了。 因此只要这段代码持有了元数据锁且是同步阻塞的，那么一定会阻碍事件线程继续往下跑。

==========翻译========

From the comparison code you provided, the retry thread will still synchronously block and wait for the metadata to return; otherwise, how would the retry loop while (triedTimes < 3) run, and how could metadata.init(); be executed without the metadata result? It's clear that the metadata query result from the previous attempt must be obtained first to determine whether a retry is needed. Therefore, this code is essentially synchronous and blocking. If each metadata query takes 3 seconds, then the method will not return until 3 seconds multiplied by the number of retries have passed.

In simple terms, when a Nacos service change notification is received, it needs to query metadata. If the query fails, the retry thread holds the lock for an extended period, preventing the service change event thread from acquiring the lock to update the service list. This creates a deadlock-like situation (though not an actual deadlock with two locks waiting for each other), because the service list cannot be refreshed, so the retry thread keeps attempting to retrieve metadata for nodes that are already offline. However, the service list can only be refreshed when the event change thread acquires the lock. Thus, the system gets stuck.

[mz0113]

Does this issue has something to do with #16067 ? what's your dubbo.metadata.storage-type (remote / local)?

@zrlw 另外，DefaultPublisher这个类是nacos-client sdk中的，它从queue中take出元素后同步阻塞的调用dubbo的instancelistener去刷新服务列表。