Skip to content

test(dialogflow): deflake and re-enable test cases in ITSystemTest#13511

Open
whowes wants to merge 1 commit into
mainfrom
whowes/b423958346
Open

test(dialogflow): deflake and re-enable test cases in ITSystemTest#13511
whowes wants to merge 1 commit into
mainfrom
whowes/b423958346

Conversation

@whowes

@whowes whowes commented Jun 18, 2026

Copy link
Copy Markdown
Contributor

Each of these test cases failed 7/30 times locally without these fixes. With the fixes applied, tests pass without flakes in 100+ runs.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request enables previously ignored integration tests (detectIntentTest and listContextsTest) in ITSystemTest and introduces the awaitility library to handle eventual consistency. The feedback suggests improving the robustness of these tests against flakiness. Specifically, for detectIntentTest, it is recommended to use untilAsserted to poll and retry all assertions together, which also removes the need for AtomicReference boilerplate. For listContextsTest, wrapping the assertion in an awaitility block is advised to prevent failures caused by replication lag.

Comment on lines 285 to 300
AtomicReference<DetectIntentResponse> responseRef =
new AtomicReference<>();
await()
.atMost(Duration.ofSeconds(10))
.pollInterval(Duration.ofSeconds(2))
.until(
() -> {
DetectIntentResponse response = sessionsClient.detectIntent(request);
responseRef.set(response);
return !response.getQueryResult().getQueryText().isEmpty();
});
QueryResult result = responseRef.get().getQueryResult();
assertEquals(EVENT_NAME, result.getQueryText());
assertEquals(ACTION_NAME, result.getAction());
assertEquals(DEFAULT_LANGUAGE_CODE, result.getLanguageCode());
assertEquals(intent.getDisplayName(), result.getIntent().getDisplayName());

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

Using until(...) with a condition that only checks if queryText is not empty might not actually prevent flakiness. Since the query input is an event, the query text in the response is immediately populated with the event name on the first attempt, even if the intent itself hasn't been fully propagated or matched yet. This causes the await() block to exit immediately, and the subsequent assertions (like matching the intent display name) might still fail due to eventual consistency.

Instead, use untilAsserted(...) to poll and retry all assertions. This also eliminates the need for the AtomicReference boilerplate.

Suggested change
AtomicReference<DetectIntentResponse> responseRef =
new AtomicReference<>();
await()
.atMost(Duration.ofSeconds(10))
.pollInterval(Duration.ofSeconds(2))
.until(
() -> {
DetectIntentResponse response = sessionsClient.detectIntent(request);
responseRef.set(response);
return !response.getQueryResult().getQueryText().isEmpty();
});
QueryResult result = responseRef.get().getQueryResult();
assertEquals(EVENT_NAME, result.getQueryText());
assertEquals(ACTION_NAME, result.getAction());
assertEquals(DEFAULT_LANGUAGE_CODE, result.getLanguageCode());
assertEquals(intent.getDisplayName(), result.getIntent().getDisplayName());
await()
.atMost(Duration.ofSeconds(10))
.pollInterval(Duration.ofSeconds(2))
.untilAsserted(
() -> {
DetectIntentResponse response = sessionsClient.detectIntent(request);
QueryResult result = response.getQueryResult();
assertEquals(EVENT_NAME, result.getQueryText());
assertEquals(ACTION_NAME, result.getAction());
assertEquals(DEFAULT_LANGUAGE_CODE, result.getLanguageCode());
assertEquals(intent.getDisplayName(), result.getIntent().getDisplayName());
});

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this relevant?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wasn't able to repro any flakiness with my original change (not retrying on failed assertions), but this seems like a reasonable improvement since Gemini pointed out that the query text will be populated with the event name (thus not empty) even if processing is not yet complete.

Comment on lines +307 to +310
boolean contextNameInActualContext =
stream(contextsClient.listContexts(request).iterateAll())
.anyMatch(actualContext -> context.getName().equals(actualContext.getName()));
assertTrue(contextNameInActualContext);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

To ensure listContextsTest is fully deflaked against eventual consistency (replication lag), we should also wrap the assertion in an awaitility block. Otherwise, if the context is not immediately visible in the list, the test will fail.

Suggested change
boolean contextNameInActualContext =
stream(contextsClient.listContexts(request).iterateAll())
.anyMatch(actualContext -> context.getName().equals(actualContext.getName()));
assertTrue(contextNameInActualContext);
await()
.atMost(Duration.ofSeconds(10))
.pollInterval(Duration.ofSeconds(2))
.untilAsserted(
() -> {
boolean contextNameInActualContext =
stream(contextsClient.listContexts(request).iterateAll())
.anyMatch(actualContext -> context.getName().equals(actualContext.getName()));
assertTrue(contextNameInActualContext);
});

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this relevant?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this one is necessary - the observed flakiness was related to the unexpected system context appearing in the list and not related to eventual consistency.

@whowes
whowes force-pushed the whowes/b423958346 branch from a95787c to 0b301eb Compare June 18, 2026 02:57
@whowes
whowes marked this pull request as ready for review June 18, 2026 16:15
@whowes
whowes requested review from a team as code owners June 18, 2026 16:15
@whowes whowes added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Jun 18, 2026
@yoshi-kokoro yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Jun 18, 2026
@whowes
whowes force-pushed the whowes/b423958346 branch from 0b301eb to 9ee2b18 Compare July 8, 2026 16:57
Comment on lines +289 to +294
.until(
() -> {
DetectIntentResponse response = sessionsClient.detectIntent(request);
responseRef.set(response);
return !response.getQueryResult().getQueryText().isEmpty();
});

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it make sense to replace this with the following?

.until(
        () -> sessionsClient.detectIntent(request),
        res -> !res.getQueryResult().getQueryText().isEmpty());

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I opted to move to the untilAsserted() approach from the Gemini suggestion instead.

Comment on lines +307 to +310
boolean contextNameInActualContext =
stream(contextsClient.listContexts(request).iterateAll())
.anyMatch(actualContext -> context.getName().equals(actualContext.getName()));
assertTrue(contextNameInActualContext);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this relevant?

@whowes
whowes force-pushed the whowes/b423958346 branch from 9ee2b18 to 42715ad Compare July 10, 2026 21:15
@whowes
whowes requested a review from jinseopkim0 July 22, 2026 22:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants