Skip to content

feat(llma): add $ai_stop_reason property to LLM analytics #54037

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
carlos-marchal-ph merged 5 commits into from
Apr 20, 2026
Merged

feat(llma): add $ai_stop_reason property to LLM analytics #54037

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
6053da9
feat(llma): add $ai_stop_reason property to LLM analytics
carlos-marchal-ph Apr 10, 2026
89c0a1d
fix(llma): add missing OTel stop_reason test coverage
carlos-marchal-ph Apr 10, 2026
f279639
Merge remote-tracking branch 'origin/master' into feat(llma)/add-ai-s…
carlos-marchal-ph Apr 20, 2026
d5f34d4
chore: revert hogql/ai.py taxonomy addition, will ship separately
carlos-marchal-ph Apr 20, 2026
4baac50
fix: remove duplicate $ai_stop_reason taxonomy entry
carlos-marchal-ph Apr 20, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions nodejs/src/ingestion/ai/otel/middleware/traceloop.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -204,6 +204,28 @@ describe('traceloop middleware', () => {
expect(event.properties!['llm.usage.total_tokens']).toBeUndefined()
expect(event.properties!['llm.response.finish_reason']).toBeUndefined()
expect(event.properties!['llm.response.stop_reason']).toBeUndefined()
// stop_reason takes priority over finish_reason
expect(event.properties!['$ai_stop_reason']).toBe('end_turn')
Comment on lines +207 to +208
Copy link
Copy Markdown
Contributor

@greptile-apps greptile-apps Bot Apr 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 finish_reason-only path is not exercised

The test has both llm.response.stop_reason and llm.response.finish_reason present. The case where only llm.response.finish_reason is provided (no stop_reason) is untested, so a regression in the ?? fallback would go undetected. A parametrised test covering both-present (stop_reason wins), finish_reason-only, and neither-present (undefined) would match the preference for parameterised tests.

Prompt To Fix With AI 
This is a comment left during a code review.
Path: nodejs/src/ingestion/ai/otel/middleware/traceloop.test.ts
Line: 207-208

Comment:
**`finish_reason`-only path is not exercised**

The test has both `llm.response.stop_reason` and `llm.response.finish_reason` present. The case where only `llm.response.finish_reason` is provided (no `stop_reason`) is untested, so a regression in the `??` fallback would go undetected. A parametrised test covering both-present (stop_reason wins), finish_reason-only, and neither-present (undefined) would match the preference for parameterised tests.

How can I resolve this? If you propose a fix, please make it concise.

})

it('maps finish_reason to $ai_stop_reason when stop_reason is absent', () => {
const event = createEvent('$ai_generation', {
'llm.request.type': 'chat',
'llm.response.finish_reason': 'stop',
})
traceloop.process(event, () => mapOtelAttributes(event))

expect(event.properties!['llm.response.finish_reason']).toBeUndefined()
expect(event.properties!['$ai_stop_reason']).toBe('stop')
})

it('leaves $ai_stop_reason undefined when neither reason is present', () => {
const event = createEvent('$ai_generation', {
'llm.request.type': 'chat',
})
traceloop.process(event, () => mapOtelAttributes(event))

expect(event.properties!['$ai_stop_reason']).toBeUndefined()
})
})
})
12 changes: 10 additions & 2 deletions nodejs/src/ingestion/ai/otel/middleware/traceloop.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,6 @@ const STRIP_KEYS = [
'traceloop.entity.output',
'llm.is_streaming',
'llm.usage.total_tokens',
'llm.response.finish_reason',
'llm.response.stop_reason',
]

interface IndexedEntry {
Expand Down Expand Up @@ -157,6 +155,16 @@ function process(event: PluginEvent, next: () => void): void {
}
}

// Map stop/finish reason to $ai_stop_reason before stripping
if (props['$ai_stop_reason'] === undefined) {
const stopReason = props['llm.response.stop_reason'] ?? props['llm.response.finish_reason']
if (stopReason !== undefined) {
props['$ai_stop_reason'] = stopReason
}
}
delete props['llm.response.stop_reason']
delete props['llm.response.finish_reason']

props['$ai_lib'] = 'opentelemetry/traceloop'

for (const key of STRIP_KEYS) {
Expand Down
21 changes: 21 additions & 0 deletions nodejs/src/ingestion/ai/otel/middleware/vercel-ai.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,27 @@ describe('vercel-ai middleware', () => {
expect(key).not.toBe('operation.name')
expect(key).not.toBe('resource.name')
}
expect(event.properties!['$ai_stop_reason']).toBe('stop')
Copy link
Copy Markdown
Contributor

@greptile-apps greptile-apps Bot Apr 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Missing test for gen_ai.response.finish_reasons array path

The gen_ai.response.finish_reasons fallback (the else if branch in vercel-ai.ts) is untested. That branch has its own logic — array type check plus [0] indexing — and currently any breakage there would go undetected. A parametrised test covering ai.response.finishReason (primary), gen_ai.response.finish_reasons array (fallback), and neither present (undefined) would cover the three cases OnceAndOnlyOnce.

Prompt To Fix With AI 
This is a comment left during a code review.
Path: nodejs/src/ingestion/ai/otel/middleware/vercel-ai.test.ts
Line: 97

Comment:
**Missing test for `gen_ai.response.finish_reasons` array path**

The `gen_ai.response.finish_reasons` fallback (the `else if` branch in `vercel-ai.ts`) is untested. That branch has its own logic — array type check plus `[0]` indexing — and currently any breakage there would go undetected. A parametrised test covering `ai.response.finishReason` (primary), `gen_ai.response.finish_reasons` array (fallback), and neither present (undefined) would cover the three cases OnceAndOnlyOnce.

How can I resolve this? If you propose a fix, please make it concise.

})

it('maps gen_ai.response.finish_reasons array to $ai_stop_reason', () => {
const event = createEvent('$ai_generation', {
'ai.operationId': 'ai.generateText.doGenerate',
'gen_ai.response.finish_reasons': ['length'],
})
convertOtelEvent(event)

expect(event.properties!['gen_ai.response.finish_reasons']).toBeUndefined()
expect(event.properties!['$ai_stop_reason']).toBe('length')
})

it('leaves $ai_stop_reason undefined when no finish reason is present', () => {
const event = createEvent('$ai_generation', {
'ai.operationId': 'ai.generateText.doGenerate',
})
convertOtelEvent(event)

expect(event.properties!['$ai_stop_reason']).toBeUndefined()
})
})

Expand Down
15 changes: 13 additions & 2 deletions nodejs/src/ingestion/ai/otel/middleware/vercel-ai.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,6 @@ const STRIP_KEYS = [
'ai.usage.promptTokens',
'ai.usage.completionTokens',
'ai.usage.tokens',
'ai.response.finishReason',
'ai.response.id',
'ai.response.model',
'ai.response.timestamp',
Expand All @@ -43,7 +42,6 @@ const STRIP_KEYS = [
'resource.name',
// Standard GenAI semantic convention attributes not mapped to $ai_* properties
'gen_ai.request.max_tokens',
'gen_ai.response.finish_reasons',
'gen_ai.response.id',
]

Expand Down Expand Up @@ -157,6 +155,19 @@ function process(event: PluginEvent, next: () => void): void {
}
}

// Map finish reason to $ai_stop_reason before stripping
if (props['$ai_stop_reason'] === undefined) {
const vercelReason = props['ai.response.finishReason']
const genAiReasons = props['gen_ai.response.finish_reasons']
if (vercelReason !== undefined) {
props['$ai_stop_reason'] = vercelReason
} else if (Array.isArray(genAiReasons) && genAiReasons.length > 0) {
props['$ai_stop_reason'] = genAiReasons[0]
}
}
delete props['ai.response.finishReason']
delete props['gen_ai.response.finish_reasons']

props['$ai_lib'] = 'opentelemetry/vercel-ai'

for (const key of STRIP_KEYS) {
Expand Down
Loading