Context Summarization for Breeze Buddy#571
Context Summarization for Breeze Buddy#571yugesh-ganipudi wants to merge 1 commit intojuspay:releasefrom
Conversation
WalkthroughAdds template-driven Breeze Buddy context summarization: a new ContextSummarizer that counts user turns and, when a configurable threshold is reached, requests an LLM summary, preserves system messages and recent turns, reconstructs the context, and integrates into the agent pipeline with runtime config toggles. Changes
Sequence DiagramsequenceDiagram
participant User
participant Pipeline as Breeze Buddy Pipeline
participant CS as ContextSummarizer
participant LLM as LLM Service
User->>Pipeline: Send message
Pipeline->>CS: add_message(message)
CS->>CS: Increment user-turn counter
alt counter < max_turns_before_summary
CS-->>Pipeline: Store message in context
else counter >= max_turns_before_summary
CS->>CS: Partition messages (system vs conversation), select recent turns
CS->>LLM: Request summary (include previous summary + old turns)
LLM-->>CS: Return generated summary
CS->>CS: Rebuild context: keep all system messages + new system summary + kept recent conversation messages
CS->>CS: Reset turn counter
CS-->>Pipeline: Provide summarized context
end
Pipeline-->>User: Continue conversation
Estimated code review effort🎯 4 (Complex) | ⏱️ ~50 minutes Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 3✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Pull request overview
Adds configurable context summarization to the Breeze Buddy voice agent to prevent long conversations from overflowing the LLM context window, using Redis-backed dynamic config and a new ContextSummarizer implementation.
Changes:
- Introduces a Breeze Buddy
ContextSummarizerthat summarizes older turns while keeping recent turns verbatim. - Adds Redis dynamic config getters to enable/disable summarization and tune summarization thresholds.
- Integrates summarization into the Breeze Buddy pipeline and documents setup/operation.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| docs/BREEZE_BUDDY_CONTEXT_SUMMARIZATION.md | Documents the feature, configuration keys, and operational guidance. |
| app/core/config/dynamic.py | Adds Redis-backed config accessors for Breeze Buddy summarization settings. |
| app/ai/voice/agents/breeze_buddy/features/summarizer/context_summarizer.py | Implements Breeze Buddy-specific summarizing OpenAI context wrapper. |
| app/ai/voice/agents/breeze_buddy/features/summarizer/init.py | Exports the summarizer for easier imports. |
| app/ai/voice/agents/breeze_buddy/agent/pipeline.py | Wires summarization into Breeze Buddy pipeline based on dynamic config. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| ) | ||
| return | ||
|
|
||
| logger.info(f"✅ Generated summary ({len(summary)} characters):\n{summary}") |
There was a problem hiding this comment.
The generated summary is logged at info level and includes the full summary text, which is likely to contain user-provided PII (addresses, phone numbers, order IDs). This should be downgraded to debug and/or redacted to avoid leaking sensitive data into production logs.
| logger.info(f"✅ Generated summary ({len(summary)} characters):\n{summary}") | |
| logger.info(f"✅ Generated summary ({len(summary)} characters).") | |
| logger.debug(f"Generated summary content:\n{summary}") |
| self._original_system_message = ( | ||
| messages[0] if messages and messages[0]["role"] == "system" else None | ||
| ) |
There was a problem hiding this comment.
self._original_system_message is initialized but never used in this implementation (the reconstruction path uses system_messages instead). Please remove it or use it consistently to avoid dead state that can confuse future maintainers.
| self._original_system_message = ( | |
| messages[0] if messages and messages[0]["role"] == "system" else None | |
| ) |
| - **`ENABLE_BB_SUMMARIZATION`** (boolean, default: `true`) | ||
| - Enables or disables automatic context summarization | ||
|
|
||
| - **`BB_MAX_TURNS_BEFORE_SUMMARY`** (integer, default: `10`) | ||
| - Number of user turns after which summarization is triggered | ||
|
|
||
| - **`BB_KEEP_RECENT_TURNS`** (integer, default: `2`) | ||
| - Number of recent conversation turns to keep in full detail after summarization |
There was a problem hiding this comment.
This doc states defaults of ENABLE_BB_SUMMARIZATION=true and BB_MAX_TURNS_BEFORE_SUMMARY=10, but the implementation in app/core/config/dynamic.py currently defaults to False and 5. Please update the documentation or code defaults so operators get the behavior described here.
| async def ENABLE_BB_SUMMARIZATION() -> bool: | ||
| """Returns ENABLE_BB_SUMMARIZATION from Redis. | ||
|
|
||
| When True, enables automatic conversation context summarization after a specified | ||
| number of turns to maintain context window efficiency while preserving conversation memory. | ||
| """ | ||
| return await get_config("ENABLE_BB_SUMMARIZATION", False, bool) | ||
|
|
There was a problem hiding this comment.
The documented defaults for Breeze Buddy summarization are enabled/10 turns, but this config function defaults to False, which will disable summarization unless Redis explicitly sets it. Either update the documentation/PR description or change the default here to match the intended behavior (and consider keeping it consistent with ENABLE_SUMMARIZATION in static config).
| async def BB_MAX_TURNS_BEFORE_SUMMARY() -> int: | ||
| """Returns BB_MAX_TURNS_BEFORE_SUMMARY from Redis. | ||
|
|
||
| Number of user turns after which the conversation context will be automatically | ||
| summarized to prevent context window overflow. | ||
| """ | ||
| return await get_config("BB_MAX_TURNS_BEFORE_SUMMARY", 5, int) | ||
|
|
There was a problem hiding this comment.
BB_MAX_TURNS_BEFORE_SUMMARY defaults to 5 here, but the new docs say the default is 10. Please align the default value and/or documentation so operators don’t get unexpected summarization frequency.
| # Keep all system messages (they contain template instructions and flow guidance) | ||
| for system_msg in system_messages: | ||
| new_messages.append(cast(ChatCompletionMessageParam, system_msg)) | ||
|
|
||
| # Add the conversation summary as a new system message | ||
| new_messages.append( | ||
| { | ||
| "role": "system", | ||
| "content": f"Previous conversation summary: {summary}", | ||
| } | ||
| ) |
There was a problem hiding this comment.
This keeps all existing system messages (including any prior "Previous conversation summary:" system message) and then appends a new summary system message. That will accumulate multiple summaries over time and can defeat the goal of shrinking context; it also makes the "previous_summary" lookup potentially grab an older summary. Consider filtering out any existing summary system messages and replacing them with the newly generated one (while still preserving template/system instruction messages).
| # Summary generation | ||
| "--- Breeze Buddy Summarizer: Generated summary: <summary text> ---" | ||
|
|
||
| # Errors | ||
| "Breeze Buddy Summarizer: Error during summarization: <error>" | ||
| "Breeze Buddy Summarizer: Summary generation resulted in empty content." |
There was a problem hiding this comment.
The monitoring section documents logging full summary text (e.g., "--- Breeze Buddy Summarizer: Generated summary: <summary text> ---"), which will include customer order details, addresses, and other PII in application logs. Persisting this sensitive conversational content in plain-text logs increases exposure if log stores are breached or overly broad access is granted. Consider limiting logs to non-sensitive metadata (counts, IDs) or redacting/anonymizing summaries before logging, and update this documentation to reflect the safer behavior.
- We can configure after how many turns we should summarize the context - We can also configure how many recent turns we need to keep in context without summarizing them
yugesh-ganipudi
force-pushed
the
BZ-48772-context-summarization-for-breeze-buddy
branch
from
2719c78 to
9d67cce
Compare
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@app/ai/voice/agents/breeze_buddy/features/summarizer/context_summarizer.py`:
- Around line 68-69: The code collects system_messages from self._messages then
later appends a new "Previous conversation summary:" system message, but it does
not remove prior summary system messages so summaries accumulate; update the
filtering logic in context_summarizer.py (references: self._messages,
system_messages, and the literal "Previous conversation summary:") to exclude
any existing system messages whose content starts with or matches "Previous
conversation summary:" before appending the new summary so only the latest
summary system message remains (apply same change to the other similar blocks
around the indicated ranges).
- Around line 78-86: The current logger calls in ContextSummarizer print full
conversation and summary content (e.g., logger.info/ debug using self._messages
and summary text), which risks leaking PII and triggers Ruff F541; change these
to log only metadata (counts, lengths, truncated hashes) or wrap full-content
logs behind a secure debug/redaction flag; update the logger calls in the
methods that reference self._messages, system_messages, conversation_messages
and any variables holding summaries so they emit summaries like "count=X,
chars=Y" or a redacted snippet when the debug flag is enabled, and replace
f-strings that interpolate full content with safe placeholders to avoid F541
errors.
- Around line 97-99: The current filtering uses value equality to build
messages_to_summarize (messages_to_summarize = [msg for msg in
conversation_messages if msg not in messages_to_keep]) which can drop
duplicates; instead select by index: compute a set of indices for
messages_to_keep from conversation_messages (or build messages_to_keep as
indices) and then slice/filter conversation_messages by index to produce
messages_to_summarize. Update the logic in context_summarizer.py (look for
conversation_messages, messages_to_keep, messages_to_summarize) to derive
messages_to_keep_indices and then build messages_to_summarize by iterating over
enumerate(conversation_messages) and including entries whose index is not in
messages_to_keep_indices.
In `@docs/BREEZE_BUDDY_CONTEXT_SUMMARIZATION.md`:
- Around line 119-142: The fenced example blocks showing the conversation (the
blocks that start with "[System] You are Rhea from Freshbus..." and the
summarized block containing "[System Summary] Customer Yugesh...") are missing a
language identifier and trigger markdownlint MD040; update both triple-backtick
fences to include a language (e.g., use ```text) so each fenced code block
becomes ```text ... ```, ensuring all opening and closing fences still match and
that the blocks containing "[System]" and "[System Summary]" are updated.
ℹ️ Review info
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (6)
app/ai/voice/agents/breeze_buddy/agent/pipeline.pyapp/ai/voice/agents/breeze_buddy/features/summarizer/__init__.pyapp/ai/voice/agents/breeze_buddy/features/summarizer/context_summarizer.pyapp/ai/voice/agents/breeze_buddy/template/types.pyapp/core/config/dynamic.pydocs/BREEZE_BUDDY_CONTEXT_SUMMARIZATION.md
| system_messages = [msg for msg in self._messages if msg["role"] == "system"] | ||
| conversation_messages = [ |
There was a problem hiding this comment.
Avoid accumulating multiple summary system messages.
Right now old "Previous conversation summary:" system messages are preserved and a new one is appended, so subsequent runs read the oldest summary and the context grows each time. Filter out the prior summary message(s) and keep only the latest summary.
🛠️ Suggested fix
- system_messages = [msg for msg in self._messages if msg["role"] == "system"]
+ system_messages: List[ChatCompletionMessageParam] = []
+ previous_summary = ""
+ for msg in self._messages:
+ if msg["role"] != "system":
+ continue
+ content = msg.get("content", "")
+ if (
+ isinstance(content, str)
+ and "Previous conversation summary:" in content
+ ):
+ previous_summary = content.replace(
+ "Previous conversation summary:", ""
+ ).strip()
+ continue
+ system_messages.append(cast(ChatCompletionMessageParam, msg))
- # Find previous summary
- previous_summary = ""
- for msg in self._messages:
- if msg["role"] == "system":
- content = msg.get("content", "")
- # Handle both string and iterable content types
- if (
- isinstance(content, str)
- and "Previous conversation summary:" in content
- ):
- previous_summary = content.replace(
- "Previous conversation summary:", ""
- ).strip()
- break
- for system_msg in system_messages:
- new_messages.append(cast(ChatCompletionMessageParam, system_msg))
+ for system_msg in system_messages:
+ new_messages.append(system_msg)Also applies to: 110-123, 224-230
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@app/ai/voice/agents/breeze_buddy/features/summarizer/context_summarizer.py`
around lines 68 - 69, The code collects system_messages from self._messages then
later appends a new "Previous conversation summary:" system message, but it does
not remove prior summary system messages so summaries accumulate; update the
filtering logic in context_summarizer.py (references: self._messages,
system_messages, and the literal "Previous conversation summary:") to exclude
any existing system messages whose content starts with or matches "Previous
conversation summary:" before appending the new summary so only the latest
summary system message remains (apply same change to the other similar blocks
around the indicated ranges).
| logger.info(f"=== BREEZE BUDDY SUMMARIZATION START ===") | ||
| logger.info( | ||
| f"Total messages in context BEFORE summarization: {len(self._messages)}" | ||
| ) | ||
| logger.info( | ||
| f"System messages: {len(system_messages)}, Conversation messages: {len(conversation_messages)}" | ||
| ) | ||
| logger.debug(f"Context BEFORE summarization:\n{self._messages}") | ||
|
|
There was a problem hiding this comment.
Redact or downgrade logs that include message/summary content.
These logs include full conversation content and summary text (names, addresses, phone/email, etc.), which is a privacy/compliance risk and can bloat logs. Prefer counts/lengths, or gate full content behind a secure redaction/debug flag. This also fixes the Ruff F541 errors on the f-strings without placeholders.
🔐 Suggested fix
- logger.info(f"=== BREEZE BUDDY SUMMARIZATION START ===")
+ logger.info("=== BREEZE BUDDY SUMMARIZATION START ===")
...
- logger.debug(f"Context BEFORE summarization:\n{self._messages}")
+ logger.debug(
+ "Context BEFORE summarization: [redacted], total=%d",
+ len(self._messages),
+ )
...
- logger.info(f"✅ Generated summary ({len(summary)} characters):\n{summary}")
+ logger.info("✅ Generated summary (%d characters).", len(summary))
...
- logger.info(f"=== BREEZE BUDDY SUMMARIZATION COMPLETE ===")
+ logger.info("=== BREEZE BUDDY SUMMARIZATION COMPLETE ===")
...
- logger.debug(f"Context AFTER summarization:\n{new_messages}")
+ logger.debug(
+ "Context AFTER summarization: [redacted], total=%d",
+ len(new_messages),
+ )Also applies to: 215-223, 231-246
🧰 Tools
🪛 Ruff (0.15.2)
[error] 78-78: f-string without any placeholders
Remove extraneous f prefix
(F541)
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@app/ai/voice/agents/breeze_buddy/features/summarizer/context_summarizer.py`
around lines 78 - 86, The current logger calls in ContextSummarizer print full
conversation and summary content (e.g., logger.info/ debug using self._messages
and summary text), which risks leaking PII and triggers Ruff F541; change these
to log only metadata (counts, lengths, truncated hashes) or wrap full-content
logs behind a secure debug/redaction flag; update the logger calls in the
methods that reference self._messages, system_messages, conversation_messages
and any variables holding summaries so they emit summaries like "count=X,
chars=Y" or a redacted snippet when the debug flag is enabled, and replace
f-strings that interpolate full content with safe placeholders to avoid F541
errors.
| messages_to_summarize = [ | ||
| msg for msg in conversation_messages if msg not in messages_to_keep | ||
| ] |
There was a problem hiding this comment.
Avoid equality-based filtering when selecting messages to summarize.
msg not in messages_to_keep relies on equality, so duplicate messages (same content) can be dropped from both keep/summarize lists. Slice by index instead.
🧩 Suggested fix
- messages_to_summarize = [
- msg for msg in conversation_messages if msg not in messages_to_keep
- ]
+ keep_start_idx = len(conversation_messages) - len(messages_to_keep)
+ messages_to_summarize = conversation_messages[:keep_start_idx]🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@app/ai/voice/agents/breeze_buddy/features/summarizer/context_summarizer.py`
around lines 97 - 99, The current filtering uses value equality to build
messages_to_summarize (messages_to_summarize = [msg for msg in
conversation_messages if msg not in messages_to_keep]) which can drop
duplicates; instead select by index: compute a set of indices for
messages_to_keep from conversation_messages (or build messages_to_keep as
indices) and then slice/filter conversation_messages by index to produce
messages_to_summarize. Update the logic in context_summarizer.py (look for
conversation_messages, messages_to_keep, messages_to_summarize) to derive
messages_to_keep_indices and then build messages_to_summarize by iterating over
enumerate(conversation_messages) and including entries whose index is not in
messages_to_keep_indices.
| ``` | ||
| [System] You are Rhea from Freshbus... | ||
| [System] Politely greet the customer using their name Yugesh... | ||
| [User 1] Hi, I placed an order | ||
| [Assistant 1] Great! Can you provide your order ID? | ||
| [User 2] It's ORDER123 | ||
| [Assistant 2] Thank you. Let me verify... | ||
| ... | ||
| [User 10] Yes, that's correct | ||
| [Assistant 10] Perfect. Your address is 123 Main St, Apt 4B | ||
| [User 11] Can you repeat the delivery time? | ||
| [Assistant 11] It will arrive tomorrow at 2 PM | ||
| [User 12] Great, thank you | ||
| ``` | ||
|
|
||
| **After Summarization (6 messages):** | ||
| ``` | ||
| [System] You are Rhea from Freshbus... | ||
| [System] Politely greet the customer using their name Yugesh... | ||
| [System Summary] Customer Yugesh placed order ORDER123. Delivery address verified as 123 Main St, Apt 4B. Customer confirmed delivery time of tomorrow at 2 PM. Customer was polite and appreciative. | ||
| [User 11] Can you repeat the delivery time? | ||
| [Assistant 11] It will arrive tomorrow at 2 PM | ||
| [User 12] Great, thank you | ||
| ``` |
There was a problem hiding this comment.
Specify a language for the example fences.
markdownlint flags these fences without language identifiers; adding text (or console) fixes MD040.
📝 Suggested fix
-```
+```text
[System] You are Rhea from Freshbus...
[System] Politely greet the customer using their name Yugesh...
...
[User 12] Great, thank you
-```
+```
-```
+```text
[System] You are Rhea from Freshbus...
[System] Politely greet the customer using their name Yugesh...
[System Summary] Customer Yugesh placed order ORDER123. Delivery address verified as 123 Main St, Apt 4B. Customer confirmed delivery time of tomorrow at 2 PM. Customer was polite and appreciative.
[User 11] Can you repeat the delivery time?
[Assistant 11] It will arrive tomorrow at 2 PM
[User 12] Great, thank you
-```
+```📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| ``` | |
| [System] You are Rhea from Freshbus... | |
| [System] Politely greet the customer using their name Yugesh... | |
| [User 1] Hi, I placed an order | |
| [Assistant 1] Great! Can you provide your order ID? | |
| [User 2] It's ORDER123 | |
| [Assistant 2] Thank you. Let me verify... | |
| ... | |
| [User 10] Yes, that's correct | |
| [Assistant 10] Perfect. Your address is 123 Main St, Apt 4B | |
| [User 11] Can you repeat the delivery time? | |
| [Assistant 11] It will arrive tomorrow at 2 PM | |
| [User 12] Great, thank you | |
| ``` | |
| **After Summarization (6 messages):** | |
| ``` | |
| [System] You are Rhea from Freshbus... | |
| [System] Politely greet the customer using their name Yugesh... | |
| [System Summary] Customer Yugesh placed order ORDER123. Delivery address verified as 123 Main St, Apt 4B. Customer confirmed delivery time of tomorrow at 2 PM. Customer was polite and appreciative. | |
| [User 11] Can you repeat the delivery time? | |
| [Assistant 11] It will arrive tomorrow at 2 PM | |
| [User 12] Great, thank you | |
| ``` |
🧰 Tools
🪛 markdownlint-cli2 (0.21.0)
[warning] 119-119: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
[warning] 135-135: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@docs/BREEZE_BUDDY_CONTEXT_SUMMARIZATION.md` around lines 119 - 142, The
fenced example blocks showing the conversation (the blocks that start with
"[System] You are Rhea from Freshbus..." and the summarized block containing
"[System Summary] Customer Yugesh...") are missing a language identifier and
trigger markdownlint MD040; update both triple-backtick fences to include a
language (e.g., use ```text) so each fenced code block becomes ```text ... ```,
ensuring all opening and closing fences still match and that the blocks
containing "[System]" and "[System Summary]" are updated.
Summary by CodeRabbit
New Features
Documentation