Match Extended Context cap to what the engine actually uses#537
Conversation
The UI advertised a 4000-character Extended Context limit, but the Open Source base-model path rendered it as a 'notes' section hard-capped at 600 chars (including a ~32-char label), so only the first ~570 chars were ever used on-device and the rest was silently dropped. Lower the user-facing/storage cap to 1200 and raise the notes section cap to 1300 so the full entry reaches the local model. The total base-prompt budget stays 2400 chars, so prompt size and latency are unchanged; this only stops wasting the budget. Both caps now document the invariant (notes maxChars >= UI cap + label) so they cannot silently diverge again. Also shrinks the Foundation Models nano context-window overflow described in #486. Refs #486
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| // `maxChars` must stay at or above `SuggestionSettingsModel.maximumExtendedContextCharacters` | ||
| // plus this label (~32 chars) so the full user-entered Extended Context survives here instead | ||
| // of being silently clipped far under the advertised cap. It still competes for the 2400-char | ||
| // total budget below (priority 40), so an unusually long prefix can trim it, but in normal use | ||
| // the whole blob lands. | ||
| sections.append(Self.contextSection("notes", "Notes the writer keeps in mind: \(notes)", priority: 40, maxChars: 1300)) |
There was a problem hiding this comment.
The relationship between notes maxChars (1300) and SuggestionSettingsModel.maximumExtendedContextCharacters + label_length (1200 + 32 = 1232) is documented in the comment but never asserted. If the label string in line 51 grows or maximumExtendedContextCharacters rises above 1268 (1300 − 32), notes will again be silently clipped without any failing test or compile-time signal. The ExtendedContextTests only validate the model-layer cap, not that the renderer's maxChars is large enough to pass the full blob through.
Consider a #if DEBUG assertion in BaseCompletionPromptRenderer or an XCTAssert in the renderer tests that directly checks notes maxChars >= SuggestionSettingsModel.maximumExtendedContextCharacters + labelLength so the two constants cannot diverge silently again.
1 participant
Summary
Extended Context advertised a 4000-character cap in the UI, but the Open Source
(base-model) path only ever used the first ~570 characters: the prompt's "notes"
section was hard-capped at 600 chars including its ~32-char label, so up to ~85%
of a long entry was silently dropped on-device, with the only on-screen signal a
counter that implied all 4000 were used.
This aligns the two ends: lower the user-facing/storage cap to 1200 characters
and raise the notes section cap to 1300 (covers the full 1200 plus the label)
so the whole entry actually reaches the local model. The total base-prompt budget
is unchanged (2400 chars), so the latency envelope is identical to today; this
only stops wasting that budget, it does not enlarge the prompt.
Why 1200: ~300 tokens is a meaningful glossary or style guide, it fits the
2400-char base prompt even alongside a maxed 1000-char prefix, and it stays well
inside Apple's 4096-token window on the Foundation Models path (which also reduces
the nano-model truncation reported in #486). Both caps now carry comments stating
the invariant (notes
maxChars>= UI cap + label) so they cannot silentlydiverge again.
Validation
The cap-sensitive test (
ExtendedContextTests) asserts truncation againstSuggestionSettingsModel.maximumExtendedContextCharactersdynamically rather thana hardcoded number, so it tracks the new 1200 value automatically; the renderer
tests use short fixtures that hit neither the old nor the new cap.
Linked issues
Refs #486 (extended context not reliably used on Apple Intelligence + nano):
lowering the cap shrinks the FM/nano context-window overflow this describes. It
does not close it; the Performance-window context-usage instrumentation that
issue also asks for is separate follow-up.
Risk / rollout notes
4000. Existing saved entries longer than 1200 are normalized to 1200 (keeping
the start) and re-persisted on next launch (
normalizedExtendedContextruns atload). Foundation Models users who previously had up to 4000 chars fed in will
lose the tail beyond 1200; this is intentional for latency and consistency, and
is still strictly more than the OSS path ever used (~570).
reallocates within it so the user's notes are no longer the section that gets
starved.
Greptile Summary
This PR fixes a silent data-loss bug where the Extended Context field advertised a 4000-character cap but the OSS/llama rendering path hard-truncated notes at ~570 characters, meaning up to 85% of a user's entry was dropped on every request. The fix is a two-sided alignment: lower the model/UI cap to 1200 chars and raise the renderer's notes
maxCharsfrom 600 to 1300, so the full user-entered text (up to 1200 chars + 32-char label = 1232 chars) survives the prompt build step.SuggestionSettingsModel.swift:maximumExtendedContextCharacters4000 → 1200; the UI character-counter and tests both reference this constant dynamically, so they track the new value without further changes.BaseCompletionPromptRenderer.swift: notes sectionmaxChars600 → 1300; the invariantmaxChars ≥ cap + label_length(1300 ≥ 1232) holds and is documented via comment, but there is no programmatic assertion linking the two constants.normalizedExtendedContextpath, which also re-persists the shortened value.Confidence Score: 4/5
Safe to merge; the two changed constants are correctly aligned and all referencing code (UI counter, truncation logic, tests) already reads from the constant dynamically.
Both constants are correctly set (1200 cap, 1300 renderer budget, 1232 combined need), existing users' over-long entries are quietly truncated and re-persisted on load, and no schema change is required. The one outstanding gap is that the coupling between
maximumExtendedContextCharactersand the renderer'smaxCharsis enforced only through comments — a future independent edit to either value would silently reintroduce the clipping bug without any test catching it.BaseCompletionPromptRenderer.swift — the notes
maxCharsvalue has no test or assertion verifying it stays large enough relative to the model's cap.Important Files Changed
maximumExtendedContextCharactersfrom 4000 to 1200; UI character-count label and tests both reference the constant dynamically so they update automatically with no hardcoded references to change.maxCharsfrom 600 to 1300, covering the new 1200-char cap plus the 32-char label (1232 ≤ 1300); the invariant is documented via comment only with no runtime assertion or dedicated test.Sequence Diagram
sequenceDiagram participant User participant AdvancedPaneView participant SuggestionSettingsModel participant BaseCompletionPromptRenderer participant FoundationModelPromptRenderer User->>AdvancedPaneView: Pastes/types Extended Context AdvancedPaneView->>SuggestionSettingsModel: setExtendedContext(text) SuggestionSettingsModel->>SuggestionSettingsModel: normalizedExtendedContext() truncates to 1200 chars SuggestionSettingsModel-->>AdvancedPaneView: "extendedContext (<=1200 chars)" Note over AdvancedPaneView: Counter shows X / 1200 characters User->>BaseCompletionPromptRenderer: Request prompt (OSS path) BaseCompletionPromptRenderer->>BaseCompletionPromptRenderer: contextSection notes maxChars 1300 was 600 Note over BaseCompletionPromptRenderer: 1200 + 32-char label = 1232 <= 1300, full blob survives User->>FoundationModelPromptRenderer: Request prompt (FM/Apple path) FoundationModelPromptRenderer->>FoundationModelPromptRenderer: sessionInstructions() injects extendedContext directly Note over FoundationModelPromptRenderer: Smaller token footprint in Apple 4096-token windowReviews (1): Last reviewed commit: "Match Extended Context cap to what the e..." | Re-trigger Greptile