[Design Doc] Muhammad Maaz — Section 3: Technical Approachadd draft technical approach section for factual_grounding, relevance…#1
Closed
Maazkorejo wants to merge 2 commits into
Conversation
…, refusal_check, completeness
Corrected formatting and grammatical inconsistencies in the technical approach documentation, ensuring clarity and consistency in the presentation of methods and implications.
Collaborator
Author
|
Closing this in favor of the combined design doc — team decided to consolidate all sections into one file instead of separate PRs per section. Section 3 content has been carried over intact: see #2 (or whatever PR number the combined one gets). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Adds a draft of Section 3 (Technical Approach) for the design document, covering the technical approach for all four evaluation criteria I own: factual_grounding, relevance, refusal_check, and completeness.
Which module(s) does it touch?
docs/section3_technical_approach.md — new file, no existing code touched. This is a design document draft, not an implementation PR; no source modules exist yet per the Week 1 no-code rule.
How did you test it?
No automated tests apply since this is documentation, not code. The claims in this section are backed by manual verification: I ran sentence-transformers (all-MiniLM-L6-v2) locally against two hand-picked (response, context) pairs — a paraphrase and a factual contradiction — and included the actual cosine similarity scores (0.982 and 0.822) in the appendix, rather than assuming the model's behavior.
Edge cases handled:
The document itself identifies edge cases the eventual implementation will need to handle: numeric/factual contradictions that still score high on cosine similarity (verified with real data), refusal-adjacent phrases like "I cannot stress enough how important this is" that shouldn't trigger refusal_check, and the more open-ended limitations of prompt-decomposition for completeness. These are documented as known risks, not yet solved in code.
Checklist: