improve LLM judge prompt

Dylan Huang · Dylan Huang · commit cf45835789b8 · 2025-08-26T16:03:54.000-07:00
diff --git a/tests/chinook/test_pydantic_chinook.py b/tests/chinook/test_pydantic_chinook.py
@@ -13,7 +13,8 @@
 
 LLM_JUDGE_PROMPT = (
     "Your job is to compare the response to the expected answer.\n"
-    "If the response contains the same information as the expected answer, return 1.0.\n"
+    "The response will be a narrative report of the query results.\n"
+    "If the response contains the same or well summarized information as the expected answer, return 1.0.\n"
     "If the response does not contain the same information or is missing information, return 0.0."
 )
 

Original file line number	Diff line number	Diff line change
`@@ -13,7 +13,8 @@`
`13`	`13`
`14`	`14`	`LLM_JUDGE_PROMPT = (`
`15`	`15`	`"Your job is to compare the response to the expected answer.\n"`
`16`		`- "If the response contains the same information as the expected answer, return 1.0.\n"`
	`16`	`+ "The response will be a narrative report of the query results.\n"`
	`17`	`+ "If the response contains the same or well summarized information as the expected answer, return 1.0.\n"`
`17`	`18`	`"If the response does not contain the same information or is missing information, return 0.0."`
`18`	`19`	`)`
`19`	`20`