add benchmark.py module#56
Conversation
mouayadkarimeh
left a comment
mouayadkarimeh
left a comment
There was a problem hiding this comment.
The latest commit contains both pre-commit fixes and the increased ruff complexity limit simultaneously. It's not just the complexity increase — the entire benchmark.py file has been prepared and is ready for review.
| def remove_think_tags(text: str) -> str: | ||
| """Remove think-tags from text. | ||
|
|
||
| Parameter: | ||
| text (str): The input text containing think-tags. | ||
|
|
||
| Returns: | ||
| str: The text with think-tags removed. | ||
| """ | ||
| think_tag_re = re.compile(r"</?\s*think\s*>", re.IGNORECASE) | ||
| return think_tag_re.sub("", text) |
There was a problem hiding this comment.
What about removing the full think block instead of just the tags?
There was a problem hiding this comment.
Now its removed
| re.IGNORECASE, | ||
| ) | ||
|
|
||
| RELEASE_DATE_RE = re.compile( |
There was a problem hiding this comment.
Are those not the same as recording date?
There was a problem hiding this comment.
Combining RECORDING_DATE_RE and RELEASING_DATE_RE IN RECORDING_RELEASING_DATE_RE
in funktion extract_date_from_text based on the qwustion wheter "Wann hat der Patient bei uns aufgenommen" or "Wann wurde der Patient bei uns entlassen"
| ) | ||
|
|
||
|
|
||
| def extract_release_date_from_text(text: str) -> str: |
There was a problem hiding this comment.
This is also similar to extract_record_data. whatabout putting both together in one function?
There was a problem hiding this comment.
combined in extract_recording_releasing_date_from_text()
| _LOGGER.info("Ergebnisse gespeichert in: %s", RESULT_CSV_WITHOUT_RAG_ONE_FILE) | ||
|
|
||
|
|
||
| def profile(file_path: Path, use_rag: bool, no_think: bool) -> None: |
There was a problem hiding this comment.
Is it doing the same as run_bmk?
There was a problem hiding this comment.
remove run_bmk_onefile(). because profile() does the same work and returned the clean answers
aktuel:
run_bmk(): run benchmark for a number of files.
profile(): run benchmark for one file and demonstrate the clean answers.
| # run_bmk(max_files = 5, use_rag=True, no_think_option= True) | ||
| # run_bmk_one_file(file_path=RAW_TEXT_DIR / "Cajal.txt" | ||
| # , use_rag=True, no_think_option=False) | ||
| profile(file_path=RAW_TEXT_DIR / "Cajal.txt", use_rag=True, no_think=True) |
There was a problem hiding this comment.
Suggestion: Instead of using a global variable you could parse those arguments from the command line (with useful defaults)?
There was a problem hiding this comment.
add Argumentparse in CLI. diffrent examples how to run profil() in the CLI are provided as comments in main section.
Examples
-
run uv benchmark.py --file-path RAW/Cajal.txt --use-rag --no-think =
run benchmark Cajal.txt, use-rag = True, no-think = True -
run uv benchmark.py --file-path RAW/Cajal.txt --use-rag =
run benchmark for Cajal.txt, use-rag = True, no-think = False -
run uv benchmark.py --file-path RAW/Cajal.txt --no-think =
run benchmark for Cajal.txt, use-rag = False, no-think = True -
run uv benchmark.py --file-path RAW/Cajal.txt =
run benchmark Cajal.txt, use-rag = False, no-think = False
|
@mouayadkarimeh thanks for updating the pr. There are still some linting issues (besides my comments above). |
…run ruff check- run test- There are some history errors that has nothing to do with benchmark.py.
| p.add_argument( | ||
| "--file-path", | ||
| dest="file_path", | ||
| help=f"Path to input text file (default: {RAW / 'Cajal.txt'})", |
There was a problem hiding this comment.
Where do you set this default (and why not here?)
| # run uv benchmark.py --file-path RAW/Cajal.txt --use-rag | ||
| # run uv benchmark.py --file-path RAW/Cajal.txt --no-think | ||
| # run uv benchmark.py --file-path RAW/Cajal.txt | ||
| profile(file_path=fp, use_rag=bool(args.use_rag), no_think=bool(args.no_think)) |
There was a problem hiding this comment.
does it make sense to call profile if file_path is a single file and run_bmk if it is a folder?
2 participants
Features
Model Comparison: Benchmarks reasoning models vs. non-reasoning models
No-Think Option Support: Integrates with the /no_think parameter to test different model behaviors
RAG Integration: Supports benchmarking with and without RAG (Retrieval-Augmented Generation) concept
Work in Progress
Answer Comparison: Implementation for comparing generated answers and calculating accuracy percentages
Profile Function: Optimization of the profile() function for more reliable result demonstration and output formatting