LLM Experiments

[robertomonteromiguel]

Motivation

Test LLM Experiment to evaluate the system-tests repo prompts

Changes

Workflow

1. ⚠️ Create your PR as draft ⚠️
2. Work on you PR until the CI passes
3. Mark it as ready for review
   • Test logic is modified? -> Get a review from RFC owner.
   • Framework is modified, or non obvious usage of it -> get a review from R&P team

🚀 Once your PR is reviewed and the CI green, you can merge it!

🛟 #apm-shared-testing 🛟

Reviewer checklist

• Anything but tests/ or manifests/ is modified ? I have the approval from R&P team
• A docker base image is modified?
  • the relevant build-XXX-image label is present
• A scenario is added, removed or renamed?
  • Get a review from R&P team

[github-actions]

CODEOWNERS have been resolved as:

questions.csv                                                           @DataDog/system-tests-core
tests/llm_experiments/__init__.py                                       @DataDog/system-tests-core
tests/llm_experiments/evaluate_system_tests_prompts.py                  @DataDog/system-tests-core
requirements.txt                                                        @DataDog/system-tests-core

[yahya-mouman]

Why not use from ddtrace.llmobs._evaluators import SemanticSimilarityEvaluator ?

[yahya-mouman]

You should probably try to pull the dataset before creating it to make the file re-runnable.