Skip to content

Align run utility signature with python sdk #13

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
andreibratu merged 2 commits into from
Feb 13, 2025
Merged

Align run utility signature with python sdk #13

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
b10773c
Align run utility signature with python sdk
Feb 13, 2025
d4d9f82
allow online evaluators in run utility
Feb 13, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion src/eval_utils/run.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -285,7 +285,9 @@ export async function runEval(
// Upsert the local Evaluators; other Evaluators are just referenced by `path` or `id`
let localEvaluators: [EvaluatorResponse, Function][] = [];
if (evaluators) {
const evaluatorsWithCallable = evaluators.filter((e) => e.callable !== null);
const evaluatorsWithCallable = evaluators.filter(
(e) => e.callable !== undefined,
);
Comment on lines +288 to +290
Copy link
Contributor

@jamesbaskerville jamesbaskerville Feb 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this could potentially break existing callers, right? What if someone is explicitly passing callable: null?

Copy link
Author

@andreibratu andreibratu Feb 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

image Screenshot 2025-02-14 at 09 16 41

I think it's well handled, the problem was that we could not specify an online evaluator


if (evaluatorsWithCallable.length > 0 && function_ == null) {
throw new Error(
Expand Down
6 changes: 4 additions & 2 deletions src/eval_utils/types.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,8 +90,10 @@ export interface Evaluator extends Identifiers {
returnType?: EvaluatorReturnTypeEnum;
/**The threshold to check the Evaluator against. If the aggregate value of the Evaluator is below this threshold, the check will fail.*/
threshold?: number;
callable: Function;
argsType: EvaluatorArgumentsType;
/**The function to run on the logs to produce the judgment - only required for local Evaluators.*/
callable?: Function;
/**The type of arguments the Evaluator expects - only required for local Evaluators.*/
argsType?: EvaluatorArgumentsType;
Comment on lines +94 to +96
Copy link
Contributor

@jamesbaskerville jamesbaskerville Feb 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suspect we should/could do some stronger typing here where we can statically check if a passed evaluator is either a remote evaluator (path/id) or local one.

Copy link
Author

@andreibratu andreibratu Feb 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

true, evaluator could be a union where you either provide a path or {callable, argsType, returnType}

Copy link
Author

@andreibratu andreibratu Feb 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

image

}

export interface TargetFreeEvaluator extends Evaluator {
Expand Down