aymara_ai.lib.runner

aymara_ai.lib.runner#

EvalRunner & AsyncEvalRunner: Simple orchestrators for Aymara SDK evals (sync and async).

Classes#

EvalRunner

Orchestrates the evaluation process for Aymara SDK using a user-provided model callable.

AsyncEvalRunner

Orchestrates the evaluation process for Aymara SDK using a user-provided async model callable.

Module Contents#

class aymara_ai.lib.runner.EvalRunner(client, model_callable)#

Orchestrates the evaluation process for Aymara SDK using a user-provided model callable. Stores all state internally. For synchronous (blocking) use with AymaraClient.

Parameters:

  • client (aymara_ai.AymaraAI)

  • model_callable (Callable[[str], Union[str, pathlib.Path, None]])

client#
model_callable#
eval_id: str | None = None#
run_id: str | None = None#
eval_run_result: aymara_ai.types.evals.eval_run_result.EvalRunResult | None = None#
run_eval(eval_params)#

Orchestrate the full eval flow (sync).

Parameters:

eval_params (Union[aymara_ai.types.eval.Eval, Dict[str, Any]])

Return type:

aymara_ai.types.evals.eval_run_result.EvalRunResult

class aymara_ai.lib.runner.AsyncEvalRunner(client, model_callable)#

Orchestrates the evaluation process for Aymara SDK using a user-provided async model callable. Stores all state internally. For asynchronous use with AsyncAymaraClient.

Parameters:

  • client (aymara_ai.AsyncAymaraAI)

  • model_callable (Callable[[str], Awaitable[Union[str, pathlib.Path, None]]])

client#
model_callable#
eval_id: str | None = None#
run_id: str | None = None#
eval_run_result: aymara_ai.types.evals.eval_run_result.EvalRunResult | None = None#
async run_eval(eval_params, timeout=30, poll_interval=5)#

Orchestrate the full eval flow (async).

Parameters:
Return type:

aymara_ai.types.evals.eval_run_result.EvalRunResult

Contents