aymara_ai.resources.evals.runs

Classes

RunsResource
AsyncRunsResource

Module Contents

class aymara_ai.resources.evals.runs.RunsResource(client)

Bases: aymara_ai._resource.SyncAPIResource

Parameters:

client (aymara_ai._client.AymaraAI)

property with_raw_response: RunsResourceWithRawResponse

This property can be used as a prefix for any HTTP method call to return the raw response object instead of the parsed content.

For more information, see https://www.github.com/aymara-ai/aymara-sdk-python#accessing-raw-response-data-eg-headers

Return type:

RunsResourceWithRawResponse

property with_streaming_response: RunsResourceWithStreamingResponse

An alternative to .with_raw_response that doesn’t eagerly read the response body.

For more information, see https://www.github.com/aymara-ai/aymara-sdk-python#with_streaming_response

Return type:

RunsResourceWithStreamingResponse

create(*, eval_uuid, responses, is_sandbox=NOT_GIVEN, workspace_uuid=NOT_GIVEN, ai_description=NOT_GIVEN, continue_thread=NOT_GIVEN, eval_run_examples=NOT_GIVEN, eval_run_uuid=NOT_GIVEN, name=NOT_GIVEN, extra_headers=None, extra_query=None, extra_body=None, timeout=NOT_GIVEN)

Create a new eval run for an eval.

Args: eval_run_data (EvalRunRequest): Data for the eval run to create. workspace_uuid (str, optional): UUID of the workspace. Defaults to None. is_sandbox (bool, optional): Whether to run in sandbox mode. Defaults to None.

Returns: EvalRunResult: The created eval run result.

Raises: AymaraAPIError: If the organization is missing or the request is invalid.

Example: POST /api/eval-runs { “eval_uuid”: “…”, … }

Parameters:

• eval_uuid (str) – Unique identifier for the eval.

• responses (Iterable[aymara_ai.types.eval_response_param.EvalResponseParam]) – List of AI responses to eval prompts.

• ai_description (Optional[str] | aymara_ai._types.NotGiven) – Description of the AI for this run, if any.

• continue_thread (Optional[bool] | aymara_ai._types.NotGiven) – Whether to continue the thread after this run.

• eval_run_examples (Optional[Iterable[aymara_ai.types.evals.eval_run_example_param.EvalRunExampleParam]] | aymara_ai._types.NotGiven) – Examples to include with the eval run, if any.

• eval_run_uuid (Optional[str] | aymara_ai._types.NotGiven) – Unique identifier for the eval run, if any.

• name (Optional[str] | aymara_ai._types.NotGiven) – Name of the eval run, if any (defaults to the eval name + timestamp).

• extra_headers (aymara_ai._types.Headers | None) – Send extra headers

• extra_query (aymara_ai._types.Query | None) – Add additional query parameters to the request

• extra_body (aymara_ai._types.Body | None) – Add additional JSON properties to the request

• timeout (float | httpx.Timeout | None | aymara_ai._types.NotGiven) – Override the client-level default timeout for this request, in seconds

• is_sandbox (Optional[bool] | aymara_ai._types.NotGiven)

• workspace_uuid (str | aymara_ai._types.NotGiven)

Return type:

aymara_ai.types.evals.eval_run_result.EvalRunResult

list(*, eval_uuid=NOT_GIVEN, limit=NOT_GIVEN, offset=NOT_GIVEN, workspace_uuid=NOT_GIVEN, extra_headers=None, extra_query=None, extra_body=None, timeout=NOT_GIVEN)

List all eval runs, with optional filtering.

Args: eval_uuid (str, optional): UUID of the eval to filter runs by. Defaults to None. workspace_uuid (str, optional): UUID of the workspace. Defaults to None.

Returns: list[EvalRunResult]: List of eval runs matching the filters.

Raises: AymaraAPIError: If the organization is missing.

Example: GET /api/eval-runs?eval_uuid=…&workspace_uuid=…

Parameters:

• extra_headers (aymara_ai._types.Headers | None) – Send extra headers

• extra_query (aymara_ai._types.Query | None) – Add additional query parameters to the request

• extra_body (aymara_ai._types.Body | None) – Add additional JSON properties to the request

• timeout (float | httpx.Timeout | None | aymara_ai._types.NotGiven) – Override the client-level default timeout for this request, in seconds

• eval_uuid (str | aymara_ai._types.NotGiven)

• limit (int | aymara_ai._types.NotGiven)

• offset (int | aymara_ai._types.NotGiven)

• workspace_uuid (str | aymara_ai._types.NotGiven)

Return type:

aymara_ai.pagination.SyncOffsetPage[aymara_ai.types.evals.eval_run_result.EvalRunResult]

delete(eval_run_uuid, *, workspace_uuid=NOT_GIVEN, extra_headers=None, extra_query=None, extra_body=None, timeout=NOT_GIVEN)

Delete an eval run.

Args: eval_run_uuid (str): UUID of the eval run to delete.

workspace_uuid (str, optional): UUID of the workspace. Defaults to None.

Returns: None

Raises: AymaraAPIError: If the organization is missing or the eval run is not found.

Example: DELETE /api/eval-runs/{eval_run_uuid}

Parameters:

• extra_headers (aymara_ai._types.Headers | None) – Send extra headers

• extra_query (aymara_ai._types.Query | None) – Add additional query parameters to the request

• extra_body (aymara_ai._types.Body | None) – Add additional JSON properties to the request

• timeout (float | httpx.Timeout | None | aymara_ai._types.NotGiven) – Override the client-level default timeout for this request, in seconds

• eval_run_uuid (str)

• workspace_uuid (str | aymara_ai._types.NotGiven)

Return type:

None

get(eval_run_uuid, *, workspace_uuid=NOT_GIVEN, extra_headers=None, extra_query=None, extra_body=None, timeout=NOT_GIVEN)

Retrieve a specific eval run by its UUID.

Args: eval_run_uuid (str): UUID of the eval run to retrieve. workspace_uuid (str, optional): UUID of the workspace. Defaults to None.

Returns: EvalRunResult: The eval run data.

Raises: AymaraAPIError: If the organization is missing or the eval run is not found.

Example: GET /api/eval-runs/{eval_run_uuid}

Parameters:

• extra_headers (aymara_ai._types.Headers | None) – Send extra headers

• extra_query (aymara_ai._types.Query | None) – Add additional query parameters to the request

• extra_body (aymara_ai._types.Body | None) – Add additional JSON properties to the request

• timeout (float | httpx.Timeout | None | aymara_ai._types.NotGiven) – Override the client-level default timeout for this request, in seconds

• eval_run_uuid (str)

• workspace_uuid (str | aymara_ai._types.NotGiven)

Return type:

aymara_ai.types.evals.eval_run_result.EvalRunResult

list_responses(eval_run_uuid, *, limit=NOT_GIVEN, offset=NOT_GIVEN, workspace_uuid=NOT_GIVEN, extra_headers=None, extra_query=None, extra_body=None, timeout=NOT_GIVEN)

Retrieve all responses for a specific eval run.

Args: eval_run_uuid (str): UUID of the eval run to retrieve responses for. workspace_uuid (str, optional): UUID of the workspace. Defaults to None.

Returns: list[EvalScoredResponse]: List of scored responses for the eval run.

Raises: AymaraAPIError: If the organization is missing or the eval run is not found.

Example: GET /api/eval-runs/{eval_run_uuid}/responses

Parameters:

• extra_headers (aymara_ai._types.Headers | None) – Send extra headers

• extra_query (aymara_ai._types.Query | None) – Add additional query parameters to the request

• extra_body (aymara_ai._types.Body | None) – Add additional JSON properties to the request

• timeout (float | httpx.Timeout | None | aymara_ai._types.NotGiven) – Override the client-level default timeout for this request, in seconds

• eval_run_uuid (str)

• limit (int | aymara_ai._types.NotGiven)

• offset (int | aymara_ai._types.NotGiven)

• workspace_uuid (str | aymara_ai._types.NotGiven)

Return type:

aymara_ai.pagination.SyncOffsetPage[aymara_ai.types.evals.scored_response.ScoredResponse]

score_responses(*, eval_uuid, responses, is_sandbox=NOT_GIVEN, workspace_uuid=NOT_GIVEN, ai_description=NOT_GIVEN, continue_thread=NOT_GIVEN, eval_run_examples=NOT_GIVEN, eval_run_uuid=NOT_GIVEN, name=NOT_GIVEN, extra_headers=None, extra_query=None, extra_body=None, timeout=NOT_GIVEN)

Run the eval with the provided responses.

Args: eval_run_data (EvalRunRequest): Data for the eval run, including responses. workspace_uuid (str, optional): UUID of the workspace. Defaults to None. is_sandbox (bool, optional): Whether to run in sandbox mode. Defaults to False.

Returns: EvalRunResult: The result of the eval run after scoring the responses.

Raises: AymaraAPIError: If the organization is missing or the request is invalid.

Example: POST /api/eval-runs/-/score-responses { “eval_uuid”: “…”, “responses”: […] }

Parameters:

• eval_uuid (str) – Unique identifier for the eval.

• responses (Iterable[aymara_ai.types.eval_response_param.EvalResponseParam]) – List of AI responses to eval prompts.

• ai_description (Optional[str] | aymara_ai._types.NotGiven) – Description of the AI for this run, if any.

• continue_thread (Optional[bool] | aymara_ai._types.NotGiven) – Whether to continue the thread after this run.

• eval_run_examples (Optional[Iterable[aymara_ai.types.evals.eval_run_example_param.EvalRunExampleParam]] | aymara_ai._types.NotGiven) – Examples to include with the eval run, if any.

• eval_run_uuid (Optional[str] | aymara_ai._types.NotGiven) – Unique identifier for the eval run, if any.

• name (Optional[str] | aymara_ai._types.NotGiven) – Name of the eval run, if any (defaults to the eval name + timestamp).

• extra_headers (aymara_ai._types.Headers | None) – Send extra headers

• extra_query (aymara_ai._types.Query | None) – Add additional query parameters to the request

• extra_body (aymara_ai._types.Body | None) – Add additional JSON properties to the request

• timeout (float | httpx.Timeout | None | aymara_ai._types.NotGiven) – Override the client-level default timeout for this request, in seconds

• is_sandbox (bool | aymara_ai._types.NotGiven)

• workspace_uuid (str | aymara_ai._types.NotGiven)

Return type:

aymara_ai.types.evals.eval_run_result.EvalRunResult

class aymara_ai.resources.evals.runs.AsyncRunsResource(client)

Bases: aymara_ai._resource.AsyncAPIResource

Parameters:

client (aymara_ai._client.AsyncAymaraAI)

property with_raw_response: AsyncRunsResourceWithRawResponse

This property can be used as a prefix for any HTTP method call to return the raw response object instead of the parsed content.

For more information, see https://www.github.com/aymara-ai/aymara-sdk-python#accessing-raw-response-data-eg-headers

Return type:

AsyncRunsResourceWithRawResponse

property with_streaming_response: AsyncRunsResourceWithStreamingResponse

An alternative to .with_raw_response that doesn’t eagerly read the response body.

For more information, see https://www.github.com/aymara-ai/aymara-sdk-python#with_streaming_response

Return type:

AsyncRunsResourceWithStreamingResponse

async create(*, eval_uuid, responses, is_sandbox=NOT_GIVEN, workspace_uuid=NOT_GIVEN, ai_description=NOT_GIVEN, continue_thread=NOT_GIVEN, eval_run_examples=NOT_GIVEN, eval_run_uuid=NOT_GIVEN, name=NOT_GIVEN, extra_headers=None, extra_query=None, extra_body=None, timeout=NOT_GIVEN)

Create a new eval run for an eval.

Args: eval_run_data (EvalRunRequest): Data for the eval run to create. workspace_uuid (str, optional): UUID of the workspace. Defaults to None. is_sandbox (bool, optional): Whether to run in sandbox mode. Defaults to None.

Returns: EvalRunResult: The created eval run result.

Raises: AymaraAPIError: If the organization is missing or the request is invalid.

Example: POST /api/eval-runs { “eval_uuid”: “…”, … }

Parameters:

• eval_uuid (str) – Unique identifier for the eval.

• responses (Iterable[aymara_ai.types.eval_response_param.EvalResponseParam]) – List of AI responses to eval prompts.

• ai_description (Optional[str] | aymara_ai._types.NotGiven) – Description of the AI for this run, if any.

• continue_thread (Optional[bool] | aymara_ai._types.NotGiven) – Whether to continue the thread after this run.

• eval_run_examples (Optional[Iterable[aymara_ai.types.evals.eval_run_example_param.EvalRunExampleParam]] | aymara_ai._types.NotGiven) – Examples to include with the eval run, if any.

• eval_run_uuid (Optional[str] | aymara_ai._types.NotGiven) – Unique identifier for the eval run, if any.

• name (Optional[str] | aymara_ai._types.NotGiven) – Name of the eval run, if any (defaults to the eval name + timestamp).

• extra_headers (aymara_ai._types.Headers | None) – Send extra headers

• extra_query (aymara_ai._types.Query | None) – Add additional query parameters to the request

• extra_body (aymara_ai._types.Body | None) – Add additional JSON properties to the request

• timeout (float | httpx.Timeout | None | aymara_ai._types.NotGiven) – Override the client-level default timeout for this request, in seconds

• is_sandbox (Optional[bool] | aymara_ai._types.NotGiven)

• workspace_uuid (str | aymara_ai._types.NotGiven)

Return type:

aymara_ai.types.evals.eval_run_result.EvalRunResult

list(*, eval_uuid=NOT_GIVEN, limit=NOT_GIVEN, offset=NOT_GIVEN, workspace_uuid=NOT_GIVEN, extra_headers=None, extra_query=None, extra_body=None, timeout=NOT_GIVEN)

List all eval runs, with optional filtering.

Args: eval_uuid (str, optional): UUID of the eval to filter runs by. Defaults to None. workspace_uuid (str, optional): UUID of the workspace. Defaults to None.

Returns: list[EvalRunResult]: List of eval runs matching the filters.

Raises: AymaraAPIError: If the organization is missing.

Example: GET /api/eval-runs?eval_uuid=…&workspace_uuid=…

Parameters:

• extra_headers (aymara_ai._types.Headers | None) – Send extra headers

• extra_query (aymara_ai._types.Query | None) – Add additional query parameters to the request

• extra_body (aymara_ai._types.Body | None) – Add additional JSON properties to the request

• timeout (float | httpx.Timeout | None | aymara_ai._types.NotGiven) – Override the client-level default timeout for this request, in seconds

• eval_uuid (str | aymara_ai._types.NotGiven)

• limit (int | aymara_ai._types.NotGiven)

• offset (int | aymara_ai._types.NotGiven)

• workspace_uuid (str | aymara_ai._types.NotGiven)

Return type:

aymara_ai._base_client.AsyncPaginator[aymara_ai.types.evals.eval_run_result.EvalRunResult, aymara_ai.pagination.AsyncOffsetPage[aymara_ai.types.evals.eval_run_result.EvalRunResult]]

async delete(eval_run_uuid, *, workspace_uuid=NOT_GIVEN, extra_headers=None, extra_query=None, extra_body=None, timeout=NOT_GIVEN)

Delete an eval run.

Args: eval_run_uuid (str): UUID of the eval run to delete.

workspace_uuid (str, optional): UUID of the workspace. Defaults to None.

Returns: None

Raises: AymaraAPIError: If the organization is missing or the eval run is not found.

Example: DELETE /api/eval-runs/{eval_run_uuid}

Parameters:

• extra_headers (aymara_ai._types.Headers | None) – Send extra headers

• extra_query (aymara_ai._types.Query | None) – Add additional query parameters to the request

• extra_body (aymara_ai._types.Body | None) – Add additional JSON properties to the request

• timeout (float | httpx.Timeout | None | aymara_ai._types.NotGiven) – Override the client-level default timeout for this request, in seconds

• eval_run_uuid (str)

• workspace_uuid (str | aymara_ai._types.NotGiven)

Return type:

None

async get(eval_run_uuid, *, workspace_uuid=NOT_GIVEN, extra_headers=None, extra_query=None, extra_body=None, timeout=NOT_GIVEN)

Retrieve a specific eval run by its UUID.

Args: eval_run_uuid (str): UUID of the eval run to retrieve. workspace_uuid (str, optional): UUID of the workspace. Defaults to None.

Returns: EvalRunResult: The eval run data.

Raises: AymaraAPIError: If the organization is missing or the eval run is not found.

Example: GET /api/eval-runs/{eval_run_uuid}

Parameters:

• extra_headers (aymara_ai._types.Headers | None) – Send extra headers

• extra_query (aymara_ai._types.Query | None) – Add additional query parameters to the request

• extra_body (aymara_ai._types.Body | None) – Add additional JSON properties to the request

• timeout (float | httpx.Timeout | None | aymara_ai._types.NotGiven) – Override the client-level default timeout for this request, in seconds

• eval_run_uuid (str)

• workspace_uuid (str | aymara_ai._types.NotGiven)

Return type:

aymara_ai.types.evals.eval_run_result.EvalRunResult

list_responses(eval_run_uuid, *, limit=NOT_GIVEN, offset=NOT_GIVEN, workspace_uuid=NOT_GIVEN, extra_headers=None, extra_query=None, extra_body=None, timeout=NOT_GIVEN)

Retrieve all responses for a specific eval run.

Args: eval_run_uuid (str): UUID of the eval run to retrieve responses for. workspace_uuid (str, optional): UUID of the workspace. Defaults to None.

Returns: list[EvalScoredResponse]: List of scored responses for the eval run.

Raises: AymaraAPIError: If the organization is missing or the eval run is not found.

Example: GET /api/eval-runs/{eval_run_uuid}/responses

Parameters:

• extra_headers (aymara_ai._types.Headers | None) – Send extra headers

• extra_query (aymara_ai._types.Query | None) – Add additional query parameters to the request

• extra_body (aymara_ai._types.Body | None) – Add additional JSON properties to the request

• timeout (float | httpx.Timeout | None | aymara_ai._types.NotGiven) – Override the client-level default timeout for this request, in seconds

• eval_run_uuid (str)

• limit (int | aymara_ai._types.NotGiven)

• offset (int | aymara_ai._types.NotGiven)

• workspace_uuid (str | aymara_ai._types.NotGiven)

Return type:

aymara_ai._base_client.AsyncPaginator[aymara_ai.types.evals.scored_response.ScoredResponse, aymara_ai.pagination.AsyncOffsetPage[aymara_ai.types.evals.scored_response.ScoredResponse]]

async score_responses(*, eval_uuid, responses, is_sandbox=NOT_GIVEN, workspace_uuid=NOT_GIVEN, ai_description=NOT_GIVEN, continue_thread=NOT_GIVEN, eval_run_examples=NOT_GIVEN, eval_run_uuid=NOT_GIVEN, name=NOT_GIVEN, extra_headers=None, extra_query=None, extra_body=None, timeout=NOT_GIVEN)

Run the eval with the provided responses.

Args: eval_run_data (EvalRunRequest): Data for the eval run, including responses. workspace_uuid (str, optional): UUID of the workspace. Defaults to None. is_sandbox (bool, optional): Whether to run in sandbox mode. Defaults to False.

Returns: EvalRunResult: The result of the eval run after scoring the responses.

Raises: AymaraAPIError: If the organization is missing or the request is invalid.

Example: POST /api/eval-runs/-/score-responses { “eval_uuid”: “…”, “responses”: […] }

Parameters:

• eval_uuid (str) – Unique identifier for the eval.

• responses (Iterable[aymara_ai.types.eval_response_param.EvalResponseParam]) – List of AI responses to eval prompts.

• ai_description (Optional[str] | aymara_ai._types.NotGiven) – Description of the AI for this run, if any.

• continue_thread (Optional[bool] | aymara_ai._types.NotGiven) – Whether to continue the thread after this run.

• eval_run_examples (Optional[Iterable[aymara_ai.types.evals.eval_run_example_param.EvalRunExampleParam]] | aymara_ai._types.NotGiven) – Examples to include with the eval run, if any.

• eval_run_uuid (Optional[str] | aymara_ai._types.NotGiven) – Unique identifier for the eval run, if any.

• name (Optional[str] | aymara_ai._types.NotGiven) – Name of the eval run, if any (defaults to the eval name + timestamp).

• extra_headers (aymara_ai._types.Headers | None) – Send extra headers

• extra_query (aymara_ai._types.Query | None) – Add additional query parameters to the request

• extra_body (aymara_ai._types.Body | None) – Add additional JSON properties to the request

• timeout (float | httpx.Timeout | None | aymara_ai._types.NotGiven) – Override the client-level default timeout for this request, in seconds

• is_sandbox (bool | aymara_ai._types.NotGiven)

• workspace_uuid (str | aymara_ai._types.NotGiven)

Return type:

aymara_ai.types.evals.eval_run_result.EvalRunResult