aymara_ai.resources.evals.evals

aymara_ai.resources.evals.evals#

Classes#

EvalsResource

AsyncEvalsResource

Module Contents#

class aymara_ai.resources.evals.evals.EvalsResource(client)[source]#

Bases: aymara_ai._resource.SyncAPIResource

Parameters:

client (aymara_ai._client.AymaraAI)

property runs: aymara_ai.resources.evals.runs.RunsResource#
Return type:

aymara_ai.resources.evals.runs.RunsResource

property with_raw_response: EvalsResourceWithRawResponse#

This property can be used as a prefix for any HTTP method call to return the raw response object instead of the parsed content.

For more information, see https://www.github.com/aymara-ai/aymara-sdk-python#accessing-raw-response-data-eg-headers

Return type:

EvalsResourceWithRawResponse

property with_streaming_response: EvalsResourceWithStreamingResponse#

An alternative to .with_raw_response that doesn’t eagerly read the response body.

For more information, see https://www.github.com/aymara-ai/aymara-sdk-python#with_streaming_response

Return type:

EvalsResourceWithStreamingResponse

create(*, ai_description, eval_type, ai_instructions=NOT_GIVEN, created_at=NOT_GIVEN, eval_instructions=NOT_GIVEN, eval_uuid=NOT_GIVEN, ground_truth=NOT_GIVEN, is_jailbreak=NOT_GIVEN, is_sandbox=NOT_GIVEN, language=NOT_GIVEN, modality=NOT_GIVEN, name=NOT_GIVEN, num_prompts=NOT_GIVEN, prompt_examples=NOT_GIVEN, status=NOT_GIVEN, updated_at=NOT_GIVEN, workspace_uuid=NOT_GIVEN, extra_headers=None, extra_query=None, extra_body=None, timeout=NOT_GIVEN)[source]#

Create a new eval using an eval type configuration.

Args: eval_request (Eval): Data for the eval to create, including eval type and configuration.

Returns: Eval: The created eval object.

Raises: AymaraAPIError: If the workspace is not found or the request is invalid.

Example: POST /api/evals { “eval_type”: “…”, “workspace_uuid”: “…”, … }

Parameters:

  • ai_description (str) – Description of the AI under evaluation.

  • eval_type (str) – Type of the eval (safety, accuracy, etc.)

  • ai_instructions (Optional[str] | aymara_ai._types.NotGiven) – Instructions the AI should follow.

  • created_at (Union[str, datetime.datetime, None] | aymara_ai._types.NotGiven) – Timestamp when the eval was created.

  • eval_instructions (Optional[str] | aymara_ai._types.NotGiven) – Additional instructions for the eval, if any.

  • eval_uuid (Optional[str] | aymara_ai._types.NotGiven) – Unique identifier for the evaluation.

  • ground_truth (Optional[aymara_ai.types.eval_create_params.GroundTruth] | aymara_ai._types.NotGiven) – Ground truth data or reference file, if any.

  • is_jailbreak (bool | aymara_ai._types.NotGiven) – Indicates if the eval is a jailbreak test.

  • is_sandbox (bool | aymara_ai._types.NotGiven) – Indicates if the eval results are sandboxed.

  • language (Optional[str] | aymara_ai._types.NotGiven) – Language code for the eval (default: “en”).

  • modality (aymara_ai.types.shared.content_type.ContentType | aymara_ai._types.NotGiven) – Content type for AI interactions.

  • name (Optional[str] | aymara_ai._types.NotGiven) – Name of the evaluation.

  • num_prompts (Optional[int] | aymara_ai._types.NotGiven) – Number of prompts/questions in the eval (default: 50).

  • prompt_examples (Optional[Iterable[aymara_ai.types.prompt_example_param.PromptExampleParam]] | aymara_ai._types.NotGiven) – List of example prompts for the eval.

  • status (Optional[aymara_ai.types.shared.status.Status] | aymara_ai._types.NotGiven) – Resource status.

  • updated_at (Union[str, datetime.datetime, None] | aymara_ai._types.NotGiven) – Timestamp when the eval was last updated.

  • workspace_uuid (Optional[str] | aymara_ai._types.NotGiven) – UUID of the associated workspace, if any.

  • extra_headers (aymara_ai._types.Headers | None) – Send extra headers

  • extra_query (aymara_ai._types.Query | None) – Add additional query parameters to the request

  • extra_body (aymara_ai._types.Body | None) – Add additional JSON properties to the request

  • timeout (float | httpx.Timeout | None | aymara_ai._types.NotGiven) – Override the client-level default timeout for this request, in seconds

Return type:

aymara_ai.types.eval.Eval

list(*, limit=NOT_GIVEN, offset=NOT_GIVEN, workspace_uuid=NOT_GIVEN, extra_headers=None, extra_query=None, extra_body=None, timeout=NOT_GIVEN)[source]#

List all evals, with optional filtering.

Args: workspace_uuid (str, optional): Optional workspace UUID for filtering.

Returns: list[Eval]: List of evals matching the filter.

Raises: AymaraAPIError: If the request is invalid.

Example: GET /api/evals?workspace_uuid=…

Parameters:

  • extra_headers (aymara_ai._types.Headers | None) – Send extra headers

  • extra_query (aymara_ai._types.Query | None) – Add additional query parameters to the request

  • extra_body (aymara_ai._types.Body | None) – Add additional JSON properties to the request

  • timeout (float | httpx.Timeout | None | aymara_ai._types.NotGiven) – Override the client-level default timeout for this request, in seconds

  • limit (int | aymara_ai._types.NotGiven)

  • offset (int | aymara_ai._types.NotGiven)

  • workspace_uuid (str | aymara_ai._types.NotGiven)

Return type:

aymara_ai.pagination.SyncOffsetPage[aymara_ai.types.eval.Eval]

delete(eval_uuid, *, workspace_uuid=NOT_GIVEN, extra_headers=None, extra_query=None, extra_body=None, timeout=NOT_GIVEN)[source]#

Delete an eval.

Args: eval_uuid (str): UUID of the eval to delete.

workspace_uuid (str, optional): Optional workspace UUID for filtering.

Returns: None

Raises: AymaraAPIError: If the eval is not found.

Example: DELETE /api/evals/{eval_uuid}

Parameters:

  • extra_headers (aymara_ai._types.Headers | None) – Send extra headers

  • extra_query (aymara_ai._types.Query | None) – Add additional query parameters to the request

  • extra_body (aymara_ai._types.Body | None) – Add additional JSON properties to the request

  • timeout (float | httpx.Timeout | None | aymara_ai._types.NotGiven) – Override the client-level default timeout for this request, in seconds

  • eval_uuid (str)

  • workspace_uuid (str | aymara_ai._types.NotGiven)

Return type:

None

get(eval_uuid, *, workspace_uuid=NOT_GIVEN, extra_headers=None, extra_query=None, extra_body=None, timeout=NOT_GIVEN)[source]#

Retrieve a specific eval by its UUID.

Args: eval_uuid (str): UUID of the eval to retrieve. workspace_uuid (str, optional): Optional workspace UUID for filtering.

Returns: Eval: The eval data.

Raises: AymaraAPIError: If the eval is not found.

Example: GET /api/evals/{eval_uuid}

Parameters:

  • extra_headers (aymara_ai._types.Headers | None) – Send extra headers

  • extra_query (aymara_ai._types.Query | None) – Add additional query parameters to the request

  • extra_body (aymara_ai._types.Body | None) – Add additional JSON properties to the request

  • timeout (float | httpx.Timeout | None | aymara_ai._types.NotGiven) – Override the client-level default timeout for this request, in seconds

  • eval_uuid (str)

  • workspace_uuid (str | aymara_ai._types.NotGiven)

Return type:

aymara_ai.types.eval.Eval

list_prompts(eval_uuid, *, limit=NOT_GIVEN, offset=NOT_GIVEN, workspace_uuid=NOT_GIVEN, extra_headers=None, extra_query=None, extra_body=None, timeout=NOT_GIVEN)[source]#

Retrieve prompts for a specific eval if they exist.

Args: eval_uuid (str): UUID of the eval to get prompts for. workspace_uuid (str, optional): Optional workspace UUID for filtering.

Returns: list[EvalPrompt]: List of prompts and metadata for the eval.

Raises: AymaraAPIError: If the eval is not found.

Example: GET /api/evals/{eval_uuid}/prompts

Parameters:

  • extra_headers (aymara_ai._types.Headers | None) – Send extra headers

  • extra_query (aymara_ai._types.Query | None) – Add additional query parameters to the request

  • extra_body (aymara_ai._types.Body | None) – Add additional JSON properties to the request

  • timeout (float | httpx.Timeout | None | aymara_ai._types.NotGiven) – Override the client-level default timeout for this request, in seconds

  • eval_uuid (str)

  • limit (int | aymara_ai._types.NotGiven)

  • offset (int | aymara_ai._types.NotGiven)

  • workspace_uuid (str | aymara_ai._types.NotGiven)

Return type:

aymara_ai.pagination.SyncOffsetPage[aymara_ai.types.eval_prompt.EvalPrompt]

class aymara_ai.resources.evals.evals.AsyncEvalsResource(client)[source]#

Bases: aymara_ai._resource.AsyncAPIResource

Parameters:

client (aymara_ai._client.AsyncAymaraAI)

property runs: aymara_ai.resources.evals.runs.AsyncRunsResource#
Return type:

aymara_ai.resources.evals.runs.AsyncRunsResource

property with_raw_response: AsyncEvalsResourceWithRawResponse#

This property can be used as a prefix for any HTTP method call to return the raw response object instead of the parsed content.

For more information, see https://www.github.com/aymara-ai/aymara-sdk-python#accessing-raw-response-data-eg-headers

Return type:

AsyncEvalsResourceWithRawResponse

property with_streaming_response: AsyncEvalsResourceWithStreamingResponse#

An alternative to .with_raw_response that doesn’t eagerly read the response body.

For more information, see https://www.github.com/aymara-ai/aymara-sdk-python#with_streaming_response

Return type:

AsyncEvalsResourceWithStreamingResponse

async create(*, ai_description, eval_type, ai_instructions=NOT_GIVEN, created_at=NOT_GIVEN, eval_instructions=NOT_GIVEN, eval_uuid=NOT_GIVEN, ground_truth=NOT_GIVEN, is_jailbreak=NOT_GIVEN, is_sandbox=NOT_GIVEN, language=NOT_GIVEN, modality=NOT_GIVEN, name=NOT_GIVEN, num_prompts=NOT_GIVEN, prompt_examples=NOT_GIVEN, status=NOT_GIVEN, updated_at=NOT_GIVEN, workspace_uuid=NOT_GIVEN, extra_headers=None, extra_query=None, extra_body=None, timeout=NOT_GIVEN)[source]#

Create a new eval using an eval type configuration.

Args: eval_request (Eval): Data for the eval to create, including eval type and configuration.

Returns: Eval: The created eval object.

Raises: AymaraAPIError: If the workspace is not found or the request is invalid.

Example: POST /api/evals { “eval_type”: “…”, “workspace_uuid”: “…”, … }

Parameters:

  • ai_description (str) – Description of the AI under evaluation.

  • eval_type (str) – Type of the eval (safety, accuracy, etc.)

  • ai_instructions (Optional[str] | aymara_ai._types.NotGiven) – Instructions the AI should follow.

  • created_at (Union[str, datetime.datetime, None] | aymara_ai._types.NotGiven) – Timestamp when the eval was created.

  • eval_instructions (Optional[str] | aymara_ai._types.NotGiven) – Additional instructions for the eval, if any.

  • eval_uuid (Optional[str] | aymara_ai._types.NotGiven) – Unique identifier for the evaluation.

  • ground_truth (Optional[aymara_ai.types.eval_create_params.GroundTruth] | aymara_ai._types.NotGiven) – Ground truth data or reference file, if any.

  • is_jailbreak (bool | aymara_ai._types.NotGiven) – Indicates if the eval is a jailbreak test.

  • is_sandbox (bool | aymara_ai._types.NotGiven) – Indicates if the eval results are sandboxed.

  • language (Optional[str] | aymara_ai._types.NotGiven) – Language code for the eval (default: “en”).

  • modality (aymara_ai.types.shared.content_type.ContentType | aymara_ai._types.NotGiven) – Content type for AI interactions.

  • name (Optional[str] | aymara_ai._types.NotGiven) – Name of the evaluation.

  • num_prompts (Optional[int] | aymara_ai._types.NotGiven) – Number of prompts/questions in the eval (default: 50).

  • prompt_examples (Optional[Iterable[aymara_ai.types.prompt_example_param.PromptExampleParam]] | aymara_ai._types.NotGiven) – List of example prompts for the eval.

  • status (Optional[aymara_ai.types.shared.status.Status] | aymara_ai._types.NotGiven) – Resource status.

  • updated_at (Union[str, datetime.datetime, None] | aymara_ai._types.NotGiven) – Timestamp when the eval was last updated.

  • workspace_uuid (Optional[str] | aymara_ai._types.NotGiven) – UUID of the associated workspace, if any.

  • extra_headers (aymara_ai._types.Headers | None) – Send extra headers

  • extra_query (aymara_ai._types.Query | None) – Add additional query parameters to the request

  • extra_body (aymara_ai._types.Body | None) – Add additional JSON properties to the request

  • timeout (float | httpx.Timeout | None | aymara_ai._types.NotGiven) – Override the client-level default timeout for this request, in seconds

Return type:

aymara_ai.types.eval.Eval

list(*, limit=NOT_GIVEN, offset=NOT_GIVEN, workspace_uuid=NOT_GIVEN, extra_headers=None, extra_query=None, extra_body=None, timeout=NOT_GIVEN)[source]#

List all evals, with optional filtering.

Args: workspace_uuid (str, optional): Optional workspace UUID for filtering.

Returns: list[Eval]: List of evals matching the filter.

Raises: AymaraAPIError: If the request is invalid.

Example: GET /api/evals?workspace_uuid=…

Parameters:

  • extra_headers (aymara_ai._types.Headers | None) – Send extra headers

  • extra_query (aymara_ai._types.Query | None) – Add additional query parameters to the request

  • extra_body (aymara_ai._types.Body | None) – Add additional JSON properties to the request

  • timeout (float | httpx.Timeout | None | aymara_ai._types.NotGiven) – Override the client-level default timeout for this request, in seconds

  • limit (int | aymara_ai._types.NotGiven)

  • offset (int | aymara_ai._types.NotGiven)

  • workspace_uuid (str | aymara_ai._types.NotGiven)

Return type:

aymara_ai._base_client.AsyncPaginator[aymara_ai.types.eval.Eval, aymara_ai.pagination.AsyncOffsetPage[aymara_ai.types.eval.Eval]]

async delete(eval_uuid, *, workspace_uuid=NOT_GIVEN, extra_headers=None, extra_query=None, extra_body=None, timeout=NOT_GIVEN)[source]#

Delete an eval.

Args: eval_uuid (str): UUID of the eval to delete.

workspace_uuid (str, optional): Optional workspace UUID for filtering.

Returns: None

Raises: AymaraAPIError: If the eval is not found.

Example: DELETE /api/evals/{eval_uuid}

Parameters:

  • extra_headers (aymara_ai._types.Headers | None) – Send extra headers

  • extra_query (aymara_ai._types.Query | None) – Add additional query parameters to the request

  • extra_body (aymara_ai._types.Body | None) – Add additional JSON properties to the request

  • timeout (float | httpx.Timeout | None | aymara_ai._types.NotGiven) – Override the client-level default timeout for this request, in seconds

  • eval_uuid (str)

  • workspace_uuid (str | aymara_ai._types.NotGiven)

Return type:

None

async get(eval_uuid, *, workspace_uuid=NOT_GIVEN, extra_headers=None, extra_query=None, extra_body=None, timeout=NOT_GIVEN)[source]#

Retrieve a specific eval by its UUID.

Args: eval_uuid (str): UUID of the eval to retrieve. workspace_uuid (str, optional): Optional workspace UUID for filtering.

Returns: Eval: The eval data.

Raises: AymaraAPIError: If the eval is not found.

Example: GET /api/evals/{eval_uuid}

Parameters:

  • extra_headers (aymara_ai._types.Headers | None) – Send extra headers

  • extra_query (aymara_ai._types.Query | None) – Add additional query parameters to the request

  • extra_body (aymara_ai._types.Body | None) – Add additional JSON properties to the request

  • timeout (float | httpx.Timeout | None | aymara_ai._types.NotGiven) – Override the client-level default timeout for this request, in seconds

  • eval_uuid (str)

  • workspace_uuid (str | aymara_ai._types.NotGiven)

Return type:

aymara_ai.types.eval.Eval

list_prompts(eval_uuid, *, limit=NOT_GIVEN, offset=NOT_GIVEN, workspace_uuid=NOT_GIVEN, extra_headers=None, extra_query=None, extra_body=None, timeout=NOT_GIVEN)[source]#

Retrieve prompts for a specific eval if they exist.

Args: eval_uuid (str): UUID of the eval to get prompts for. workspace_uuid (str, optional): Optional workspace UUID for filtering.

Returns: list[EvalPrompt]: List of prompts and metadata for the eval.

Raises: AymaraAPIError: If the eval is not found.

Example: GET /api/evals/{eval_uuid}/prompts

Parameters:

  • extra_headers (aymara_ai._types.Headers | None) – Send extra headers

  • extra_query (aymara_ai._types.Query | None) – Add additional query parameters to the request

  • extra_body (aymara_ai._types.Body | None) – Add additional JSON properties to the request

  • timeout (float | httpx.Timeout | None | aymara_ai._types.NotGiven) – Override the client-level default timeout for this request, in seconds

  • eval_uuid (str)

  • limit (int | aymara_ai._types.NotGiven)

  • offset (int | aymara_ai._types.NotGiven)

  • workspace_uuid (str | aymara_ai._types.NotGiven)

Return type:

aymara_ai._base_client.AsyncPaginator[aymara_ai.types.eval_prompt.EvalPrompt, aymara_ai.pagination.AsyncOffsetPage[aymara_ai.types.eval_prompt.EvalPrompt]]

Contents