aymara_ai.resources.evals.evals

aymara_ai.resources.evals.evals#

Classes#

`EvalsResource`
`AsyncEvalsResource`

Module Contents#

class aymara_ai.resources.evals.evals.EvalsResource(client)[source]#

Bases: aymara_ai._resource.SyncAPIResource

Parameters:: client (aymara_ai._client.AymaraAI)

property runs: aymara_ai.resources.evals.runs.RunsResource#

Return type:: aymara_ai.resources.evals.runs.RunsResource

property with_raw_response: EvalsResourceWithRawResponse#

This property can be used as a prefix for any HTTP method call to return the raw response object instead of the parsed content.

For more information, see https://www.github.com/aymara-ai/aymara-sdk-python#accessing-raw-response-data-eg-headers

Return type:: EvalsResourceWithRawResponse

property with_streaming_response: EvalsResourceWithStreamingResponse#

An alternative to .with_raw_response that doesn’t eagerly read the response body.

For more information, see https://www.github.com/aymara-ai/aymara-sdk-python#with_streaming_response

Return type:: EvalsResourceWithStreamingResponse

create(*, ai_description, eval_type, ai_instructions=omit, created_at=omit, created_by=omit, eval_instructions=omit, eval_uuid=omit, ground_truth=omit, is_jailbreak=omit, is_sandbox=omit, language=omit, modality=omit, name=omit, num_prompts=omit, prompt_examples=omit, status=omit, updated_at=omit, workspace_uuid=omit, extra_headers=None, extra_query=None, extra_body=None, timeout=not_given)[source]#

Create a new eval using an eval type configuration.

Args: eval_request (Eval): Data for the eval to create, including eval type and configuration.

Returns: Eval: The created eval object.

Raises: AymaraAPIError: If the request is invalid.

Example: POST /api/evals { “eval_type”: “…”, “workspace_uuid”: “…”, … }

Parameters:

ai_description (str) – Description of the AI under evaluation.
eval_type (str) – Type of the eval (safety, accuracy, etc.)
ai_instructions (Optional[str] | aymara_ai._types.Omit) – Instructions the AI should follow.
created_at (Union[str, datetime.datetime, None] | aymara_ai._types.Omit) – Timestamp when the eval was created.
created_by (Optional[str] | aymara_ai._types.Omit) – Name of the user who created the evaluation.
eval_instructions (Optional[str] | aymara_ai._types.Omit) – Additional instructions for the eval, if any.
eval_uuid (Optional[str] | aymara_ai._types.Omit) – Unique identifier for the evaluation.
ground_truth (Optional[aymara_ai.types.eval_create_params.GroundTruth] | aymara_ai._types.Omit) – Ground truth data or reference file, if any.
is_jailbreak (bool | aymara_ai._types.Omit) – Indicates if the eval is a jailbreak test.
is_sandbox (bool | aymara_ai._types.Omit) – Indicates if the eval results are sandboxed.
language (Optional[str] | aymara_ai._types.Omit) – Language code for the eval (default: “en”).
modality (aymara_ai.types.shared.content_type.ContentType | aymara_ai._types.Omit) – Content type for AI interactions.
name (Optional[str] | aymara_ai._types.Omit) – Name of the evaluation.
num_prompts (Optional[int] | aymara_ai._types.Omit) – Number of prompts/questions in the eval (default: 50).
prompt_examples (Optional[Iterable[aymara_ai.types.prompt_example_param.PromptExampleParam]] | aymara_ai._types.Omit) – List of example prompts for the eval.
status (Optional[aymara_ai.types.shared.status.Status] | aymara_ai._types.Omit) – Resource status.
updated_at (Union[str, datetime.datetime, None] | aymara_ai._types.Omit) – Timestamp when the eval was last updated.
workspace_uuid (Optional[str] | aymara_ai._types.Omit) – UUID of the associated workspace, if any.
extra_headers (aymara_ai._types.Headers | None) – Send extra headers
extra_query (aymara_ai._types.Query | None) – Add additional query parameters to the request
extra_body (aymara_ai._types.Body | None) – Add additional JSON properties to the request
timeout (float | httpx.Timeout | None | aymara_ai._types.NotGiven) – Override the client-level default timeout for this request, in seconds

Return type:

aymara_ai.types.eval.Eval

update(eval_uuid, *, workspace_uuid=omit, ai_description=omit, ai_instructions=omit, eval_instructions=omit, ground_truth=omit, name=omit, prompt_creates=omit, prompt_updates=omit, extra_headers=None, extra_query=None, extra_body=None, timeout=not_given)[source]#

Update an existing eval’s fields and prompts.

Args: eval_uuid (str): UUID of the eval to update. update_request (EvalUpdateRequest): Update data including fields and prompt modifications. workspace_uuid (str, optional): Optional workspace UUID for filtering.

Returns: Eval: The updated eval data.

Raises: AymaraAPIError: If the eval is not found or update is invalid.

Example: PUT /api/evals/{eval_uuid} { “name”: “Updated Eval Name”, “ai_description”: “Updated description”, “prompt_updates”: [ {“prompt_uuid”: “…”, “content”: “New content”, “action”: “update”}, {“prompt_uuid”: “…”, “action”: “delete”} ], “prompt_creates”: [ {“content”: “New prompt”, “category”: “test”} ] }

Parameters:

ai_description (Optional[str] | aymara_ai._types.Omit) – New description of the AI under evaluation.
ai_instructions (Optional[str] | aymara_ai._types.Omit) – New instructions the AI should follow.
eval_instructions (Optional[str] | aymara_ai._types.Omit) – New additional instructions for the eval.
ground_truth (Optional[aymara_ai.types.eval_update_params.GroundTruth] | aymara_ai._types.Omit) – New ground truth data or reference file.
name (Optional[str] | aymara_ai._types.Omit) – New name for the evaluation.
prompt_creates (Optional[Iterable[aymara_ai.types.eval_update_params.PromptCreate]] | aymara_ai._types.Omit) – List of new prompts to add.
prompt_updates (Optional[Iterable[aymara_ai.types.eval_update_params.PromptUpdate]] | aymara_ai._types.Omit) – List of prompt updates to apply.
extra_headers (aymara_ai._types.Headers | None) – Send extra headers
extra_query (aymara_ai._types.Query | None) – Add additional query parameters to the request
extra_body (aymara_ai._types.Body | None) – Add additional JSON properties to the request
timeout (float | httpx.Timeout | None | aymara_ai._types.NotGiven) – Override the client-level default timeout for this request, in seconds
eval_uuid (str)
workspace_uuid (str | aymara_ai._types.Omit)

Return type:

aymara_ai.types.eval.Eval

list(*, limit=omit, offset=omit, workspace_uuid=omit, extra_headers=None, extra_query=None, extra_body=None, timeout=not_given)[source]#

List all evals, with optional filtering.

Args: workspace_uuid (str, optional): Optional workspace UUID for filtering. Use “**” for enterprise-wide access, omit for user’s current workspace.

Returns: list[Eval]: List of evals matching the filter.

Raises: AymaraAPIError: If the request is invalid.

Example: GET /api/evals?workspace_uuid=…

Parameters:

extra_headers (aymara_ai._types.Headers | None) – Send extra headers
extra_query (aymara_ai._types.Query | None) – Add additional query parameters to the request
extra_body (aymara_ai._types.Body | None) – Add additional JSON properties to the request
timeout (float | httpx.Timeout | None | aymara_ai._types.NotGiven) – Override the client-level default timeout for this request, in seconds
limit (int | aymara_ai._types.Omit)
offset (int | aymara_ai._types.Omit)
workspace_uuid (str | aymara_ai._types.Omit)

Return type:

aymara_ai.pagination.SyncOffsetPage[aymara_ai.types.eval.Eval]

delete(eval_uuid, *, workspace_uuid=omit, extra_headers=None, extra_query=None, extra_body=None, timeout=not_given)[source]#

Delete an eval.

Args: eval_uuid (str): UUID of the eval to delete.

workspace_uuid (str, optional): Optional workspace UUID for filtering.

Returns: None

Raises: AymaraAPIError: If the eval is not found.

Example: DELETE /api/evals/{eval_uuid}

Parameters:

extra_headers (aymara_ai._types.Headers | None) – Send extra headers
extra_query (aymara_ai._types.Query | None) – Add additional query parameters to the request
extra_body (aymara_ai._types.Body | None) – Add additional JSON properties to the request
timeout (float | httpx.Timeout | None | aymara_ai._types.NotGiven) – Override the client-level default timeout for this request, in seconds
eval_uuid (str)
workspace_uuid (str | aymara_ai._types.Omit)

Return type:

None

analyze(*, created_after=omit, created_before=omit, created_by=omit, eval_type=omit, has_score_runs=omit, is_jailbreak=omit, is_sandbox=omit, language=omit, limit=omit, max_pass_rate=omit, min_pass_rate=omit, modality=omit, name=omit, offset=omit, run_created_after=omit, run_created_before=omit, score_run_status=omit, sort_by=omit, sort_order=omit, status=omit, workspace_uuid=omit, extra_headers=None, extra_query=None, extra_body=None, timeout=not_given)[source]#

Analysis for evals with advanced filtering and aggregated statistics.

This endpoint allows analyzing across both eval metadata and score run performance data, providing comprehensive filtering capabilities and aggregated statistics for each eval.

Args: analysis_request (EvalAnalysisRequest): Analysis parameters and filters including: - Eval metadata filters (name, type, status, language, etc.) - Score run performance filters (pass rate, run count, etc.) - Sorting and pagination options

Returns: EvalAnalysisResponse: Paginated results with matching evals and their statistics

Raises: AymaraAPIError: If the request is invalid or analysis parameters are malformed

Example: POST /api/v2/eval_analysis { “name”: “safety”, “eval_type”: “safety”, “min_pass_rate”: 0.8, “has_score_runs”: true, “sort_by”: “pass_rate”, “sort_order”: “desc”, “limit”: 20, “offset”: 0 }

Parameters:

created_after (Union[str, datetime.datetime, None] | aymara_ai._types.Omit) – Filter evals created after this date
created_before (Union[str, datetime.datetime, None] | aymara_ai._types.Omit) – Filter evals created before this date
created_by (Optional[str] | aymara_ai._types.Omit) – Filter by creator email
eval_type (Optional[str] | aymara_ai._types.Omit) – Filter by eval type (safety, accuracy, jailbreak, image_safety)
has_score_runs (Optional[bool] | aymara_ai._types.Omit) – Only include evals that have score runs
is_jailbreak (Optional[bool] | aymara_ai._types.Omit) – Filter by jailbreak status
is_sandbox (Optional[bool] | aymara_ai._types.Omit) – Filter by sandbox status
language (Optional[str] | aymara_ai._types.Omit) – Filter by language code (e.g., en, es)
limit (int | aymara_ai._types.Omit) – Maximum number of results (1-100)
max_pass_rate (Optional[float] | aymara_ai._types.Omit) – Maximum average pass rate (0.0-1.0)
min_pass_rate (Optional[float] | aymara_ai._types.Omit) – Minimum average pass rate (0.0-1.0)
modality (Optional[str] | aymara_ai._types.Omit) – Filter by modality (text, image)
name (Optional[str] | aymara_ai._types.Omit) – Filter by eval names (case-insensitive partial match)
offset (int | aymara_ai._types.Omit) – Number of results to skip
run_created_after (Union[str, datetime.datetime, None] | aymara_ai._types.Omit) – Filter by score runs created after this date
run_created_before (Union[str, datetime.datetime, None] | aymara_ai._types.Omit) – Filter by score runs created before this date
score_run_status (Optional[str] | aymara_ai._types.Omit) – Filter by any score run status
sort_by (typing_extensions.Literal[created_at, updated_at, EvalsResource.analyze.name, pass_rate, num_score_runs, last_run_date] | aymara_ai._types.Omit) – Field to sort by
sort_order (typing_extensions.Literal[asc, desc] | aymara_ai._types.Omit) – Sort order
status (Optional[str] | aymara_ai._types.Omit) – Filter by eval status (created, processing, finished, failed)
workspace_uuid (Optional[str] | aymara_ai._types.Omit) – Filter by workspace UUID
extra_headers (aymara_ai._types.Headers | None) – Send extra headers
extra_query (aymara_ai._types.Query | None) – Add additional query parameters to the request
extra_body (aymara_ai._types.Body | None) – Add additional JSON properties to the request
timeout (float | httpx.Timeout | None | aymara_ai._types.NotGiven) – Override the client-level default timeout for this request, in seconds

Return type:

aymara_ai.types.eval_analyze_response.EvalAnalyzeResponse

get(eval_uuid, *, workspace_uuid=omit, extra_headers=None, extra_query=None, extra_body=None, timeout=not_given)[source]#

Retrieve a specific eval by its UUID.

Args: eval_uuid (str): UUID of the eval to retrieve. workspace_uuid (str, optional): Optional workspace UUID for filtering.

Returns: Eval: The eval data.

Raises: AymaraAPIError: If the eval is not found.

Example: GET /api/evals/{eval_uuid}

Parameters:

extra_headers (aymara_ai._types.Headers | None) – Send extra headers
extra_query (aymara_ai._types.Query | None) – Add additional query parameters to the request
extra_body (aymara_ai._types.Body | None) – Add additional JSON properties to the request
timeout (float | httpx.Timeout | None | aymara_ai._types.NotGiven) – Override the client-level default timeout for this request, in seconds
eval_uuid (str)
workspace_uuid (str | aymara_ai._types.Omit)

Return type:

aymara_ai.types.eval.Eval

list_prompts(eval_uuid, *, limit=omit, offset=omit, workspace_uuid=omit, extra_headers=None, extra_query=None, extra_body=None, timeout=not_given)[source]#

Retrieve prompts for a specific eval if they exist.

Args: eval_uuid (str): UUID of the eval to get prompts for. workspace_uuid (str, optional): Optional workspace UUID for filtering.

Returns: list[EvalPrompt]: List of prompts and metadata for the eval.

Raises: AymaraAPIError: If the eval is not found.

Example: GET /api/evals/{eval_uuid}/prompts

Parameters:

extra_headers (aymara_ai._types.Headers | None) – Send extra headers
extra_query (aymara_ai._types.Query | None) – Add additional query parameters to the request
extra_body (aymara_ai._types.Body | None) – Add additional JSON properties to the request
timeout (float | httpx.Timeout | None | aymara_ai._types.NotGiven) – Override the client-level default timeout for this request, in seconds
eval_uuid (str)
limit (int | aymara_ai._types.Omit)
offset (int | aymara_ai._types.Omit)
workspace_uuid (str | aymara_ai._types.Omit)

Return type:

aymara_ai.pagination.SyncOffsetPage[aymara_ai.types.eval_prompt.EvalPrompt]

class aymara_ai.resources.evals.evals.AsyncEvalsResource(client)[source]#

Bases: aymara_ai._resource.AsyncAPIResource

Parameters:: client (aymara_ai._client.AsyncAymaraAI)

property runs: aymara_ai.resources.evals.runs.AsyncRunsResource#

Return type:: aymara_ai.resources.evals.runs.AsyncRunsResource

property with_raw_response: AsyncEvalsResourceWithRawResponse#

This property can be used as a prefix for any HTTP method call to return the raw response object instead of the parsed content.

For more information, see https://www.github.com/aymara-ai/aymara-sdk-python#accessing-raw-response-data-eg-headers

Return type:: AsyncEvalsResourceWithRawResponse

property with_streaming_response: AsyncEvalsResourceWithStreamingResponse#

An alternative to .with_raw_response that doesn’t eagerly read the response body.

For more information, see https://www.github.com/aymara-ai/aymara-sdk-python#with_streaming_response

Return type:: AsyncEvalsResourceWithStreamingResponse

async create(*, ai_description, eval_type, ai_instructions=omit, created_at=omit, created_by=omit, eval_instructions=omit, eval_uuid=omit, ground_truth=omit, is_jailbreak=omit, is_sandbox=omit, language=omit, modality=omit, name=omit, num_prompts=omit, prompt_examples=omit, status=omit, updated_at=omit, workspace_uuid=omit, extra_headers=None, extra_query=None, extra_body=None, timeout=not_given)[source]#

Create a new eval using an eval type configuration.

Args: eval_request (Eval): Data for the eval to create, including eval type and configuration.

Returns: Eval: The created eval object.

Raises: AymaraAPIError: If the request is invalid.

Example: POST /api/evals { “eval_type”: “…”, “workspace_uuid”: “…”, … }

Parameters:

ai_description (str) – Description of the AI under evaluation.
eval_type (str) – Type of the eval (safety, accuracy, etc.)
ai_instructions (Optional[str] | aymara_ai._types.Omit) – Instructions the AI should follow.
created_at (Union[str, datetime.datetime, None] | aymara_ai._types.Omit) – Timestamp when the eval was created.
created_by (Optional[str] | aymara_ai._types.Omit) – Name of the user who created the evaluation.
eval_instructions (Optional[str] | aymara_ai._types.Omit) – Additional instructions for the eval, if any.
eval_uuid (Optional[str] | aymara_ai._types.Omit) – Unique identifier for the evaluation.
ground_truth (Optional[aymara_ai.types.eval_create_params.GroundTruth] | aymara_ai._types.Omit) – Ground truth data or reference file, if any.
is_jailbreak (bool | aymara_ai._types.Omit) – Indicates if the eval is a jailbreak test.
is_sandbox (bool | aymara_ai._types.Omit) – Indicates if the eval results are sandboxed.
language (Optional[str] | aymara_ai._types.Omit) – Language code for the eval (default: “en”).
modality (aymara_ai.types.shared.content_type.ContentType | aymara_ai._types.Omit) – Content type for AI interactions.
name (Optional[str] | aymara_ai._types.Omit) – Name of the evaluation.
num_prompts (Optional[int] | aymara_ai._types.Omit) – Number of prompts/questions in the eval (default: 50).
prompt_examples (Optional[Iterable[aymara_ai.types.prompt_example_param.PromptExampleParam]] | aymara_ai._types.Omit) – List of example prompts for the eval.
status (Optional[aymara_ai.types.shared.status.Status] | aymara_ai._types.Omit) – Resource status.
updated_at (Union[str, datetime.datetime, None] | aymara_ai._types.Omit) – Timestamp when the eval was last updated.
workspace_uuid (Optional[str] | aymara_ai._types.Omit) – UUID of the associated workspace, if any.
extra_headers (aymara_ai._types.Headers | None) – Send extra headers
extra_query (aymara_ai._types.Query | None) – Add additional query parameters to the request
extra_body (aymara_ai._types.Body | None) – Add additional JSON properties to the request
timeout (float | httpx.Timeout | None | aymara_ai._types.NotGiven) – Override the client-level default timeout for this request, in seconds

Return type:

aymara_ai.types.eval.Eval

async update(eval_uuid, *, workspace_uuid=omit, ai_description=omit, ai_instructions=omit, eval_instructions=omit, ground_truth=omit, name=omit, prompt_creates=omit, prompt_updates=omit, extra_headers=None, extra_query=None, extra_body=None, timeout=not_given)[source]#

Update an existing eval’s fields and prompts.

Returns: Eval: The updated eval data.

Raises: AymaraAPIError: If the eval is not found or update is invalid.

Parameters:

ai_description (Optional[str] | aymara_ai._types.Omit) – New description of the AI under evaluation.
ai_instructions (Optional[str] | aymara_ai._types.Omit) – New instructions the AI should follow.
eval_instructions (Optional[str] | aymara_ai._types.Omit) – New additional instructions for the eval.
ground_truth (Optional[aymara_ai.types.eval_update_params.GroundTruth] | aymara_ai._types.Omit) – New ground truth data or reference file.
name (Optional[str] | aymara_ai._types.Omit) – New name for the evaluation.
prompt_creates (Optional[Iterable[aymara_ai.types.eval_update_params.PromptCreate]] | aymara_ai._types.Omit) – List of new prompts to add.
prompt_updates (Optional[Iterable[aymara_ai.types.eval_update_params.PromptUpdate]] | aymara_ai._types.Omit) – List of prompt updates to apply.
extra_headers (aymara_ai._types.Headers | None) – Send extra headers
extra_query (aymara_ai._types.Query | None) – Add additional query parameters to the request
extra_body (aymara_ai._types.Body | None) – Add additional JSON properties to the request
timeout (float | httpx.Timeout | None | aymara_ai._types.NotGiven) – Override the client-level default timeout for this request, in seconds
eval_uuid (str)
workspace_uuid (str | aymara_ai._types.Omit)

Return type:

aymara_ai.types.eval.Eval

list(*, limit=omit, offset=omit, workspace_uuid=omit, extra_headers=None, extra_query=None, extra_body=None, timeout=not_given)[source]#

List all evals, with optional filtering.

Args: workspace_uuid (str, optional): Optional workspace UUID for filtering. Use “**” for enterprise-wide access, omit for user’s current workspace.

Returns: list[Eval]: List of evals matching the filter.

Raises: AymaraAPIError: If the request is invalid.

Example: GET /api/evals?workspace_uuid=…

Parameters:

extra_headers (aymara_ai._types.Headers | None) – Send extra headers
extra_query (aymara_ai._types.Query | None) – Add additional query parameters to the request
extra_body (aymara_ai._types.Body | None) – Add additional JSON properties to the request
timeout (float | httpx.Timeout | None | aymara_ai._types.NotGiven) – Override the client-level default timeout for this request, in seconds
limit (int | aymara_ai._types.Omit)
offset (int | aymara_ai._types.Omit)
workspace_uuid (str | aymara_ai._types.Omit)

Return type:

aymara_ai._base_client.AsyncPaginator[aymara_ai.types.eval.Eval, aymara_ai.pagination.AsyncOffsetPage[aymara_ai.types.eval.Eval]]

async delete(eval_uuid, *, workspace_uuid=omit, extra_headers=None, extra_query=None, extra_body=None, timeout=not_given)[source]#

Delete an eval.

Args: eval_uuid (str): UUID of the eval to delete.

workspace_uuid (str, optional): Optional workspace UUID for filtering.

Returns: None

Raises: AymaraAPIError: If the eval is not found.

Example: DELETE /api/evals/{eval_uuid}

Parameters:

extra_headers (aymara_ai._types.Headers | None) – Send extra headers
extra_query (aymara_ai._types.Query | None) – Add additional query parameters to the request
extra_body (aymara_ai._types.Body | None) – Add additional JSON properties to the request
timeout (float | httpx.Timeout | None | aymara_ai._types.NotGiven) – Override the client-level default timeout for this request, in seconds
eval_uuid (str)
workspace_uuid (str | aymara_ai._types.Omit)

Return type:

None

async analyze(*, created_after=omit, created_before=omit, created_by=omit, eval_type=omit, has_score_runs=omit, is_jailbreak=omit, is_sandbox=omit, language=omit, limit=omit, max_pass_rate=omit, min_pass_rate=omit, modality=omit, name=omit, offset=omit, run_created_after=omit, run_created_before=omit, score_run_status=omit, sort_by=omit, sort_order=omit, status=omit, workspace_uuid=omit, extra_headers=None, extra_query=None, extra_body=None, timeout=not_given)[source]#

Analysis for evals with advanced filtering and aggregated statistics.

This endpoint allows analyzing across both eval metadata and score run performance data, providing comprehensive filtering capabilities and aggregated statistics for each eval.

Returns: EvalAnalysisResponse: Paginated results with matching evals and their statistics

Raises: AymaraAPIError: If the request is invalid or analysis parameters are malformed

Parameters:

created_after (Union[str, datetime.datetime, None] | aymara_ai._types.Omit) – Filter evals created after this date
created_before (Union[str, datetime.datetime, None] | aymara_ai._types.Omit) – Filter evals created before this date
created_by (Optional[str] | aymara_ai._types.Omit) – Filter by creator email
eval_type (Optional[str] | aymara_ai._types.Omit) – Filter by eval type (safety, accuracy, jailbreak, image_safety)
has_score_runs (Optional[bool] | aymara_ai._types.Omit) – Only include evals that have score runs
is_jailbreak (Optional[bool] | aymara_ai._types.Omit) – Filter by jailbreak status
is_sandbox (Optional[bool] | aymara_ai._types.Omit) – Filter by sandbox status
language (Optional[str] | aymara_ai._types.Omit) – Filter by language code (e.g., en, es)
limit (int | aymara_ai._types.Omit) – Maximum number of results (1-100)
max_pass_rate (Optional[float] | aymara_ai._types.Omit) – Maximum average pass rate (0.0-1.0)
min_pass_rate (Optional[float] | aymara_ai._types.Omit) – Minimum average pass rate (0.0-1.0)
modality (Optional[str] | aymara_ai._types.Omit) – Filter by modality (text, image)
name (Optional[str] | aymara_ai._types.Omit) – Filter by eval names (case-insensitive partial match)
offset (int | aymara_ai._types.Omit) – Number of results to skip
run_created_after (Union[str, datetime.datetime, None] | aymara_ai._types.Omit) – Filter by score runs created after this date
run_created_before (Union[str, datetime.datetime, None] | aymara_ai._types.Omit) – Filter by score runs created before this date
score_run_status (Optional[str] | aymara_ai._types.Omit) – Filter by any score run status
sort_by (typing_extensions.Literal[created_at, updated_at, AsyncEvalsResource.analyze.name, pass_rate, num_score_runs, last_run_date] | aymara_ai._types.Omit) – Field to sort by
sort_order (typing_extensions.Literal[asc, desc] | aymara_ai._types.Omit) – Sort order
status (Optional[str] | aymara_ai._types.Omit) – Filter by eval status (created, processing, finished, failed)
workspace_uuid (Optional[str] | aymara_ai._types.Omit) – Filter by workspace UUID
extra_headers (aymara_ai._types.Headers | None) – Send extra headers
extra_query (aymara_ai._types.Query | None) – Add additional query parameters to the request
extra_body (aymara_ai._types.Body | None) – Add additional JSON properties to the request
timeout (float | httpx.Timeout | None | aymara_ai._types.NotGiven) – Override the client-level default timeout for this request, in seconds

Return type:

aymara_ai.types.eval_analyze_response.EvalAnalyzeResponse

async get(eval_uuid, *, workspace_uuid=omit, extra_headers=None, extra_query=None, extra_body=None, timeout=not_given)[source]#

Retrieve a specific eval by its UUID.

Args: eval_uuid (str): UUID of the eval to retrieve. workspace_uuid (str, optional): Optional workspace UUID for filtering.

Returns: Eval: The eval data.

Raises: AymaraAPIError: If the eval is not found.

Example: GET /api/evals/{eval_uuid}

Parameters:

extra_headers (aymara_ai._types.Headers | None) – Send extra headers
extra_query (aymara_ai._types.Query | None) – Add additional query parameters to the request
extra_body (aymara_ai._types.Body | None) – Add additional JSON properties to the request
timeout (float | httpx.Timeout | None | aymara_ai._types.NotGiven) – Override the client-level default timeout for this request, in seconds
eval_uuid (str)
workspace_uuid (str | aymara_ai._types.Omit)

Return type:

aymara_ai.types.eval.Eval

list_prompts(eval_uuid, *, limit=omit, offset=omit, workspace_uuid=omit, extra_headers=None, extra_query=None, extra_body=None, timeout=not_given)[source]#

Retrieve prompts for a specific eval if they exist.

Args: eval_uuid (str): UUID of the eval to get prompts for. workspace_uuid (str, optional): Optional workspace UUID for filtering.

Returns: list[EvalPrompt]: List of prompts and metadata for the eval.

Raises: AymaraAPIError: If the eval is not found.

Example: GET /api/evals/{eval_uuid}/prompts

Parameters:

extra_headers (aymara_ai._types.Headers | None) – Send extra headers
extra_query (aymara_ai._types.Query | None) – Add additional query parameters to the request
extra_body (aymara_ai._types.Body | None) – Add additional JSON properties to the request
timeout (float | httpx.Timeout | None | aymara_ai._types.NotGiven) – Override the client-level default timeout for this request, in seconds
eval_uuid (str)
limit (int | aymara_ai._types.Omit)
offset (int | aymara_ai._types.Omit)
workspace_uuid (str | aymara_ai._types.Omit)

Return type:

aymara_ai._base_client.AsyncPaginator[aymara_ai.types.eval_prompt.EvalPrompt, aymara_ai.pagination.AsyncOffsetPage[aymara_ai.types.eval_prompt.EvalPrompt]]