SDK Reference

SDK Reference#

SDK#

class aymara_ai.core.AymaraAI(api_key: str | None = None, base_url: str = 'https://api.aymara.ai')[source]#

Bases: TestMixin, ScoreRunMixin, SummaryMixin, UploadMixin, PolicyMixin, AymaraAIProtocol

Aymara AI SDK Client

This class provides methods for interacting with the Aymara AI API, including creating and managing tests, scoring tests, and retrieving results.

Parameters:

api_key (str, optional) – API key for authenticating with the Aymara AI API. Read from the AYMARA_API_KEY environment variable if not provided.
base_url (str, optional) – Base URL for the Aymara AI API, defaults to “https://api.aymara.ai”.
max_wait_time_secs (int, optional) – Maximum wait time for test creation, defaults to 120 seconds.

static get_pass_stats(score_runs: ScoreRunResponse | List[ScoreRunResponse]) → DataFrame[source]#

Create a DataFrame of pass rates and pass totals from one or more score runs.

Parameters:: score_runs (Union[ScoreRunResponse, List[ScoreRunResponse]]) – One or a list of test score runs to graph.
Returns:: DataFrame of pass rates per test score run.
Return type:: pd.DataFrame

static get_pass_stats_accuracy(score_run: AccuracyScoreRunResponse) → DataFrame[source]#

Create a DataFrame of pass rates and pass totals from one accuracy score run.

Parameters:: score_run (AccuracyScoreRunResponse) – One accuracy test score run to graph.
Returns:: DataFrame of pass rates per accuracy test question type.
Return type:: pd.DataFrame

Draw a bar graph of pass rates from one or more score runs.

Parameters:

score_runs (Union[List[ScoreRunResponse], ScoreRunResponse]) – One or a list of test score runs to graph.
title (str, optional) – Graph title.
ylim_min (float, optional) – y-axis lower limit, defaults to rounding down to the nearest ten.
ylim_max (float, optional) – y-axis upper limit, defaults to matplotlib’s preference but is capped at 100.
yaxis_is_percent (bool, optional) – Whether to show the pass rate as a percent (instead of the total number of questions passed), defaults to True.
ylabel (str) – Label of the y-axis, defaults to ‘Answers Passed’.
xaxis_is_score_run_uuids – Whether the x-axis represents tests (True) or score runs (False), defaults to True.
xlabel (str) – Label of the x-axis, defaults to ‘Score Runs’ if xaxis_is_score_run_uuids=True and ‘Tests’ otherwise.
xtick_rot (float) – rotation of the x-axis tick labels, defaults to 30.
xtick_labels_dict (dict, optional) – Maps test_names (keys) to x-axis tick labels (values).
kwargs – Options to pass to matplotlib.pyplot.bar.

Draw a bar graph of pass rates from one accuracy score run.

Parameters:

score_run (AccuracyScoreRunResponse) – The accuracy score run to graph.
title (str, optional) – Graph title.
ylim_min (float, optional) – y-axis lower limit, defaults to rounding down to the nearest ten.
ylim_max (float, optional) – y-axis upper limit, defaults to matplotlib’s preference but is capped at 100.
yaxis_is_percent (bool, optional) – Whether to show the pass rate as a percent (instead of the total number of questions passed), defaults to True.
ylabel (str) – Label of the y-axis, defaults to ‘Answers Passed’.
xlabel (str) – Label of the x-axis, defaults to ‘Question Types’.
xtick_rot (float) – rotation of the x-axis tick labels, defaults to 30.
xtick_labels_dict (dict, optional) – Maps test_names (keys) to x-axis tick labels (values).
kwargs – Options to pass to matplotlib.pyplot.bar.

static show_image_test_answers(tests: List[SafetyTestResponse], test_answers: Dict[str, List[ImageStudentAnswerInput]], score_runs: List[ScoreRunResponse] | None = None, n_images_per_test: int | None = 5, figsize: Tuple[int, int] | None = None) → None[source]#

Display a grid of image test answers with their test questions as captions. If score runs are included, display their test scores as captions instead and add a red border to failed images.

Parameters:

tests (List of SafetyTestResponse objects.) – Tests corresponding to the test answers.
test_answers (Dictionary of test UUIDs to lists of ImageStudentAnswerInput objects.) – Test answers.
score_runs (List of ScoreRunResponse objects, optional) – Score runs corresponding to the test answers.
n_images_per_test (int, optional) – Number of images to display per test.
figsize (integer tuple, optional) – Figure size. Defaults to (n_images_per_test * 3, n_tests * 2 * 4).

class aymara_ai.core.ScoreRunMixin(*args, **kwargs)[source]#

Bases: UploadMixin, AymaraAIProtocol

Mixin class that provides score run functionality. Inherits from UploadMixin to get image upload capabilities.

delete_score_run(score_run_uuid: str) → None[source]#

Delete a score run synchronously.

Parameters:: score_run_uuid (str) – UUID of the score run.

async delete_score_run_async(score_run_uuid: str) → None[source]#

Delete a score run asynchronously.

Parameters:: score_run_uuid (str) – UUID of the score run.

get_score_run(score_run_uuid: str) → ScoreRunResponse[source]#

Get the current status of a score run synchronously, and answers if it is completed.

Parameters:: score_run_uuid (str) – UUID of the score run.
Returns:: Score run response.
Return type:: ScoreRunResponse

async get_score_run_async(score_run_uuid: str) → ScoreRunResponse[source]#

Get the current status of a score run asynchronously, and answers if it is completed.

Parameters:: score_run_uuid (str) – UUID of the score run.
Returns:: Score run response.
Return type:: ScoreRunResponse

list_score_runs(test_uuid: str | None = None) → ListScoreRunResponse[source]#

List all score runs synchronously.

Parameters:: test_uuid (Optional[str]) – UUID of the test.
Returns:: List of score run responses.
Return type:: ListScoreRunResponse

async list_score_runs_async(test_uuid: str | None = None) → ListScoreRunResponse[source]#

List all score runs asynchronously.

Parameters:: test_uuid (Optional[str]) – UUID of the test.
Returns:: List of score run responses.
Return type:: ListScoreRunResponse

score_test(test_uuid: str, student_answers: List[BaseStudentAnswerInput], scoring_examples: List[ScoringExample] | None = None, max_wait_time_secs: int | None = None) → ScoreRunResponse[source]#

Score a test synchronously.

Parameters:

test_uuid (str) – UUID of the test.
student_answers (List[BaseStudentAnswerInput]) – List of BaseStudentAnswerInput objects containing student responses.
scoring_examples (Optional[List[ScoringExample]]) – Optional list of examples to guide the scoring process.
max_wait_time_secs (int, optional) – Maximum wait time for test scoring, defaults to 120, 300, and 300 seconds for safety, jailbreak, and accuracy tests, respectively.

Returns:

Score response.

Return type:

ScoreRunResponse

async score_test_async(test_uuid: str, student_answers: List[BaseStudentAnswerInput], scoring_examples: List[ScoringExample] | None = None, max_wait_time_secs: int | None = None) → ScoreRunResponse[source]#

Score a test asynchronously.

Parameters:

test_uuid (str) – UUID of the test.
student_answers (List[BaseStudentAnswerInput]) – List of BaseStudentAnswerInput objects containing student responses.
scoring_examples (Optional[List[ScoringExample]]) – Optional list of examples to guide the scoring process.
max_wait_time_secs (optional, int) – Maximum wait time for test scoring, defaults to 120, 300, and 300 seconds for safety, jailbreak, and accuracy tests, respectively.

Returns:

Score response.

Return type:

ScoreRunResponse

class aymara_ai.core.SummaryMixin(*args, **kwargs)[source]#

Bases: AymaraAIProtocol

create_summary(score_runs: List[ScoreRunResponse] | List[str]) → ScoreRunSuiteSummaryResponse[source]#

Create summaries for a list of score runs and wait for completion synchronously.

Parameters:: score_runs – List of score runs or their UUIDs for which to create summaries.
Returns:: Summary response.
Return type:: ScoreRunSuiteSummaryResponse

async create_summary_async(score_runs: List[ScoreRunResponse] | List[str]) → ScoreRunSuiteSummaryResponse[source]#

Create summaries for a list of score runs and wait for completion asynchronously.

Parameters:: score_runs – List of score runs or their UUIDs for which to create summaries.
Returns:: Summary response.
Return type:: ScoreRunsSummaryResponse

delete_summary(summary_uuid: str) → None[source]#

Delete a summary synchronously.

Parameters:: summary_uuid (str) – UUID of the summary.

async delete_summary_async(summary_uuid: str) → None[source]#

Delete a summary asynchronously.

Parameters:: summary_uuid (str) – UUID of the summary.

get_summary(summary_uuid: str) → ScoreRunSuiteSummaryResponse[source]#

Get the current status of an summary synchronously.

Parameters:: summary_uuid – UUID of the summary. :type summary_uuid: str
Returns:: Summary response.
Return type:: ScoreRunSuiteSummaryResponse

async get_summary_async(summary_uuid: str) → ScoreRunSuiteSummaryResponse[source]#

Get the current status of an summary asynchronously.

Parameters:: summary_uuid (str) – UUID of the summary.
Returns:: Summary response.
Return type:: ScoreRunSuiteSummaryResponse

list_summaries() → List[ScoreRunSuiteSummaryResponse][source]#: List all summaries synchronously.

async list_summaries_async() → List[ScoreRunSuiteSummaryResponse][source]#: List all summaries asynchronously.

class aymara_ai.core.TestMixin(*args, **kwargs)[source]#

Bases: AymaraAIProtocol

create_accuracy_test(test_name: str, student_description: str, knowledge_base: str, test_language: str = 'en', num_test_questions_per_question_type: int = 5, max_wait_time_secs: int | None = 300) → AccuracyTestResponse[source]#

Create an Aymara accuracy test synchronously and wait for completion.

Parameters:

test_name (str) – Name of the test. Should be between 1 and 100 characters.
student_description (str) – Description of the AI that will take the test (e.g., its purpose, expected use, typical user). The more specific your description is, the less generic the test questions will be.
knowledge_base (str) – Knowledge base text that will be used to generate accuracy test questions.
test_language (str, optional) – Language of the test, defaults to en.
num_test_questions_per_question_type (int, optional) – Number of test questions per question type, defaults to 5. Should be between 1 and 100 questions.
max_wait_time_secs (int, optional) – Maximum wait time for test creation, defaults to 300 seconds.

Returns:

Test response containing test details and generated questions.

Return type:

AccuracyTestResponse

Raises:

ValueError – If the test_name length is not within the allowed range.
ValueError – If num_test_questions is not within the allowed range.
ValueError – If knowledge_base is not provided for accuracy tests.

async create_accuracy_test_async(test_name: str, student_description: str, knowledge_base: str, test_language: str = 'en', num_test_questions_per_question_type: int = 5, max_wait_time_secs: int | None = 300) → AccuracyTestResponse[source]#

Create an Aymara accuracy test asynchronously and wait for completion.

Parameters:

test_name (str) – Name of the test. Should be between 1 and 100 characters.
student_description (str) – Description of the AI that will take the test (e.g., its purpose, expected use, typical user). The more specific your description is, the less generic the test questions will be.
knowledge_base (str) – Knowledge base text that will be used to generate accuracy test questions.
test_language (str, optional) – Language of the test, defaults to en.
num_test_questions_per_question_type (int, optional) – Number of test questions per question type, defaults to 20. Should be between 1 and 100 questions.
max_wait_time_secs (int, optional) – Maximum wait time for test creation, defaults to 300 seconds.

Returns:

Test response containing test details and generated questions.

Return type:

AccuracyTestResponse

Raises:

ValueError – If the test_name length is not within the allowed range.
ValueError – If num_test_questions is not within the allowed range.
ValueError – If knowledge_base is not provided for accuracy tests.

create_image_safety_test(test_name: str, student_description: str, test_policy: str, test_language: str = 'en', num_test_questions: int = 20, max_wait_time_secs: int | None = 120, additional_instructions: str | None = None, good_examples: List[GoodExample] | None = None, bad_examples: List[BadExample] | None = None)[source]#

Create an Aymara image safety test synchronously and wait for completion.

Parameters:

test_name (str) – Name of the test. Should be between 1 and 100 characters.
student_description (str) – Description of the AI that will take the test (e.g., its purpose, expected use, typical user). The more specific your description is, the less generic the test questions will be.
test_policy (str) – Policy of the test, which will measure compliance against this policy (required for safety tests).
test_language (str, optional) – Language of the test, defaults to en.
num_test_questions (int, optional) – Number of test questions, defaults to 20. Should be between 1 and 100 questions.
max_wait_time_secs (int, optional) – Maximum wait time for test creation, defaults to 120 seconds.
additional_instructions (str, optional) – Optional additional instructions for test generation
good_examples (List[GoodExample], optional) – Optional list of good examples to guide question generation
bad_examples (List[BadExample], optional) – Optional list of bad examples to guide question generation

Returns:

Test response containing test details and generated questions.

Return type:

SafetyTestResponse

Raises:

ValueError – If the test_name length is not within the allowed range.
ValueError – If num_test_questions is not within the allowed range.
ValueError – If test_policy is not provided for safety tests.

async create_image_safety_test_async(test_name: str, student_description: str, test_policy: str, test_language: str = 'en', num_test_questions: int = 20, max_wait_time_secs: int | None = 120, additional_instructions: str | None = None, good_examples: List[GoodExample] | None = None, bad_examples: List[BadExample] | None = None)[source]#

Create an Aymara image safety test asynchronously and wait for completion.

Parameters:

test_name (str) – Name of the test. Should be between 1 and 100 characters.
student_description (str) – Description of the AI that will take the test (e.g., its purpose, expected use, typical user). The more specific your description is, the less generic the test questions will be.
test_policy (str) – Policy of the test, which will measure compliance against this policy (required for safety tests).
test_language (str, optional) – Language of the test, defaults to en.
num_test_questions (int, optional) – Number of test questions, defaults to 20. Should be between 1 and 100 questions.
max_wait_time_secs (int, optional) – Maximum wait time for test creation, defaults to 120 seconds.
additional_instructions (str, optional) – Optional additional instructions for test generation
good_examples (List[GoodExample], optional) – Optional list of good examples to guide question generation
bad_examples (List[BadExample], optional) – Optional list of bad examples to guide question generation

Returns:

Test response containing test details and generated questions.

Return type:

SafetyTestResponse

Raises:

ValueError – If the test_name length is not within the allowed range.
ValueError – If num_test_questions is not within the allowed range.
ValueError – If test_policy is not provided for safety tests.

create_jailbreak_test(test_name: str, student_description: str, test_system_prompt: str, test_language: str = 'en', max_wait_time_secs: int = 300, additional_instructions: str | None = None, good_examples: List[GoodExample] | None = None, bad_examples: List[BadExample] | None = None, limit_num_questions: int | None = None) → JailbreakTestResponse[source]#

Create an Aymara jailbreak test synchronously and wait for completion.

Parameters:

test_name (str) – Name of the test. Should be between 1 and 100 characters.
student_description (str) – Description of the AI that will take the test (e.g., its purpose, expected use, typical user). The more specific your description is, the less generic the test questions will be.
test_system_prompt (str) – System prompt of the jailbreak test.
test_language (str, optional) – Language of the test, defaults to en.
max_wait_time_secs (int, optional) – Maximum wait time for test creation, defaults to 300 seconds.
additional_instructions (str, optional) – Optional additional instructions for test generation
good_examples (List[GoodExample], optional) – Optional list of good examples to guide question generation
bad_examples (List[BadExample], optional) – Optional list of bad examples to guide question generation
limit_num_questions (int, optional) – Optional limit on the number of questions generated

Returns:

Test response containing test details and generated questions.

Return type:

JailbreakTestResponse

Raises:

ValueError – If the test_name length is not within the allowed range.
ValueError – If num_test_questions is not within the allowed range.
ValueError – If test_system_prompt is not provided for jailbreak tests.

async create_jailbreak_test_async(test_name: str, student_description: str, test_system_prompt: str, test_language: str = 'en', max_wait_time_secs: int = 300, additional_instructions: str | None = None, good_examples: List[GoodExample] | None = None, bad_examples: List[BadExample] | None = None, limit_num_questions: int | None = None) → JailbreakTestResponse[source]#

Create an Aymara jailbreak test asynchronously and wait for completion.

Parameters:

test_name (str) – Name of the test. Should be between 1 and 100 characters.
student_description (str) – Description of the AI that will take the test (e.g., its purpose, expected use, typical user). The more specific your description is, the less generic the test questions will be.
test_system_prompt (str) – System prompt of the jailbreak test.
test_language (str, optional) – Language of the test, defaults to en.
max_wait_time_secs (int, optional) – Maximum wait time for test creation, defaults to 300 seconds.
additional_instructions (str, optional) – Optional additional instructions for test generation
good_examples (List[GoodExample], optional) – Optional list of good examples to guide question generation
bad_examples (List[BadExample], optional) – Optional list of bad examples to guide question generation
limit_num_questions (int, optional) – Optional limit on the number of questions generated

Returns:

Test response containing test details and generated questions.

Return type:

JailbreakTestResponse

Raises:

ValueError – If the test_name length is not within the allowed range.
ValueError – If num_test_questions is not within the allowed range.
ValueError – If test_system_prompt is not provided for jailbreak tests.

create_safety_test(test_name: str, student_description: str, test_policy: str, test_language: str = 'en', num_test_questions: int = 20, max_wait_time_secs: int = 120, additional_instructions: str | None = None, good_examples: List[GoodExample] | None = None, bad_examples: List[BadExample] | None = None) → SafetyTestResponse[source]#

Create an Aymara safety test synchronously and wait for completion.

Parameters:

test_name (str) – Name of the test. Should be between 1 and 100 characters.
student_description (str) – Description of the AI that will take the test (e.g., its purpose, expected use, typical user). The more specific your description is, the less generic the test questions will be.
test_policy (str) – Policy of the test, which will measure compliance against this policy (required for safety tests).
test_language (str, optional) – Language of the test, defaults to en.
num_test_questions (int, optional) – Number of test questions, defaults to 20. Should be between 1 and 100 questions.
max_wait_time_secs (int, optional) – Maximum wait time for test creation, defaults to 120 seconds.
additional_instructions (str, optional) – Optional additional instructions for test generation
good_examples (List[GoodExample], optional) – Optional list of good examples to guide question generation
bad_examples (List[BadExample], optional) – Optional list of bad examples to guide question generation

Returns:

Test response containing test details and generated questions.

Return type:

SafetyTestResponse

Raises:

ValueError – If the test_name length is not within the allowed range.
ValueError – If num_test_questions is not within the allowed range.
ValueError – If test_policy is not provided for safety tests.

async create_safety_test_async(test_name: str, student_description: str, test_policy: str, test_language: str = 'en', num_test_questions: int = 20, max_wait_time_secs: int | None = 120, additional_instructions: str | None = None, good_examples: List[GoodExample] | None = None, bad_examples: List[BadExample] | None = None) → SafetyTestResponse[source]#

Create an Aymara safety test asynchronously and wait for completion.

Parameters:

test_name (str) – Name of the test. Should be between 1 and 100 characters.
student_description (str) – Description of the AI that will take the test (e.g., its purpose, expected use, typical user). The more specific your description is, the less generic the test questions will be.
test_policy (str) – Policy of the test, which will measure compliance against this policy (required for safety tests).
test_language (str, optional) – Language of the test, defaults to en.
num_test_questions (int, optional) – Number of test questions, defaults to 20. Should be between 1 and 100 questions.
max_wait_time_secs (int, optional) – Maximum wait time for test creation, defaults to 120 seconds.
additional_instructions (str, optional) – Optional additional instructions for test generation
good_examples (List[GoodExample], optional) – Optional list of good examples to guide question generation
bad_examples (List[BadExample], optional) – Optional list of bad examples to guide question generation

Returns:

Test response containing test details and generated questions.

Return type:

SafetyTestResponse

Raises:

ValueError – If the test_name length is not within the allowed range.
ValueError – If num_test_questions is not within the allowed range.
ValueError – If test_policy is not provided for safety tests.

delete_test(test_uuid: str) → None[source]#: Delete a test synchronously.

async delete_test_async(test_uuid: str) → None[source]#: Delete a test asynchronously.

get_test(test_uuid: str) → BaseTestResponse[source]#

Get the current status of a test synchronously, and questions if it is completed.

Parameters:: test_uuid (str) – UUID of the test.
Returns:: Test response.
Return type:: TestResponse

async get_test_async(test_uuid: str) → BaseTestResponse[source]#

Get the current status of a test asynchronously, and questions if it is completed.

Parameters:: test_uuid (str) – UUID of the test.
Returns:: Test response.
Return type:: TestResponse

list_tests() → ListTestResponse[source]#: List all tests synchronously.

async list_tests_async() → ListTestResponse[source]#: List all tests asynchronously.

SDK Types#

Types for the SDK

class aymara_ai.types.AccuracyQuestionResponse(*, question_text: str, question_uuid: str, accuracy_question_type: str | None = None)[source]#

Bases: QuestionResponse

Question in the test

accuracy_question_type: Annotated[str | None, FieldInfo(annotation=NoneType, required=False, default=None, description='Type of the question for accuracy tests')]#

classmethod from_question_schema(question: QuestionSchema) → AccuracyQuestionResponse[source]#

model_config: ClassVar[ConfigDict] = {}#: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

to_question_schema() → QuestionSchema[source]#

class aymara_ai.types.AccuracyScoreRunResponse(*, score_run_uuid: str, score_run_status: Status, test: BaseTestResponse, answers: List[AccuracyScoredAnswerResponse] | None = None, created_at: datetime, failure_reason: str | None = None, pass_rate: float | None = None)[source]#

Bases: ScoreRunResponse

Score run response for accuracy tests.

answers: Annotated[List[AccuracyScoredAnswerResponse] | None, FieldInfo(annotation=NoneType, required=False, default=None, description='List of scored answers')]#

model_config: ClassVar[ConfigDict] = {}#: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

to_scores_df() → DataFrame[source]#: Create a scores DataFrame.

class aymara_ai.types.AccuracyScoredAnswerResponse(*, answer_uuid: str, question_uuid: str, answer_text: str | None = None, answer_image_path: str | None = None, answer_image_url: str | None = None, question_text: str, explanation: str | None = None, confidence: float | None = None, is_passed: bool | None = None, exclude_from_scoring: bool = False, student_refused: bool = False, accuracy_question_type: str)[source]#

Bases: ScoredAnswerResponse

A single answer to a question in the test that has been scored.

accuracy_question_type: Annotated[str, FieldInfo(annotation=NoneType, required=True, description='Type of the question for accuracy tests')]#

classmethod from_answer_out_schema(answer: AnswerOutSchema) → AccuracyScoredAnswerResponse[source]#

model_config: ClassVar[ConfigDict] = {}#: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class aymara_ai.types.AccuracyTestResponse(*, test_uuid: str, test_type: TestType, test_name: str, test_status: Status, created_at: datetime, num_test_questions: int | None = None, questions: List[AccuracyQuestionResponse] | None = None, failure_reason: str | None = None, good_examples: List[GoodExample] | None = None, bad_examples: List[BadExample] | None = None, knowledge_base: str)[source]#

Bases: BaseTestResponse

Accuracy test response.

knowledge_base: Annotated[str, FieldInfo(annotation=NoneType, required=True, description='Knowledge base to test against')]#

model_config: ClassVar[ConfigDict] = {}#: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

questions: Annotated[List[AccuracyQuestionResponse] | None, FieldInfo(annotation=NoneType, required=False, default=None, description='Questions in the test')]#

class aymara_ai.types.BadExample(*, question_text: str, explanation: str | None = None)[source]#

Bases: BaseModel

A bad example (counter-example) of the kind of question to generate

explanation: Annotated[str | None, FieldInfo(annotation=NoneType, required=False, default=None, description='Explanation of why this is a counter-example')]#

model_config: ClassVar[ConfigDict] = {}#: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

question_text: Annotated[str, FieldInfo(annotation=NoneType, required=True, description='Example question text')]#

to_example_in_schema() → ExampleInSchema[source]#

class aymara_ai.types.BaseStudentAnswerInput(*, question_uuid: str, is_refusal: bool = False, exclude_from_scoring: bool = False)[source]#

Bases: BaseModel

Base class for student answers

exclude_from_scoring: Annotated[bool, FieldInfo(annotation=NoneType, required=False, default=False, description='Whether the answer should be excluded from scoring')]#

is_refusal: Annotated[bool, FieldInfo(annotation=NoneType, required=False, default=False, description='Whether the student refused to answer')]#

model_config: ClassVar[ConfigDict] = {}#: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

question_uuid: Annotated[str, FieldInfo(annotation=NoneType, required=True, description='UUID of the question')]#

to_answer_in_schema() → AnswerInSchema[source]#

validate_answer_provided() → BaseStudentAnswerInput[source]#: Validate that an answer is provided unless exclude_from_scoring is True.

class aymara_ai.types.BaseTestResponse(*, test_uuid: str, test_type: TestType, test_name: str, test_status: Status, created_at: datetime, num_test_questions: int | None = None, questions: List[QuestionResponse] | None = None, failure_reason: str | None = None, good_examples: List[GoodExample] | None = None, bad_examples: List[BadExample] | None = None)[source]#

Bases: BaseModel

Test response. May or may not have questions, depending on the test status.

bad_examples: Annotated[List[BadExample] | None, FieldInfo(annotation=NoneType, required=False, default=None, description='Bad examples for the test')]#

created_at: Annotated[datetime, FieldInfo(annotation=NoneType, required=True, description='Timestamp of the test creation')]#

failure_reason: Annotated[str | None, FieldInfo(annotation=NoneType, required=False, default=None, description='Reason for the test failure')]#

classmethod from_test_out_schema_and_questions(test: TestOutSchema, questions: List[QuestionSchema] | None = None, failure_reason: str | None = None) → BaseTestResponse[source]#

good_examples: Annotated[List[GoodExample] | None, FieldInfo(annotation=NoneType, required=False, default=None, description='Good examples for the test')]#

model_config: ClassVar[ConfigDict] = {}#: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

num_test_questions: Annotated[int | None, FieldInfo(annotation=NoneType, required=False, default=None, description='Number of test questions')]#

questions: Annotated[List[QuestionResponse] | None, FieldInfo(annotation=NoneType, required=False, default=None, description='Questions in the test')]#

test_name: Annotated[str, FieldInfo(annotation=NoneType, required=True, description='Name of the test')]#

test_status: Annotated[Status, FieldInfo(annotation=NoneType, required=True, description='Status of the test')]#

test_type: Annotated[TestType, FieldInfo(annotation=NoneType, required=True, description='Type of the test')]#

test_uuid: Annotated[str, FieldInfo(annotation=NoneType, required=True, description='UUID of the test')]#

to_questions_df() → DataFrame[source]#: Create a questions DataFrame.

class aymara_ai.types.CreateScoreRunInput(*, test_uuid: str, student_responses: List[TextStudentAnswerInput], scoring_examples: List[ScoringExample] | None = None)[source]#

Bases: BaseModel

Parameters for scoring a test

model_config: ClassVar[ConfigDict] = {}#: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

scoring_examples: Annotated[List[ScoringExample] | None, FieldInfo(annotation=NoneType, required=False, default=None, description='Examples to guide scoring')]#

student_responses: Annotated[List[TextStudentAnswerInput], FieldInfo(annotation=NoneType, required=True, description='Student responses')]#

test_uuid: Annotated[str, FieldInfo(annotation=NoneType, required=True, description='UUID of the test')]#

class aymara_ai.types.GoodExample(*, question_text: str, explanation: str | None = None)[source]#

Bases: BaseModel

A good example of the kind of question to generate

explanation: Annotated[str | None, FieldInfo(annotation=NoneType, required=False, default=None, description='Explanation of why this is a good example')]#

model_config: ClassVar[ConfigDict] = {}#: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

question_text: Annotated[str, FieldInfo(annotation=NoneType, required=True, description='Example question text')]#

to_example_in_schema() → ExampleInSchema[source]#

class aymara_ai.types.ImageScoringExample(*, question_text: str, image_description: str, explanation: str | None = None, is_passing: bool)[source]#

Bases: BaseModel

An example answer to guide the scoring process

explanation: Annotated[str | None, FieldInfo(annotation=NoneType, required=False, default=None, description='Explanation of why this answer should pass/fail')]#

image_description: Annotated[str, FieldInfo(annotation=NoneType, required=True, description='Description of the image')]#

is_passing: Annotated[bool, FieldInfo(annotation=NoneType, required=True, description='Whether this is a passing example')]#

model_config: ClassVar[ConfigDict] = {}#: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

question_text: Annotated[str, FieldInfo(annotation=NoneType, required=True, description='Example question text')]#

to_scoring_example_in_schema() → ScoringExampleInSchema[source]#

class aymara_ai.types.ImageStudentAnswerInput(*, question_uuid: str, is_refusal: bool = False, exclude_from_scoring: bool = False, answer_image_path: str | None = None)[source]#

Bases: BaseStudentAnswerInput

Student image answer for a question

answer_image_path: Annotated[str | None, FieldInfo(annotation=NoneType, required=False, default=None, description='Path to the image')]#

classmethod from_answer_in_schema(answer: AnswerInSchema) → ImageStudentAnswerInput[source]#

model_config: ClassVar[ConfigDict] = {}#: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class aymara_ai.types.JailbreakTestResponse(*, test_uuid: str, test_type: TestType, test_name: str, test_status: Status, created_at: datetime, num_test_questions: int | None = None, questions: List[QuestionResponse] | None = None, failure_reason: str | None = None, good_examples: List[GoodExample] | None = None, bad_examples: List[BadExample] | None = None, test_system_prompt: str)[source]#

Bases: BaseTestResponse

Jailbreak test response.

model_config: ClassVar[ConfigDict] = {}#: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

test_system_prompt: Annotated[str, FieldInfo(annotation=NoneType, required=True, description='System prompt to jailbreak')]#

class aymara_ai.types.ListScoreRunResponse(root: RootModelRootType = PydanticUndefined)[source]#

Bases: RootModel

List of score runs.

model_config: ClassVar[ConfigDict] = {}#: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

root: List[ScoreRunResponse]#

to_df() → DataFrame[source]#: Create a DataFrame from the list of ScoreRunResponses.

class aymara_ai.types.ListScoreRunSuiteSummaryResponse(root: RootModelRootType = PydanticUndefined)[source]#

Bases: RootModel

List of score run suite summaries.

model_config: ClassVar[ConfigDict] = {}#: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

root: List[ScoreRunSuiteSummaryResponse]#

class aymara_ai.types.ListTestResponse(root: RootModelRootType = PydanticUndefined)[source]#

Bases: RootModel

List of tests.

model_config: ClassVar[ConfigDict] = {}#: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

root: List[BaseTestResponse]#

to_df() → DataFrame[source]#: Create a DataFrame from the list of TestResponses.

class aymara_ai.types.QuestionResponse(*, question_text: str, question_uuid: str)[source]#

Bases: BaseModel

Question in the test

classmethod from_question_schema(question: QuestionSchema) → QuestionResponse[source]#

model_config: ClassVar[ConfigDict] = {}#: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

question_text: Annotated[str, FieldInfo(annotation=NoneType, required=True, description='Question in the test')]#

question_uuid: Annotated[str, FieldInfo(annotation=NoneType, required=True, description='UUID of the question')]#

to_question_schema() → QuestionSchema[source]#

class aymara_ai.types.SafetyTestResponse(*, test_uuid: str, test_type: TestType, test_name: str, test_status: Status, created_at: datetime, num_test_questions: int | None = None, questions: List[QuestionResponse] | None = None, failure_reason: str | None = None, good_examples: List[GoodExample] | None = None, bad_examples: List[BadExample] | None = None, test_policy: str)[source]#

Bases: BaseTestResponse

Safety test response.

model_config: ClassVar[ConfigDict] = {}#: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

test_policy: Annotated[str, FieldInfo(annotation=NoneType, required=True, description='Safety Policy to test against')]#

class aymara_ai.types.ScoreRunResponse(*, score_run_uuid: str, score_run_status: Status, test: BaseTestResponse, answers: List[ScoredAnswerResponse] | None = None, created_at: datetime, failure_reason: str | None = None, pass_rate: float | None = None)[source]#

Bases: BaseModel

Score run response. May or may not have answers, depending on the score run status.

answers: Annotated[List[ScoredAnswerResponse] | None, FieldInfo(annotation=NoneType, required=False, default=None, description='List of scored answers')]#

created_at: Annotated[datetime, FieldInfo(annotation=NoneType, required=True, description='Timestamp of the score run creation')]#

failure_reason: Annotated[str | None, FieldInfo(annotation=NoneType, required=False, default=None, description='Reason for the score run failure')]#

classmethod from_score_run_out_schema_and_answers(score_run: ScoreRunOutSchema, answers: List[AnswerOutSchema] | None = None, failure_reason: str | None = None) → ScoreRunResponse[source]#

model_config: ClassVar[ConfigDict] = {}#: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

pass_rate: Annotated[float | None, FieldInfo(annotation=NoneType, required=False, default=None, description='Pass rate of the score run')]#

score_run_status: Annotated[Status, FieldInfo(annotation=NoneType, required=True, description='Status of the score run')]#

score_run_uuid: Annotated[str, FieldInfo(annotation=NoneType, required=True, description='UUID of the score run')]#

test: Annotated[BaseTestResponse, FieldInfo(annotation=NoneType, required=True, description='Test response')]#

to_scores_df() → DataFrame[source]#: Create a scores DataFrame.

class aymara_ai.types.ScoreRunSuiteSummaryResponse(*, score_run_suite_summary_uuid: str, score_run_suite_summary_status: Status, overall_passing_answers_summary: str | None = None, overall_failing_answers_summary: str | None = None, overall_improvement_advice: str | None = None, score_run_summaries: List[ScoreRunSummaryResponse], created_at: datetime, failure_reason: str | None = None)[source]#

Bases: BaseModel

Score run suite summary response.

created_at: Annotated[datetime, FieldInfo(annotation=NoneType, required=True, description='Timestamp of the score run suite summary creation')]#

failure_reason: Annotated[str | None, FieldInfo(annotation=NoneType, required=False, default=None, description='Reason for the score run failure')]#

classmethod from_summary_out_schema_and_failure_reason(summary: ScoreRunSuiteSummaryOutSchema, failure_reason: str | None = None) → ScoreRunSuiteSummaryResponse[source]#

model_config: ClassVar[ConfigDict] = {}#: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

overall_failing_answers_summary: Annotated[str | None, FieldInfo(annotation=NoneType, required=False, default=None, description='Summary of the failing answers')]#

overall_improvement_advice: Annotated[str | None, FieldInfo(annotation=NoneType, required=False, default=None, description='Advice for improvement')]#

overall_passing_answers_summary: Annotated[str | None, FieldInfo(annotation=NoneType, required=False, default=None, description='Summary of the passing answers')]#

score_run_suite_summary_status: Annotated[Status, FieldInfo(annotation=NoneType, required=True, description='Status of the score run suite summary')]#

score_run_suite_summary_uuid: Annotated[str, FieldInfo(annotation=NoneType, required=True, description='UUID of the score run suite summary')]#

score_run_summaries: Annotated[List[ScoreRunSummaryResponse], FieldInfo(annotation=NoneType, required=True, description='List of score run summaries')]#

to_df() → DataFrame[source]#: Create a scores DataFrame.

class aymara_ai.types.ScoreRunSummaryResponse(*, score_run_summary_uuid: str, passing_answers_summary: str, failing_answers_summary: str, improvement_advice: str, test_name: str, test_type: TestType, score_run_uuid: str)[source]#

Bases: BaseModel

Score run summary response.

failing_answers_summary: Annotated[str, FieldInfo(annotation=NoneType, required=True, description='Summary of the failing answers')]#

classmethod from_score_run_summary_out_schema(summary: ScoreRunSummaryOutSchema) → ScoreRunSummaryResponse[source]#

improvement_advice: Annotated[str, FieldInfo(annotation=NoneType, required=True, description='Advice for improvement')]#

model_config: ClassVar[ConfigDict] = {}#: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

passing_answers_summary: Annotated[str, FieldInfo(annotation=NoneType, required=True, description='Summary of the passing answers')]#

score_run_summary_uuid: Annotated[str, FieldInfo(annotation=NoneType, required=True, description='UUID of the score run summary')]#

score_run_uuid: Annotated[str, FieldInfo(annotation=NoneType, required=True, description='UUID of the score run')]#

test_name: Annotated[str, FieldInfo(annotation=NoneType, required=True, description='Name of the test')]#

test_type: Annotated[TestType, FieldInfo(annotation=NoneType, required=True, description='Type of the test')]#

class aymara_ai.types.ScoredAnswerResponse(*, answer_uuid: str, question_uuid: str, answer_text: str | None = None, answer_image_path: str | None = None, answer_image_url: str | None = None, question_text: str, explanation: str | None = None, confidence: float | None = None, is_passed: bool | None = None, exclude_from_scoring: bool = False, student_refused: bool = False)[source]#

Bases: BaseModel

A single answer to a question in the test that has been scored.

answer_image_path: Annotated[str | None, FieldInfo(annotation=NoneType, required=False, default=None, description='Path to the answer image')]#

answer_image_url: Annotated[str | None, FieldInfo(annotation=NoneType, required=False, default=None, description='URL to the answer image')]#

answer_text: Annotated[str | None, FieldInfo(annotation=NoneType, required=False, default=None, description='Answer to the question')]#

answer_uuid: Annotated[str, FieldInfo(annotation=NoneType, required=True, description='UUID of the answer')]#

confidence: Annotated[float | None, FieldInfo(annotation=NoneType, required=False, default=None, description='Confidence score')]#

exclude_from_scoring: Annotated[bool, FieldInfo(annotation=NoneType, required=False, default=False, description='Whether the answer is excluded from scoring')]#

explanation: Annotated[str | None, FieldInfo(annotation=NoneType, required=False, default=None, description='Explanation for the score')]#

classmethod from_answer_out_schema(answer: AnswerOutSchema) → ScoredAnswerResponse[source]#

is_passed: Annotated[bool | None, FieldInfo(annotation=NoneType, required=False, default=None, description='Whether the answer is passed')]#

model_config: ClassVar[ConfigDict] = {}#: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

question_text: Annotated[str, FieldInfo(annotation=NoneType, required=True, description='Question in the test')]#

question_uuid: Annotated[str, FieldInfo(annotation=NoneType, required=True, description='UUID of the question')]#

student_refused: Annotated[bool, FieldInfo(annotation=NoneType, required=False, default=False, description='Whether the student refused to answer')]#

class aymara_ai.types.ScoringExample(*, question_text: str, answer_text: str, explanation: str | None = None, is_passing: bool)[source]#

Bases: BaseModel

An example answer to guide the scoring process

answer_text: Annotated[str, FieldInfo(annotation=NoneType, required=True, description='Example answer text')]#

explanation: Annotated[str | None, FieldInfo(annotation=NoneType, required=False, default=None, description='Explanation of why this answer should pass/fail')]#

is_passing: Annotated[bool, FieldInfo(annotation=NoneType, required=True, description='Whether this is a passing example')]#

model_config: ClassVar[ConfigDict] = {}#: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

question_text: Annotated[str, FieldInfo(annotation=NoneType, required=True, description='Example question text')]#

to_scoring_example_in_schema() → ScoringExampleInSchema[source]#

class aymara_ai.types.Status(value)[source]#

Bases: str, Enum

Status for Test or Score Run

COMPLETED = 'COMPLETED'#

FAILED = 'FAILED'#

PENDING = 'PENDING'#

PROCESSING = 'PROCESSING'#

UPLOADING = 'UPLOADING'#

classmethod from_api_status(api_status: TestStatus | ScoreRunStatus | ScoreRunSuiteSummaryStatus) → Status[source]#

Transform an API status to the user-friendly status.

Parameters:: api_status (Union[TestStatus, ScoreRunStatus]) – API status (either TestStatus or ScoreRunStatus).
Returns:: Transformed status.
Return type:: Status

class aymara_ai.types.TextStudentAnswerInput(*, question_uuid: str, is_refusal: bool = False, exclude_from_scoring: bool = False, answer_text: str | None)[source]#

Bases: BaseStudentAnswerInput

Student text answer for a question

answer_text: Annotated[str | None, FieldInfo(annotation=NoneType, required=True, description='Answer text provided by the student')]#

classmethod from_answer_in_schema(answer: AnswerInSchema) → TextStudentAnswerInput[source]#

model_config: ClassVar[ConfigDict] = {}#: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

SDK Reference

Contents

SDK Reference#

SDK#

SDK Types#