Sample - OpenAI API
POST/evals

Create an evaluation

Creates a new evaluation definition for testing a model's performance. Supply data_source_config and testing_criteria to define the evaluation data schema and the graders applied to each run. Use the created evaluation to run tests with different models and model parameters.

  • RetriesRetries up to 2×, 500ms backoff, 30s timeout.

4 body fields

Evaluation definition containing the data source configuration and testing criteria.

namestringoptional
The name of the evaluation.
metadataobjectoptional
Optional key-value metadata attached to the evaluation. Use at most 16 entries, with keys of no more than 64 characters and string values, or set it to null.
data_source_configobjectrequired
The configuration for the data source used for the evaluation runs. Dictates the schema of the data used in the evaluation.
testing_criteriaarray<LabelModelGrader>required
A list of graders for all eval runs in this group. Graders can reference variables in the data source using double curly braces notation, like `{{item.variable_name}}`. To reference the model's output, use the `sample` namespace (ie, `{{sample.output_text}}`).

2 status codes
201Returns the created evaluation object with its unique identifier, data source configuration, testing criteria, creation timestamp, and metadata.
objectstringrequired
The object type.
Allowed:evalDefault:eval
idstringrequired
Unique identifier for the evaluation.
namestringrequired
The name of the evaluation.
data_source_configobjectrequired
Configuration of data sources used in runs of the evaluation.
testing_criteriaarray<LabelModelGrader>required
A list of testing criteria.
created_atintegerrequired
The Unix timestamp (in seconds) for when the eval was created.
metadataobjectrequired
Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard. Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.
429Returned when the request is rejected because the rate limit has been exceeded.
errorobjectrequired

Error handling

data_source_config and testing_criteria are required when creating an evaluation. metadata may contain up to 16 key-value pairs with keys limited to 64 characters and string values. A 429 is returned when the request exceeds the rate limit.