Submit an eval run (config + task YAML files)
Validates the eval config and every task file synchronously, then starts the run.
Validation covers YAML shape, required and unknown fields, connector mapping and modes against the source org’s scoped agent, grader name uniqueness, category-prefix consistency across judge assertions, a read-only compile of every deterministic grader’s SQL, and the per-task size cap.
Strict by default. Any invalid task file rejects the whole submission with 400 VALIDATION_FAILED. The
error’s message lists every issue, one per line, capped at 20 lines; validate_only: true runs the same
checks without creating a run. Pass on_invalid_task: ON_INVALID_TASK_SKIP to run with the valid files only.
Size caps: the inline eval config is held to 1 MiB and the body to 4 MiB (400 VALIDATION_FAILED, issue
ISSUE_CODE_PAYLOAD_TOO_LARGE); each task file to 100 MiB (issue ISSUE_CODE_FILE_TOO_LARGE) and the task
files together to 256 MiB (400 FIELD_TOO_LARGE).
The run executes in the token’s org. A source_org in the config that names another org is refused with 403.
Authentication
OAuth 2.0 client-credentials bearer from POST /oauth/v1/token (same as the Conversations API). The token is issued to one org and bound to one user; the run executes in that org and its conversations run as that user. If the eval config names a source_org, it must be the same org. An org that is not enabled for the Eval API sees 404 NOT_FOUND (“API not found”) on every route. An enabled org whose credential is not entitled to run evals receives 403 REQUEST_ACCESS_DENIED. submitted_by.email_addr is the email of the user the credential is bound to.
Request
The eval config (EvalConfigFile) as inline YAML.
Task files by Files API id (purpose=EVAL_RUNS_TASK). Each file holds one or more tasks (EvalTaskFile documents separated by ---); one file with the whole suite is the normal case. Upload filenames unique across the submission; goal_id unique across all files.
ON_INVALID_TASK_REJECT_RUN (default): any invalid task rejects the whole submission with 400 VALIDATION_FAILED. ON_INVALID_TASK_SKIP: invalid tasks are dropped, the run proceeds with the rest, and the dropped tasks are listed on the run as metadata.skipped_tasks. ON_INVALID_TASK_SKIP with no surviving task → 400 VALIDATION_FAILED with ISSUE_CODE_ALL_TASKS_REJECTED. ON_INVALID_TASK_SKIP covers per-task issues only: an issue on the eval config or on the submission as a whole (including a task file id that cannot be used) rejects the submission under either setting.
Run the same validation and create nothing. A valid submission answers 200 with the run as it would be created: metadata with submitted_by, tasks and skipped_tasks but no id, and no lifecycle; an invalid one answers the same VALIDATION_FAILED error as a real create. The dry run guarantees the submission is accepted, not that the run succeeds; it does not exercise the pipeline.
Response
Dry run (validate_only: true): the submission is valid and nothing was created; the body is the run preview: metadata without id, and no lifecycle. The 200 lets a client tell a dry run from an accepted run by status alone.
What a run is, fixed when the submission is accepted; a validate_only preview carries it without id.
What moves while the run executes; absent on a validate_only preview.
Non-blocking findings about the run, the twin of the validation issues[]. Absent when there are none, never null, never changes lifecycle.status. Carried on the create (202, or 200 on a validate_only dry run) and on the get, list and cancel responses; findings a pipeline step records later are merged in on later reads.