Eval API overview

v1alpha1 is alpha: no compatibility guarantee.

An eval run is one eval config plus one or more task files. A task is one simulated conversation, identified by goal_id (unique within the run). The run executes in the org the bearer token belongs to, as the user the token is bound to, against a copy of the scoped agent’s configuration that cleanup removes.

Requirements

• The org is enabled for the Eval API and the Files API. Otherwise: 404 NOT_FOUND.
• A Conversations API credential (POST /oauth/v1/token, client credentials) bound to a user with Creator Studio view access and, where enforced, MW Setup edit access. Otherwise: 403 REQUEST_ACCESS_DENIED.

Flow

1. Upload each task file with the Files API. See Creating tasks.
2. Submit with POST /rest/v1alpha1/eval_runs, passing the eval config inline as eval_config.content. Returns 202 with the run, or 400 VALIDATION_FAILED. validate_only: true returns 200 with a run preview and creates nothing.
3. Poll GET /rest/v1alpha1/eval_runs/{eval_run_id} no more than every 30 s, until lifecycle.report.status is REPORT_STATUS_AVAILABLE or lifecycle.status is EVAL_RUN_STATUS_COMPLETED, EVAL_RUN_STATUS_FAILED, or EVAL_RUN_STATUS_CANCELED.
4. Fetch GET /rest/v1alpha1/eval_runs/{eval_run_id}/report for per-task results and execution traces.
5. Stop a run with POST /rest/v1alpha1/eval_runs/{eval_run_id}/cancel.

Runs are visible only to the submitting user and the org’s eval admins; any other id returns 404 RESOURCE_NOT_FOUND.

Creating tasks

Tasks are uploaded with the Files API’s POST /rest/v1alpha1/files, one task file per request, then referenced by id in task_files[].file_id on POST /rest/v1alpha1/eval_runs.

curl -X POST "https://api.moveworks.ai/rest/v1alpha1/files" \
  -H "Authorization: Bearer $TOKEN" \
  -F "purpose=EVAL_RUNS_TASK" \
  -F "file=@TC-calendar.yaml"

• purpose must be EVAL_RUNS_TASK and must precede the file part. YAML is detected as text/plain, which this purpose accepts.
• The upload filename becomes the task file name: ≤ 128 chars matching ^[A-Za-z0-9_.-]+$, unique within the submission.
• The returned id is valid for 24 hours and only for the uploading credential. The run copies the bytes at submit.
• Every upload counts against the org rate limit (120 requests per minute). Put many tasks in one ----separated file.
• Upload checks are deferred to submit, which reports ISSUE_CODE_FILE_NOT_FOUND, ISSUE_CODE_FILE_WRONG_PURPOSE, ISSUE_CODE_FILE_TOO_LARGE, or ISSUE_CODE_DUPLICATE_TASK_FILE_NAME per task file.

The eval config is not uploaded; it is sent inline as eval_config.content.

Errors

Limits

Runs over the active-run limit (per submitting user and service-wide) wait as EVAL_RUN_STATUS_QUEUED; lifecycle.queue_reason gives the limit. A run stops counting once it is failed, canceled, or completed with cleanup terminal. By default one invalid task rejects the submission; on_invalid_task: ON_INVALID_TASK_SKIP drops invalid tasks instead.

Eval config file

YAML sent inline as eval_config.content, one per run.

title: Calendar Eval
requestor:
  name: Alice Chen
  email: alice.chen@example.com
scoped_agent_id: <id from the Agent Studio URL>
connector_map:
  - name: events
    mode: mocked
    eval_URL: https://<purplesuite host>/api/purple-suite/calendar
  - name: hr_directory
    mode: as_is

Task file

YAML with one or more tasks separated by ---.

goal_id: TC-090
frozen_time: '2026-04-13T10:30:00.000-07:00'
import_data:
  collections:
    employees:
      - id: EMP-001
        firstName: Alice
        lastName: Chen
        email: alice.chen@example.com
user_agent:
  user_agent_prompt: |
    You are Alice Chen (alice.chen@example.com), Pacific time.
    You asked the calendar assistant: "Book a meeting with Neal at 2pm today".
  user_turns:
    - utterance: Book a meeting with Neal at 2pm today
graders:
  - name: create
    type: deterministic
    collections_to_fetch: [events]
    query: >
      SELECT COUNT(*) FROM events
      WHERE id NOT IN ({initial_event_ids})
        AND start_date_time LIKE '2026-04-13T14:00:00%'
    expected_value: 1
  - name: response_quality
    type: llm_as_judge
    high_level_behavior: |
      - Response Content: Confirms a meeting with Neal Thompson at 2:00 PM PT

Graders

comparison_type values: equals/eq, not_equal/ne, greater_than/gt, greater_than_or_equal/gte, less_than/lt, less_than_or_equal/lte.