> This page is for API Reference & Tools. > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.moveworks.com/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.moveworks.com/_mcp/server. # Eval API overview > Run simulated-environment evals of an agent and fetch per-task reports with the full execution trace `v1alpha1` is alpha: no compatibility guarantee. An eval run is one [eval config](#eval-config-file) plus one or more [task files](#task-file). A task is one simulated conversation, identified by `goal_id` (unique within the run). The run executes in the org the bearer token belongs to, as the user the token is bound to, against a copy of the scoped agent's configuration that cleanup removes. ## Requirements * The org is enabled for the Eval API and the Files API. Otherwise: `404 NOT_FOUND`. * A Conversations API credential (`POST /oauth/v1/token`, client credentials) bound to a user with Creator Studio view access and, where enforced, MW Setup edit access. Otherwise: `403 REQUEST_ACCESS_DENIED`. ## Flow 1. Upload each [task file](#task-file) with the Files API. See [Creating tasks](#creating-tasks). 2. Submit with [`POST /rest/v1alpha1/eval_runs`](/api-reference/alpha-eval-api/eval-runs/create-eval-run), passing the eval config inline as `eval_config.content`. Returns `202` with the run, or `400 VALIDATION_FAILED`. `validate_only: true` returns `200` with a run preview and creates nothing. 3. Poll [`GET /rest/v1alpha1/eval_runs/{eval_run_id}`](/api-reference/alpha-eval-api/eval-runs/get-eval-run) no more than every 30 s, until `lifecycle.report.status` is `REPORT_STATUS_AVAILABLE` or `lifecycle.status` is `EVAL_RUN_STATUS_COMPLETED`, `EVAL_RUN_STATUS_FAILED`, or `EVAL_RUN_STATUS_CANCELED`. 4. Fetch [`GET /rest/v1alpha1/eval_runs/{eval_run_id}/report`](/api-reference/alpha-eval-api/eval-runs/get-eval-run-report) for per-task results and execution traces. 5. Stop a run with [`POST /rest/v1alpha1/eval_runs/{eval_run_id}/cancel`](/api-reference/alpha-eval-api/eval-runs/cancel-eval-run). Runs are visible only to the submitting user and the org's eval admins; any other id returns `404 RESOURCE_NOT_FOUND`. ## Creating tasks Tasks are uploaded with the Files API's [`POST /rest/v1alpha1/files`](/api-reference/alpha-files-api/files/upload-file), one [task file](#task-file) per request, then referenced by `id` in `task_files[].file_id` on [`POST /rest/v1alpha1/eval_runs`](/api-reference/alpha-eval-api/eval-runs/create-eval-run). ```bash curl -X POST "https://api.moveworks.ai/rest/v1alpha1/files" \ -H "Authorization: Bearer $TOKEN" \ -F "purpose=EVAL_RUNS_TASK" \ -F "file=@TC-calendar.yaml" ``` * `purpose` must be `EVAL_RUNS_TASK` and must precede the `file` part. YAML is detected as `text/plain`, which this purpose accepts. * The upload filename becomes the task file name: ≤ 128 chars matching `^[A-Za-z0-9_.-]+$`, unique within the submission. * The returned `id` is valid for 24 hours and only for the uploading credential. The run copies the bytes at submit. * Every upload counts against the org rate limit (120 requests per minute). Put many tasks in one `---`-separated file. * Upload checks are deferred to submit, which reports `ISSUE_CODE_FILE_NOT_FOUND`, `ISSUE_CODE_FILE_WRONG_PURPOSE`, `ISSUE_CODE_FILE_TOO_LARGE`, or `ISSUE_CODE_DUPLICATE_TASK_FILE_NAME` per task file. The eval config is not uploaded; it is sent inline as `eval_config.content`. ## Errors | Where | What | | ------------------------------------------- | ------------------------------------------------------------------------------------------------------ | | HTTP 4xx/5xx `{"error":{"code","message"}}` | Transport and authorization | | `400 VALIDATION_FAILED` `message` | Submission issues, one per line (`file#task CODE at path (line L, column C)`), up to 20 then `+N more` | | `EvalRun.lifecycle.error` | Run failure (`RunError`) | | `TaskResult.error` in the report | Task failure | | `EvalRun.lifecycle.cleanup` | Cleanup outcome, independent of `lifecycle.status` | ## Limits | Limit | Value | | --------------------------- | -------------------------------------------------------- | | Tasks per run | 500 | | Task size, fully expanded | 2 MiB (`ISSUE_CODE_TASK_FILE_TOO_LARGE`) | | Task file upload | 100 MiB each, 256 MiB per submission | | Eval config / inline body | 1 MiB / 4 MiB | | Seed data per run, expanded | 64 MiB (else `RUN_ERROR_CODE_ENVIRONMENT_SETUP_FAILED`) | | Upload filename | ≤ 128 chars, `^[A-Za-z0-9_.-]+$`, no `..`, not only dots | | Run duration | 6 hours, then force-failed | | Rate limit | Per org; `429` without `Retry-After` | Runs over the active-run limit (per submitting user and service-wide) wait as `EVAL_RUN_STATUS_QUEUED`; `lifecycle.queue_reason` gives the limit. A run stops counting once it is failed, canceled, or completed with cleanup terminal. By default one invalid task rejects the submission; `on_invalid_task: ON_INVALID_TASK_SKIP` drops invalid tasks instead. ## Eval config file YAML sent inline as `eval_config.content`, one per run. ```yaml title: Calendar Eval requestor: name: Alice Chen email: alice.chen@example.com scoped_agent_id: connector_map: - name: events mode: mocked eval_URL: https:///api/purple-suite/calendar - name: hr_directory mode: as_is ``` **`title`** `string` — required --- **`requestor`** `object` — required Persona the simulated user plays. Does not change who the conversations run as. Optional string attributes: `location`, `timezone` (default `America/Los_Angeles`), `country_code` (default `USA`), `last_name`, `preferred_name`, `pronunciation`, `employee_id`, `department`, `division`, `cost_center_id`, `cost_center_name`, `role`, `manager_email`, `office`, `city`, `state`, `region`, `seat_link`, `seat_name`. --- **`requestor.name`** `string` — required --- **`requestor.email`** `string` — required --- **`scoped_agent_id`** `string` — required --- **`platform`** `enum` — default: hera `hera` or `minerva`. Agents with MCP connectors require `hera`. --- **`mcp_enabled_apps`** `list of string` `hera` only. Simulated-app ids to enable for `connector_map` rows whose `eval_URL` is an MCP endpoint that names no app. --- **`source_org`** `string` If set, must be the token's org (else `403 REQUEST_ACCESS_DENIED`). --- **`target_org`** `string` If set, must equal `source_org` (else `ISSUE_CODE_UNSUPPORTED_TARGET_ORG`). --- **`connector_map`** `list of object` — required One entry per connector the agent calls. Undeclared connectors are skipped and reported as `WARNING_CODE_UNMAPPED_CONNECTOR_SKIPPED`. --- **`connector_map[].name`** `string` — required --- **`connector_map[].mode`** `enum` — default: mocked `mocked`: the connector points at `eval_URL`. `as_is`: the real connector and system, writes enabled; the run carries a warning. --- **`connector_map[].eval_URL`** `string` Required for `mocked`, omitted for `as_is`. --- **`connector_map[].original_URL`** `string` Disambiguates connectors that share a `name`. --- ## Task file YAML with one or more tasks separated by `---`. ```yaml goal_id: TC-090 frozen_time: '2026-04-13T10:30:00.000-07:00' import_data: collections: employees: - id: EMP-001 firstName: Alice lastName: Chen email: alice.chen@example.com user_agent: user_agent_prompt: | You are Alice Chen (alice.chen@example.com), Pacific time. You asked the calendar assistant: "Book a meeting with Neal at 2pm today". user_turns: - utterance: Book a meeting with Neal at 2pm today graders: - name: create type: deterministic collections_to_fetch: [events] query: > SELECT COUNT(*) FROM events WHERE id NOT IN ({initial_event_ids}) AND start_date_time LIKE '2026-04-13T14:00:00%' expected_value: 1 - name: response_quality type: llm_as_judge high_level_behavior: | - Response Content: Confirms a meeting with Neal Thompson at 2:00 PM PT ``` **`goal_id`** `string` — required --- **`frozen_time`** `string` Pinned "now" for date logic. --- **`import_data.collections`** `object` — required Seed records per collection, in the simulated app's API shape. --- **`user_agent.user_agent_prompt`** `string` — required --- **`user_agent.user_turns`** `list of object` Scripted turns (`utterance`). Omit for a goal-based task. --- **`graders`** `list of object` — required At least one. Every grader has `name` (unique within the task file), `type`, and optional `desc`. --- ### Graders | `type` | Fields | Checks | | --------------- | ----------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------- | | `deterministic` | `collections_to_fetch` (list of string), `query` (string), `expected_value`, `comparison_type` (default `equals`) | Read-only SQLite over the end state, as flattened snake_case tables. `{initial_event_ids}` expands to the seeded event ids. | | `llm_as_judge` | `high_level_behavior` (string) | One assertion per `- ` bullet. Optional `Category:` prefix (1–4 words) must be on every bullet or none. | | `tool_sequence` | `expected_sequence` (list of string) | Tools the agent called, order-insensitive. | `comparison_type` values: `equals`/`eq`, `not_equal`/`ne`, `greater_than`/`gt`, `greater_than_or_equal`/`gte`, `less_than`/`lt`, `less_than_or_equal`/`lte`.