> ## Documentation Index
> Fetch the complete documentation index at: https://docs.guidinghand.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Start a test run

> Runs a test set’s cases, or a list of cases, against each of `agents`. It freezes a copy of each case (and its start state’s version), so changing them later doesn’t change the run. At most 5000 attempts per run. Members can start runs. `test_run.completed` is sent when it’s done.



## OpenAPI

````yaml /api-reference/openapi.json post /v1/test_runs
openapi: 3.1.0
info:
  title: GuidingHand API
  version: 1.0.0
  description: >-
    Create sessions (an invite link with a code for the person at the computer),
    run tasks on their computer with one of your agents, follow them, answer
    their questions and approvals, and fetch history and recordings. Test your
    agents with evals: test cases on fresh machines, run in test sets, scored by
    a check script or a rubric. Agents are versioned (save a draft, publish it),
    have tools (their own file system, and your HTTP endpoints), and every
    change is in the org’s audit log.
servers:
  - url: https://guidinghand.ai
    description: Production
  - url: https://dev.guidinghand.ai
    description: Development (Stripe test mode)
security:
  - bearerAuth: []
tags:
  - name: Agents
  - name: Agent files
  - name: Agent tools
  - name: Sessions
  - name: Tasks
  - name: Webhooks
  - name: Audit log
  - name: Test cases
  - name: Test sets
  - name: Test runs
  - name: Start states
  - name: Runners
paths:
  /v1/test_runs:
    post:
      tags:
        - Test runs
      summary: Start a test run
      description: >-
        Runs a test set’s cases, or a list of cases, against each of `agents`.
        It freezes a copy of each case (and its start state’s version), so
        changing them later doesn’t change the run. At most 5000 attempts per
        run. Members can start runs. `test_run.completed` is sent when it’s
        done.
      operationId: createTestRun
      requestBody:
        required: true
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/TestRunInput'
      responses:
        '200':
          description: OK
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/TestRun'
        '400':
          $ref: '#/components/responses/E400'
        '401':
          $ref: '#/components/responses/E401'
        '403':
          $ref: '#/components/responses/E403'
        '404':
          $ref: '#/components/responses/E404'
components:
  schemas:
    TestRunInput:
      type: object
      description: Send `test_set_id` or `test_case_ids`.
      properties:
        test_set_id:
          type: string
          description: Run this test set’s cases.
        test_case_ids:
          type: array
          items:
            type: string
          description: Or run these test cases (ids or keys).
        agents:
          type: array
          items:
            type: string
          default:
            - default
          description: The agents to test, side by side.
        repeat:
          type: integer
          minimum: 1
          maximum: 20
          description: Tries per case for every case.
        repeat_by_case:
          type: object
          additionalProperties:
            type: integer
            minimum: 1
            maximum: 20
          description: >-
            Tries for some cases, by id or key. Tries per case:
            `repeat_by_case[case]`, else `repeat`, else the set’s per-case
            `repeat`, else the set’s `repeat`, else 1.
          example:
            DSP-01: 5
        name:
          type: string
        metadata:
          $ref: '#/components/schemas/Metadata'
    TestRun:
      type: object
      description: >-
        A run of test cases against one or more agents, N tries each. Every try
        is an attempt.
      properties:
        object:
          const: test_run
        test_run_id:
          type: string
          example: tr_Vb7sT2Lc0aQ3
        name:
          type: string
        test_set_id:
          type:
            - string
            - 'null'
          description: The test set it ran, or null for a list of test cases.
        agents:
          type: array
          items:
            type: string
          example:
            - default
            - lenovo-support
        status:
          $ref: '#/components/schemas/TestRunStatus'
          description: '`completed` once none of its attempts are left.'
        progress:
          type: object
          description: >-
            The run’s tries: how many are queued, running, passed, failed or
            ended in an error.
          properties:
            total:
              type: integer
            queued:
              type: integer
            running:
              type: integer
            passed:
              type: integer
            failed:
              type: integer
            error:
              type: integer
        pass_rate:
          type: object
          additionalProperties:
            type:
              - number
              - 'null'
            minimum: 0
            maximum: 1
          description: Per agent, from 0 to 1. Errors don’t count against an agent.
          example:
            default: 0.54
            lenovo-support: 0.63
        metadata:
          $ref: '#/components/schemas/Metadata'
        created_at:
          type: string
          format: date-time
        started_at:
          type:
            - string
            - 'null'
          format: date-time
        finished_at:
          type:
            - string
            - 'null'
          format: date-time
    Metadata:
      type: object
      additionalProperties:
        type: string
        maxLength: 500
      maxProperties: 50
      description: >-
        Your own ids and labels (e.g. a ticket or customer id), returned as
        given.
      example:
        customer_id: cus_42
    TestRunStatus:
      type: string
      enum:
        - queued
        - running
        - completed
        - cancelled
    Error:
      type: object
      required:
        - error
      properties:
        error:
          type: object
          required:
            - type
            - message
          properties:
            type:
              type: string
              enum:
                - invalid_request
                - authentication
                - payment_required
                - permission
                - not_found
                - conflict
                - rate_limit
                - server_error
            message:
              type: string
            code:
              type: string
              description: >-
                Why it was refused, when there’s more to say than the status.
                Starting a task: `insufficient_balance`,
                `auto_topup_limit_reached` or `card_needs_attention` (402: the
                balance can’t cover a minute; add funds, raise the auto top-up
                limit, or update the card), `concurrency_limit` (429),
                `model_unavailable` (503: the agent’s `model` can’t run on this
                server; choose another). On `/respond`: `already_answered`
                (someone answered or decided first: see `answered_by`) or
                `not_pending` (that question or approval isn’t open any more).
                The GuidingHand app hears two more, which the API never returns:
                `customer_answers_off` and `customer_approvals_off` (the agent
                keeps its questions or approvals for your team). Deleting a
                start state that test cases use: `in_use` (409, with
                `test_case_ids`).
            answered_by:
              type: string
              enum:
                - customer
                - operator
                - timeout
              description: >-
                With `already_answered`: who answered or decided first
                (`customer`: the person at the computer; `timeout`: nobody
                decided an approval within the agent’s
                `approval_timeout_minutes`, so it was denied).
            test_case_ids:
              type: array
              items:
                type: string
              description: 'With `in_use`: the test cases that use the start state.'
  responses:
    E400:
      description: The request is invalid.
      content:
        application/json:
          schema:
            $ref: '#/components/schemas/Error'
    E401:
      description: Missing or invalid API key.
      content:
        application/json:
          schema:
            $ref: '#/components/schemas/Error'
    E403:
      description: Your role can’t do this.
      content:
        application/json:
          schema:
            $ref: '#/components/schemas/Error'
    E404:
      description: Not found in this org.
      content:
        application/json:
          schema:
            $ref: '#/components/schemas/Error'
  securitySchemes:
    bearerAuth:
      type: http
      scheme: bearer
      description: An org API key (`gh_live_…`) from Settings → API keys in the console.

````