Start a test run
curl --request POST \
--url https://guidinghand.ai/v1/test_runs \
--header 'Authorization: Bearer <token>' \
--header 'Content-Type: application/json' \
--data '
{
"test_set_id": "<string>",
"test_case_ids": [
"<string>"
],
"agents": [
"default"
],
"repeat": 10,
"repeat_by_case": {
"DSP-01": 5
},
"name": "<string>",
"metadata": {
"customer_id": "cus_42"
}
}
'const options = {
method: 'POST',
headers: {Authorization: 'Bearer <token>', 'Content-Type': 'application/json'},
body: JSON.stringify({
test_set_id: '<string>',
test_case_ids: ['<string>'],
agents: ['default'],
repeat: 10,
repeat_by_case: {'DSP-01': 5},
name: '<string>',
metadata: {customer_id: 'cus_42'}
})
};
fetch('https://guidinghand.ai/v1/test_runs', options)
.then(res => res.json())
.then(res => console.log(res))
.catch(err => console.error(err));import requests
url = "https://guidinghand.ai/v1/test_runs"
payload = {
"test_set_id": "<string>",
"test_case_ids": ["<string>"],
"agents": ["default"],
"repeat": 10,
"repeat_by_case": { "DSP-01": 5 },
"name": "<string>",
"metadata": { "customer_id": "cus_42" }
}
headers = {
"Authorization": "Bearer <token>",
"Content-Type": "application/json"
}
response = requests.post(url, json=payload, headers=headers)
print(response.text){
"object": "test_run",
"test_run_id": "tr_Vb7sT2Lc0aQ3",
"name": "<string>",
"test_set_id": "<string>",
"agents": [
"default",
"lenovo-support"
],
"status": "queued",
"progress": {
"total": 123,
"queued": 123,
"running": 123,
"passed": 123,
"failed": 123,
"error": 123
},
"pass_rate": {
"default": 0.54,
"lenovo-support": 0.63
},
"metadata": {
"customer_id": "cus_42"
},
"created_at": "2023-11-07T05:31:56Z",
"started_at": "2023-11-07T05:31:56Z",
"finished_at": "2023-11-07T05:31:56Z"
}{
"error": {
"type": "invalid_request",
"message": "<string>",
"code": "<string>",
"answered_by": "customer",
"test_case_ids": [
"<string>"
]
}
}{
"error": {
"type": "invalid_request",
"message": "<string>",
"code": "<string>",
"answered_by": "customer",
"test_case_ids": [
"<string>"
]
}
}{
"error": {
"type": "invalid_request",
"message": "<string>",
"code": "<string>",
"answered_by": "customer",
"test_case_ids": [
"<string>"
]
}
}{
"error": {
"type": "invalid_request",
"message": "<string>",
"code": "<string>",
"answered_by": "customer",
"test_case_ids": [
"<string>"
]
}
}Test runs
Start a test run
Runs a test set’s cases, or a list of cases, against each of agents. It freezes a copy of each case (and its start state’s version), so changing them later doesn’t change the run. At most 5000 attempts per run. Members can start runs. test_run.completed is sent when it’s done.
POST
/
v1
/
test_runs
Start a test run
curl --request POST \
--url https://guidinghand.ai/v1/test_runs \
--header 'Authorization: Bearer <token>' \
--header 'Content-Type: application/json' \
--data '
{
"test_set_id": "<string>",
"test_case_ids": [
"<string>"
],
"agents": [
"default"
],
"repeat": 10,
"repeat_by_case": {
"DSP-01": 5
},
"name": "<string>",
"metadata": {
"customer_id": "cus_42"
}
}
'const options = {
method: 'POST',
headers: {Authorization: 'Bearer <token>', 'Content-Type': 'application/json'},
body: JSON.stringify({
test_set_id: '<string>',
test_case_ids: ['<string>'],
agents: ['default'],
repeat: 10,
repeat_by_case: {'DSP-01': 5},
name: '<string>',
metadata: {customer_id: 'cus_42'}
})
};
fetch('https://guidinghand.ai/v1/test_runs', options)
.then(res => res.json())
.then(res => console.log(res))
.catch(err => console.error(err));import requests
url = "https://guidinghand.ai/v1/test_runs"
payload = {
"test_set_id": "<string>",
"test_case_ids": ["<string>"],
"agents": ["default"],
"repeat": 10,
"repeat_by_case": { "DSP-01": 5 },
"name": "<string>",
"metadata": { "customer_id": "cus_42" }
}
headers = {
"Authorization": "Bearer <token>",
"Content-Type": "application/json"
}
response = requests.post(url, json=payload, headers=headers)
print(response.text){
"object": "test_run",
"test_run_id": "tr_Vb7sT2Lc0aQ3",
"name": "<string>",
"test_set_id": "<string>",
"agents": [
"default",
"lenovo-support"
],
"status": "queued",
"progress": {
"total": 123,
"queued": 123,
"running": 123,
"passed": 123,
"failed": 123,
"error": 123
},
"pass_rate": {
"default": 0.54,
"lenovo-support": 0.63
},
"metadata": {
"customer_id": "cus_42"
},
"created_at": "2023-11-07T05:31:56Z",
"started_at": "2023-11-07T05:31:56Z",
"finished_at": "2023-11-07T05:31:56Z"
}{
"error": {
"type": "invalid_request",
"message": "<string>",
"code": "<string>",
"answered_by": "customer",
"test_case_ids": [
"<string>"
]
}
}{
"error": {
"type": "invalid_request",
"message": "<string>",
"code": "<string>",
"answered_by": "customer",
"test_case_ids": [
"<string>"
]
}
}{
"error": {
"type": "invalid_request",
"message": "<string>",
"code": "<string>",
"answered_by": "customer",
"test_case_ids": [
"<string>"
]
}
}{
"error": {
"type": "invalid_request",
"message": "<string>",
"code": "<string>",
"answered_by": "customer",
"test_case_ids": [
"<string>"
]
}
}Authorizations
An org API key (gh_live_…) from Settings → API keys in the console.
Body
application/json
Send test_set_id or test_case_ids.
Run this test set’s cases.
Or run these test cases (ids or keys).
The agents to test, side by side.
Tries per case for every case.
Required range:
1 <= x <= 20Tries for some cases, by id or key. Tries per case: repeat_by_case[case], else repeat, else the set’s per-case repeat, else the set’s repeat, else 1.
Show child attributes
Show child attributes
Example:
{ "DSP-01": 5 }
Your own ids and labels (e.g. a ticket or customer id), returned as given.
Show child attributes
Show child attributes
Example:
{ "customer_id": "cus_42" }
Response
OK
A run of test cases against one or more agents, N tries each. Every try is an attempt.
Example:
"tr_Vb7sT2Lc0aQ3"
The test set it ran, or null for a list of test cases.
Example:
["default", "lenovo-support"]
completed once none of its attempts are left.
Available options:
queued, running, completed, cancelled The run’s tries: how many are queued, running, passed, failed or ended in an error.
Show child attributes
Show child attributes
Per agent, from 0 to 1. Errors don’t count against an agent.
Show child attributes
Show child attributes
Example:
{ "default": 0.54, "lenovo-support": 0.63 }
Your own ids and labels (e.g. a ticket or customer id), returned as given.
Show child attributes
Show child attributes
Example:
{ "customer_id": "cus_42" }