Experiments API

Experiments let you compare two versions of an agent configuration and measure which performs better.

All endpoints are organization-scoped: /:org_id/experiments.

List experiments

GET /:org_id/experiments

Create experiment

POST /:org_id/experiments
Content-Type: application/json

{
  "name": "Rails expert v2 test",
  "resource_id": "resource_abc123",
  "control_version_hash": "hash_control...",
  "variant_version_hash": "hash_variant...",
  "traffic_split": 50
}

Show experiment

GET /:org_id/experiments/:id

Update experiment

PATCH /:org_id/experiments/:id
Content-Type: application/json

{
  "traffic_split": 70
}

Experiment lifecycle

Start

POST /:org_id/experiments/:id/start

Begins routing traffic to control and variant groups.

Stop

POST /:org_id/experiments/:id/stop

Stops the experiment. No new sessions are assigned to variants.

Deploy

POST /:org_id/experiments/:id/deploy
Content-Type: application/json

{
  "variant": "variant"
}

Promotes the winning variant to production. The experiment is archived.

Metrics

GET /:org_id/experiments/:id/metrics

Returns metrics for both control and variant groups:

{
  "control": {
    "total_sessions": 100,
    "success_rate": 85.5,
    "avg_cost": 0.0234,
    "avg_turns": 12.5,
    "p95_cost": 0.0567
  },
  "variant": {
    "total_sessions": 98,
    "success_rate": 89.2,
    "avg_cost": 0.0198,
    "avg_turns": 10.8,
    "p95_cost": 0.0489
  },
  "significance": {
    "p_value": 0.05,
    "significant": false,
    "confidence": 0.95
  }
}

Delete experiment

DELETE /:org_id/experiments/:id