A/B tests
The 7 tools of the A/B tests toolset in the Analytics and data area, with their permissions, approval rules and inputs.
On this page
Toolset experiments, 7 tools.
Records
experiment
A storefront A/B test: its setup, status, arms and where it is used. include results adds the arm figures, each chance to beat control, the guardrails, the verdict and the health checks.
- Permission:
settings.read - Tools:
experiment_list,experiment_get - Includes:
results
Filters on the list:
| Filter | Type | Required | Description |
|---|---|---|---|
tab | string | one of "running", "scheduled", "drafts", "ended", "archived" |
Actions
experiment_create
Create a draft A/B test. Nothing changes for shoppers until it is started.
Arms need a control arm, 2 to 4 arms in all, and weights that sum to 100. Build the content of each arm in the Customizer or the banner editor afterwards; a draft does nothing until then.
- Action key:
experiment.create - Permission:
settings.write - Approval: Waits for approval when the store requires it for this action.
- Effect: Changes a draft; nothing goes live until it is published.
- Retries: runs at most once; a repeat with the same idempotency key returns the first result.
- Needs experiments on the store's plan
| Field | Type | Required | Description |
|---|---|---|---|
name | string | yes | 1 to 120 characters |
hypothesis | string | at most 1000 characters | |
surface | string | yes | one of "theme", "banner" |
arms | list of object | yes | 2 to 4 items |
arms[].key | string | yes | one of "control", "b", "c", "d" |
arms[].name | string | yes | 1 to 80 characters |
arms[].weight | integer | yes | 1 to 99 |
primary_metric | string | yes | one of "conversion", "add_to_cart", "checkout_start", "revenue_per_visitor", "aov", "banner_click", "form_submit", "automation_conversion", "automation_revenue" |
guardrails | list of string | at most 3 items. one of "conversion", "add_to_cart", "checkout_start", "revenue_per_visitor", "aov", "banner_click", "form_submit", "automation_conversion", "automation_revenue" | |
traffic_share | integer | 1 to 100 | |
min_days | integer | 7 to 90 |
Example input:
{
"name": "Hero headline",
"surface": "theme",
"arms": [
{
"key": "control",
"name": "Current headline",
"weight": 50
},
{
"key": "b",
"name": "Shorter headline",
"weight": 50
}
],
"primary_metric": "conversion"
}experiment_end
End a test for good: everyone sees the original again until an arm is shipped, and its results freeze.
- Action key:
experiment.end - Permission:
settings.write - Approval: Always waits for a person to approve it.
- Effect: Changes the live store.
- Destructive: in compact mode it runs with
run_destructive_action. - Retries: safe to repeat with the same input.
| Field | Type | Required | Description |
|---|---|---|---|
experiment_id | string | yes | A/B test ID from experiment_list |
Example input:
{
"experiment_id": "019bb13b-634d-7e53-b5ee-25855ef03835"
}experiment_pause
Pause a running test: everyone sees the original and nothing is counted. It keeps its place.
- Action key:
experiment.pause - Permission:
settings.write - Approval: Waits for approval when the store requires it for this action.
- Effect: Changes the live store.
- Retries: safe to repeat with the same input.
| Field | Type | Required | Description |
|---|---|---|---|
experiment_id | string | yes | A/B test ID from experiment_list |
Example input:
{
"experiment_id": "019bb13b-634d-7e53-b5ee-25855ef03835"
}experiment_ship
Show one arm of an ended test to everyone, at once and with no publish needed.
- Action key:
experiment.ship - Permission:
settings.write - Approval: Always waits for a person to approve it.
- Effect: Changes the live store.
- Retries: safe to repeat with the same input.
| Field | Type | Required | Description |
|---|---|---|---|
experiment_id | string | yes | A/B test ID from experiment_list |
arm | string | yes | one of "control", "b", "c", "d" |
Example input:
{
"experiment_id": "019bb13b-634d-7e53-b5ee-25855ef03835",
"arm": "b"
}experiment_start
Start a draft or scheduled test: visitors begin seeing its arms. It takes one place of the plan.
- Action key:
experiment.start - Permission:
settings.write - Approval: Always waits for a person to approve it.
- Effect: Changes the live store.
- Retries: safe to repeat with the same input.
- Needs experiments on the store's plan
| Field | Type | Required | Description |
|---|---|---|---|
experiment_id | string | yes | A/B test ID from experiment_list |
Example input:
{
"experiment_id": "019bb13b-634d-7e53-b5ee-25855ef03835"
}