EchoStash

What Are Datasets?

Datasets are collections of named test cases, each containing a set of input variables. When you run a test that references a dataset, the test runs once per case in the dataset, producing parameterized results.

Creating a Dataset

Create datasets in the Evals tab or via the API. Each dataset has a unique ID (slug) and contains one or more named cases:

http

POST /api/prompts/{promptId}/eval/datasets

{
  "datasetId": "customer-greetings",
  "description": "Various customer greeting scenarios",
  "cases": [
    { "name": "formal", "input": { "name": "Dr. Smith", "tone": "formal" } },
    { "name": "casual", "input": { "name": "Alice", "tone": "casual" } },
    { "name": "no-name", "input": { "tone": "friendly" } }
  ]
}

Inline Variables vs. Datasets

You can provide variables in two ways:

Inline (given) - Variables are defined directly in the test. Good for one-off tests.
Dataset reference (dataset) - The test runs against every case in the referenced dataset. Good for parameterized testing.

yaml

tests:
  # Inline variables — runs once
  - name: "Quick check"
    given:
      name: "Alice"
    expect_render:
      - contains: "Alice"

  # Dataset reference — runs once per case
  - name: "All customers"
    dataset: customer-greetings
    expect_render:
      - contains: "Hello"
      - word_count:
          min: 10

Importing from CSV/JSON

Upload a CSV or JSON file to create a dataset (max 5MB, 1000 rows). The importer maps columns to variable names automatically. For CSV files, you can specify which column should be used as the case name.

http

POST /api/prompts/{promptId}/eval/datasets/import
Content-Type: multipart/form-data

file: customers.csv
nameColumn: customer_id

CSV example:

csv

customer_id,name,tone,language
vip_001,Dr. Smith,formal,en
casual_001,Alice,casual,en
intl_001,Yuki,polite,ja

Dataset API

Method	Endpoint	Description
`GET`	`/eval/datasets`	List all datasets
`POST`	`/eval/datasets`	Create a dataset
`POST`	`/eval/datasets/import`	Import from CSV/JSON file
`PUT`	`/eval/datasets/:id`	Update a dataset
`DELETE`	`/eval/datasets/:id`	Delete a dataset