Testing in Playground

Testing in the Playground

The TestingEditor is a built-in testing panel in the playground that lets you write and run eval configs in YAML without leaving the editor. It uses the same eval engine as the full Evals system but with a streamlined workflow for development.

YAML Config Format

The testing panel uses a YAML format compatible with the VS Code extension:

suite: "Playground Tests"
config:
  target: ../prompt.pdk

datasets:
  greetings:
    description: "Greeting test data"
    cases:
      - name: formal
        input:
          name: "Dr. Smith"
          tone: formal
      - name: casual
        input:
          name: "Alice"
          tone: casual

tests:
  - name: "Contains name"
    dataset: greetings
    expect_render:
      - contains: "{{name}}"

  - name: "Quick check"
    given:
      name: "Bob"
    expect_render:
      - starts_with: "Hello"
    expect_llm:
      - sentiment: positive

Real-Time Validation

The editor validates your YAML as you type, with inline diagnostics for:

• YAML syntax errors
• Missing required fields (suite name, test names)
• Unknown assertion operators
• Operator placement errors (render operator in expect_llm, or vice versa)
• Undefined dataset references
• Tests without any assertions

Split File Mode

You can also work with separate files matching the VS Code extension format:

• .eval files - Suite config and tests
• .dset files - Dataset definitions

The testing panel supports both combined (single YAML) and split file workflows. Changes in split mode merge back into the combined config automatically.

Monaco Integration

The YAML editor is powered by Monaco (the same editor as VS Code) with:

• Autocomplete for assertion operators
• Hover documentation for each operator
• Inline error squiggles
• Syntax highlighting for YAML