Skip to main content

Parameters

Prompt Files

Prompt files are plain text files that contain the prompts you want to test. If you have only one file, you can include multiple prompts in the file, separated by the delimiter ---. If you have multiple files, each prompt should be in a separate file.

You can use Jinja templating syntax to include variables in your prompts, which will be replaced with actual values from the vars CSV file during evaluation.

Example of a single prompt file with multiple prompts (prompts.txt):

Translate the following text to French: "{{name}}: {{text}}"
---
Translate the following text to German: "{{name}}: {{text}}"

Vars File

The Vars file is a CSV, JSON, or YAML file that contains the values for the variables used in the prompts. The first row of the CSV file should contain the variable names, and each subsequent row should contain the corresponding values for each test case.

Vars are substituted by Jinja templating syntax into prompts.

Example of a vars file (vars.csv):

"name","text"
"Bob","Hello, world!"
"Joe","Goodbye, everyone!"

Example of a vars file (vars.json):

[
{ "name": "Bob", "text": "Hello, world!" },
{ "name": "Joe", "text": "Goodbye, everyone!" }
]

Output File

The results of the evaluation are written to this file. Each record in the output file corresponds to a test case and includes the original prompt, the output generated by the LLM, and the values of the variables used in the test case.

For example outputs, see the examples/ directory.

Configuration File

You can specify any option in a configuration file (e.g., config.json). This can help you avoid repetitive command-line options and simplify the CLI invocation.

By default, no config is loaded.

Example of a configuration file:

{
"providers": ["openai:completion"],
"vars": "/path/to/vars.csv"
}

EvaluateOptions

PropertyTypeRequiredDescription
providers[]YesA list of API providers to be evaluated
prompts[str]YesA list of prompts to be evaluated.
varsPosixPathNoA list of variables names and values that are substituted in the prompts
promptPosixPathNoAn object containing options related to the prompts.