Parameters

Prompt Files

Prompt files are plain text files that contain the prompts you want to test. If you have only one file, you can include multiple prompts in the file, separated by the delimiter ---. If you have multiple files, each prompt should be in a separate file.

You can use Jinja templating syntax to include variables in your prompts, which will be replaced with actual values from the vars CSV file during evaluation.

Example of a single prompt file with multiple prompts (prompts.txt):

Translate the following text to French: "{{name}}: {{text}}"
---
Translate the following text to German: "{{name}}: {{text}}"

Vars File

The Vars file is a CSV, JSON, or YAML file that contains the values for the variables used in the prompts. The first row of the CSV file should contain the variable names, and each subsequent row should contain the corresponding values for each test case.

Vars are substituted by Jinja templating syntax into prompts.

Example of a vars file (vars.csv):

"name","text"
"Bob","Hello, world!"
"Joe","Goodbye, everyone!"

Example of a vars file (vars.json):

[
  { "name": "Bob", "text": "Hello, world!" },
  { "name": "Joe", "text": "Goodbye, everyone!" }
]

Output File

The results of the evaluation are written to this file. Each record in the output file corresponds to a test case and includes the original prompt, the output generated by the LLM, and the values of the variables used in the test case.

For example outputs, see the examples/ directory.

Configuration File

You can specify any option in a configuration file (e.g., config.json). This can help you avoid repetitive command-line options and simplify the CLI invocation.

By default, no config is loaded.

Example of a configuration file:

{
  "providers": ["openai:completion"],
  "vars": "/path/to/vars.csv"
}

EvaluateOptions

Property	Type	Required	Description
providers	[]	Yes	A list of API providers to be evaluated
prompts	[str]	Yes	A list of prompts to be evaluated.
vars	PosixPath	No	A list of variables names and values that are substituted in the prompts
prompt	PosixPath	No	An object containing options related to the prompts.