Parameters
Prompt Files
Prompt files are plain text files that contain the prompts you want to test. If you have only one file, you can include multiple prompts in the file, separated by the delimiter ---
. If you have multiple files, each prompt should be in a separate file.
You can use Jinja templating syntax to include variables in your prompts, which will be replaced with actual values from the vars
CSV file during evaluation.
Example of a single prompt file with multiple prompts (prompts.txt
):
Translate the following text to French: "{{name}}: {{text}}"
---
Translate the following text to German: "{{name}}: {{text}}"
Vars File
The Vars file is a CSV, JSON, or YAML file that contains the values for the variables used in the prompts. The first row of the CSV file should contain the variable names, and each subsequent row should contain the corresponding values for each test case.
Vars are substituted by Jinja templating syntax into prompts.
Example of a vars file (vars.csv
):
"name","text"
"Bob","Hello, world!"
"Joe","Goodbye, everyone!"
Example of a vars file (vars.json
):
[
{ "name": "Bob", "text": "Hello, world!" },
{ "name": "Joe", "text": "Goodbye, everyone!" }
]
Output File
The results of the evaluation are written to this file. Each record in the output file corresponds to a test case and includes the original prompt, the output generated by the LLM, and the values of the variables used in the test case.
For example outputs, see the examples/ directory.
Configuration File
You can specify any option in a configuration file (e.g., config.json
). This can help you avoid repetitive command-line options and simplify the CLI invocation.
By default, no config is loaded.
Example of a configuration file:
{
"providers": ["openai:completion"],
"vars": "/path/to/vars.csv"
}
EvaluateOptions
Property | Type | Required | Description |
---|---|---|---|
providers | [] | Yes | A list of API providers to be evaluated |
prompts | [str] | Yes | A list of prompts to be evaluated. |
vars | PosixPath | No | A list of variables names and values that are substituted in the prompts |
prompt | PosixPath | No | An object containing options related to the prompts. |