Skip to main content

Evaluating Agents

The framework includes an evaluation system that helps you test your agents' behavior in a structured way. You can define test suites, run them against your agents, and generate detailed reports to analyze the results. When running evaluations locally, you can also benchmark different language models to see how they affect your agents' responses.

At a high level, the evaluation process involves three main components:

  • Test Case: A JSON file that defines a single, specific task for an agent to perform. It includes the initial prompt, any required files (artifacts), and the criteria for a successful outcome.
  • Test Suite: A JSON file that groups one or more test cases into a single evaluation run. It also defines the environment for the evaluation, such as whether to run the agents locally or connect to a remote Agent Mesh.
  • Evaluation Settings: A configuration block within the test suite that specifies how to score the agent's performance. You can choose from several methods, from simple metric-based comparisons to more advanced evaluations using a language model.

This document guides you through creating test cases, assembling them into test suites, and running evaluations to test your agents.

Creating a Test Case​

A test case is a JSON file that defines a specific task for an agent to perform. It serves as the fundamental building block of an evaluation. You create a test case to represent a single interaction you want to test, such as asking a question, providing a file for processing, or requesting a specific action from an agent.

Test Case Configuration​

The following fields are available in the test case JSON file.

  • test_case_id (Required): A unique identifier for the test case.
  • query (Required): The initial prompt to be sent to the agent.
  • target_agent (Required): The name of the agent to which the query should be sent.
  • category (Optional): The category of the test case. Defaults to Other.
  • description (Optional): A description of the test case.
  • artifacts (Optional): A list of artifacts to be sent with the initial query. Each artifact has a type and a path.
  • wait_time (Optional): The maximum time in seconds to wait for a response from the agent. Defaults to 60.
  • evaluation (Optional): The evaluation criteria for the test case.
    • expected_tools (Optional): A list of tools that the agent is expected to use. Defaults to an empty list.
    • expected_response (Optional): The expected final response from the agent. Defaults to an empty string.
    • criterion (Optional): The criterion to be used by the llm_evaluator. Defaults to an empty string.

Test Case Examples​

Here is an example of a simple test case. It sends a greeting to an agent and checks for a standard response.

{
    "test_case_id": "hello_world",
    "category": "Content Generation",
    "description": "A simple test case to check the basic functionality of the system.",
    "query": "Hello, world!",
    "target_agent": "OrchestratorAgent",
    "wait_time": 30,
    "evaluation": {
        "expected_tools": [],
        "expected_response": "Hello! How can I help you today?",
        "criterion": "Evaluate if the agent provides a standard greeting."
    }
}

This next example shows a more complex test case. It includes a CSV file as an artifact and asks the agent to filter the data in the file.

{
  "test_case_id": "filter_csv_employees_by_age_and_country",
  "category": "Tool Usage",
  "description": "A test case to filter employees from a CSV file based on age and country.",
  "target_agent": "OrchestratorAgent",
  "query": "From the attached CSV, please list the names of all people who are older or equal to 30 and live in the USA.",
  "artifacts": [
    {
      "type": "file",
      "path": "artifacts/sample.csv"
    }
  ],
  "wait_time": 120,
  "evaluation": {
    "expected_tools": ["extract_content_from_artifact"],
    "expected_response": "The person who is 30 or older and lives in the USA is John Doe.",
    "criterion": "Evaluate if the agent correctly filters the CSV data."
  }
}

Creating a Test Suite​

The test suite is a JSON file that defines the parameters of an evaluation run. You use it to group test cases and configure the environment in which they run.

A common convention in the test suite configuration is to use keys ending with _VAR. These keys indicate that the corresponding value is the name of an environment variable from which the framework should read the actual value. This practice helps you keep sensitive information—like API keys and credentials—out of your configuration files. This convention applies to the broker object, the env object within llm_models, and the env object within the llm_evaluator in evaluation_settings.

You can run evaluations in two modes: local and remote. Both modes require a connection to a Solace event broker to function.

Local Evaluation​

In a local evaluation, the evaluation framework brings up a local instance of Solace Agent Mesh (SAM) and runs the agents on your local machine. This mode is useful for development and testing because it allows you to iterate quickly on your agents and test cases. You can also use this mode to benchmark different language models against your agents to see how they perform.

To run a local evaluation, you need to install the sam-rest-gateway plugin. This plugin allows the evaluation framework to communicate with the local SAM instance. You can install it with the following command:

pip install "sam-rest-gateway @ git+https://github.com/SolaceLabs/solace-agent-mesh-core-plugins#subdirectory=sam-rest-gateway"

Local Test Suite Configuration​

For a local evaluation, you must define the agents, broker, llm_models, and test_cases fields.

The agents field is a required list of paths to the agent configuration files. You must specify at least one agent.

"agents": [ "examples/agents/a2a_agents_example.yaml" ]

The broker field is a required object containing the connection details for the Solace event broker.

"broker": {
    "SOLACE_BROKER_URL_VAR": "SOLACE_BROKER_URL",
    "SOLACE_BROKER_USERNAME_VAR": "SOLACE_BROKER_USERNAME",
    "SOLACE_BROKER_PASSWORD_VAR": "SOLACE_BROKER_PASSWORD",
    "SOLACE_BROKER_VPN_VAR": "SOLACE_BROKER_VPN"
}

The llm_models field is a required list of language models to use. You must specify at least one model. The env object contains environment variables required by the model, such as the model name, endpoint, and API key.

"llm_models": [
    {
        "name": "gpt-4-1",
        "env": {
            "LLM_SERVICE_PLANNING_MODEL_NAME": "openai/azure-gpt-4-1",
            "LLM_SERVICE_ENDPOINT_VAR": "LLM_SERVICE_ENDPOINT",
            "LLM_SERVICE_API_KEY_VAR": "LLM_SERVICE_API_KEY"
        }
    },
    {
        "name": "gemini-1.5-pro",
        "env": {
            "LLM_SERVICE_PLANNING_MODEL_NAME": "google/gemini-1.5-pro-latest",
            "LLM_SERVICE_ENDPOINT_VAR": "LLM_SERVICE_ENDPOINT_GOOGLE",
            "LLM_SERVICE_API_KEY_VAR": "LLM_SERVICE_API_KEY_GOOGLE"
        }
    }
]

The test_cases field is a required list of paths to the test case JSON files. You must specify at least one test case.

"test_cases": [ "tests/evaluation/test_cases/hello_world.test.json" ]

You can also provide optional settings for results_dir_name, runs, workers, and evaluation_settings.

The results_dir_name field is an optional string that specifies the name of the directory for evaluation results. It defaults to tests.

"results_dir_name": "my-local-test-results"

The runs field is an optional integer that specifies the number of times to run each test case. It defaults to 1.

"runs": 3

The workers field is an optional integer that specifies the number of parallel workers for running tests. It defaults to 4.

"workers": 8

The evaluation_settings field is an optional object that allows you to configure the evaluation. This object can contain tool_match, response_match, and llm_evaluator settings.

"evaluation_settings": {
    "tool_match": {
        "enabled": true
    },
    "response_match": {
        "enabled": true
    },
    "llm_evaluator": {
        "enabled": true,
        "env": {
            "LLM_SERVICE_PLANNING_MODEL_NAME": "openai/gemini-2.5-pro",
            "LLM_SERVICE_ENDPOINT_VAR": "LLM_SERVICE_ENDPOINT",
            "LLM_SERVICE_API_KEY_VAR": "LLM_SERVICE_API_KEY"
        }
    }
}

Example Local Test Suite​

{
    "agents": [
        "examples/agents/a2a_agents_example.yaml",
        "examples/agents/multimodal_example.yaml",
        "examples/agents/orchestrator_example.yaml"
    ],
    "broker": {
        "SOLACE_BROKER_URL_VAR": "SOLACE_BROKER_URL",
        "SOLACE_BROKER_USERNAME_VAR": "SOLACE_BROKER_USERNAME",
        "SOLACE_BROKER_PASSWORD_VAR": "SOLACE_BROKER_PASSWORD",
        "SOLACE_BROKER_VPN_VAR": "SOLACE_BROKER_VPN"
    },
    "llm_models": [
        {
            "name": "gpt-4-1",
            "env": {
                "LLM_SERVICE_PLANNING_MODEL_NAME": "openai/azure-gpt-4-1",
                "LLM_SERVICE_ENDPOINT_VAR": "LLM_SERVICE_ENDPOINT",
                "LLM_SERVICE_API_KEY_VAR": "LLM_SERVICE_API_KEY"
            }
        }
    ],
    "results_dir_name": "sam-local-eval-test",
    "runs": 3,
    "workers": 4,
    "test_cases": [
        "tests/evaluation/test_cases/filter_csv_employees_by_age_and_country.test.json",
        "tests/evaluation/test_cases/hello_world.test.json"
    ],
    "evaluation_settings": {
        "tool_match": {
            "enabled": true
        },
        "response_match": {
            "enabled": true
        },
        "llm_evaluator": {
            "enabled": true,
            "env": {
                "LLM_SERVICE_PLANNING_MODEL_NAME": "openai/gemini-2.5-pro",
                "LLM_SERVICE_ENDPOINT_VAR": "LLM_SERVICE_ENDPOINT",
                "LLM_SERVICE_API_KEY_VAR": "LLM_SERVICE_API_KEY"
            }
        }
    }
}

Remote Evaluation​

In a remote evaluation, the evaluation framework sends requests to a remote Agent Mesh instance. This mode is useful for testing agents in a production-like environment where the agents are running on a separate server. The remote environment must have a REST gateway running to accept requests from the evaluation framework. You can also use an authentication token to communicate securely with the remote SAM instance.

Remote Test Suite Configuration​

For a remote evaluation, you must define the broker, remote, and test_cases fields.

The broker field is a required object with connection details for the Solace event broker.

"broker": {
    "SOLACE_BROKER_URL_VAR": "SOLACE_BROKER_URL",
    "SOLACE_BROKER_USERNAME_VAR": "SOLACE_BROKER_USERNAME",
    "SOLACE_BROKER_PASSWORD_VAR": "SOLACE_BROKER_PASSWORD",
    "SOLACE_BROKER_VPN_VAR": "SOLACE_BROKER_VPN"
}

The remote field is a required object containing the connection details for the remote Agent Mesh instance.

"remote": {
    "EVAL_REMOTE_URL_VAR": "EVAL_REMOTE_URL",
    "EVAL_AUTH_TOKEN_VAR": "EVAL_AUTH_TOKEN",
    "EVAL_NAMESPACE_VAR": "EVAL_NAMESPACE"
}

The test_cases field is a required list of paths to the test case JSON files. You must specify at least one test case.

"test_cases": [ "tests/evaluation/test_cases/hello_world.test.json" ]

You can also provide optional settings for results_dir_name, runs, and evaluation_settings.

The results_dir_name field is an optional string that specifies the name of the directory for evaluation results. It defaults to tests.

"results_dir_name": "my-remote-test-results"

The runs field is an optional integer that specifies the number of times to run each test case. It defaults to 1.

"runs": 5

The evaluation_settings field is an optional object that allows you to configure the evaluation.

"evaluation_settings": {
    "tool_match": {
        "enabled": true
    },
    "response_match": {
        "enabled": true
    }
}

Example Remote Test Suite​

{
    "broker": {
        "SOLACE_BROKER_URL_VAR": "SOLACE_BROKER_URL",
        "SOLACE_BROKER_USERNAME_VAR": "SOLACE_BROKER_USERNAME",
        "SOLACE_BROKER_PASSWORD_VAR": "SOLACE_BROKER_PASSWORD",
        "SOLACE_BROKER_VPN_VAR": "SOLACE_BROKER_VPN"
    },
    "remote": {
        "EVAL_REMOTE_URL_VAR": "EVAL_REMOTE_URL",
        "EVAL_AUTH_TOKEN_VAR": "EVAL_AUTH_TOKEN",
        "EVAL_NAMESPACE_VAR": "EVAL_NAMESPACE"
    },
    "results_dir_name": "sam-remote-eval-test",
    "runs": 1,
    "test_cases": [
        "tests/evaluation/test_cases/filter_csv_employees_by_age_and_country.test.json",
        "tests/evaluation/test_cases/hello_world.test.json"
    ],
    "evaluation_settings": {
        "tool_match": {
            "enabled": true
        },
        "response_match": {
            "enabled": true
        },
        "llm_evaluator": {
            "enabled": true,
            "env": {
                "LLM_SERVICE_PLANNING_MODEL_NAME": "openai/gemini-2.5-pro",
                "LLM_SERVICE_ENDPOINT_VAR": "LLM_SERVICE_ENDPOINT",
                "LLM_SERVICE_API_KEY_VAR": "LLM_SERVICE_API_KEY"
            }
        }
    }
}

Evaluation Settings​

The evaluation_settings block in the test suite JSON file allows you to configure how the evaluation is performed. Each enabled setting provides a score from 0 to 1, which contributes to the overall score for the test case.

tool_match​

The tool_match setting compares the tools the agent used with the expected_tools defined in the test case. This is a simple, direct comparison and does not use a language model for the evaluation. It is most effective when the agent's expected behavior is straightforward and there is a clear, correct sequence of tools to be used. In more complex scenarios where multiple paths could lead to a successful outcome, this method may not be the best way to evaluate the agent's performance.

response_match​

The response_match setting compares the agent's final response with the expected_response from the test case. This comparison is based on the ROUGE metric, which evaluates the similarity between two responses by comparing their sequence of words. This method does not use a language model for the evaluation and does not work well with synonyms, so it is most effective when the expected answer is consistent. For more information about the ROUGE metric, see the official documentation.

llm_evaluator​

The llm_evaluator setting uses a language model to evaluate the entire lifecycle of a request within the agent mesh. This includes the initial prompt, all tool calls, delegation between agents, artifact inputs and outputs, and the final message output. The evaluation is based on a criterion you provide in the test case, which defines what a successful outcome looks like. This is the most comprehensive evaluation method because it considers the full context of the request's execution.

Running Evaluations​

After you create your test cases and test suite, you can run the evaluation from the command line using the sam eval command.

Command​

sam eval <PATH> [OPTIONS]

The command takes the path to the evaluation test suite JSON file as a required argument.

Options​

  • -v, --verbose: Enable verbose output to see detailed logs during the evaluation run.
  • -h, --help: Show a help message with information about the command and its options.

Example​

sam eval tests/evaluation/local_example.json --verbose

Interpreting the Results​

After an evaluation run is complete, the framework stores the results in a directory. The path to this directory is results/ followed by the results_dir_name you specified in the test suite.

Results Directory​

The results directory has the following structure:

<results_dir_name>/
├── report.html
├── stats.json
└── <model_name>/
    ├── full_messages.json
    ├── results.json
    └── <test_case_id>/
        └── run_1/
            ├── messages.json
            ├── summary.json
            └── test_case_info.json
        └── run_2/
            ├── ...
  • report.html: An HTML report that provides a comprehensive overview of the evaluation results. It includes a summary of the test runs, a breakdown of the results for each test case, and detailed logs for each test run. This report is the primary tool for analyzing the results of an evaluation.
  • stats.json: A JSON file containing detailed statistics about the evaluation run, including scores for each evaluation metric.
  • <model_name>/: A directory for each language model tested (or a single remote directory for remote evaluations).
    • full_messages.json: A log of all messages exchanged during the evaluation for that model.
    • results.json: The raw evaluation results for each test case.
    • <test_case_id>/: A directory for each test case, containing a run_n subdirectory for each run of the test case. These directories contain detailed logs and artifacts for each run.

HTML Report​

The report.html file provides a comprehensive overview of the evaluation results. It includes a summary of the test runs, a breakdown of the results for each test case, and detailed logs for each test run. This report is the primary tool for analyzing the results of an evaluation. You can open this file in a web browser to view the report.