> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://help.sigmacomputing.com/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://help.sigmacomputing.com/_mcp/server.

# Build an evaluation suite for your agents

> Build an evaluation suite to systematically test Sigma agent behavior before production use, using the Sigma UI, the REST API, and AI_USAGE.

> **Note**
>
> The use of AI features is subject to the following [disclaimer](/docs/notice-for-enabling-ai-enabled-features-in-sigma).

> **Note**
>
> Usage of this feature uses consumption credits. For details about what usage is billed, see [About billable usage events](/docs/about-billable-usage-events).

To test whether an agent is behaving as expected, you can build an evaluation suite to test agent behavior systematically, rather than testing the agent ad hoc in chat.

For example, you might want to evaluate agent behavior:

* Before deploying an agent for production use cases, such as embedded in a host application.
* Before and after making changes to agent instructions or action tool guidance, to see if it has a positive or negative effect on agent behavior.
* Before and after updating the AI context available to a data model, to see if it has a positive or negative effect on analysis or source selection.

This page uses a **Traffic Overview Analyst** agent as a running example. That agent has access to a Google Analytics *EVENTS* table and one action tool, **Record important insights**, which inserts a row into a table when the agent is asked to log a finding.

## User and system requirements

To build and run an evaluation suite for a Sigma agent, you must have the following:

* The **Manage agents** and **Create, edit, and publish workbooks** permissions enabled on your account type
* **Can edit** access to the workbook that contains the agent
* Access to the [`AI_USAGE` table](/docs/ai-usage) for your organization, which records agent inputs and outputs

To run evaluations with the Sigma UI, you also need [write access](/docs/set-up-write-access) on the connection with the `AI_USAGE` table so that you can create input tables to hold your evaluation cases and scores.

To run evaluations with the Sigma REST API, you also need [API client credentials](/reference/generate-client-credentials).

## Define evaluation cases

Define specific evaluation cases for each agent that you want to evaluate, because different agents have different instructions, call different tools, and perform different tasks.

Useful evaluation cases for this agent might include both agent behavior and performance:

* **Did the tool get called?**: Prompt the agent to record a specific insight and confirm that the agent called the **Record important insights** tool.
* **Was the tool used correctly?**: Prompt the agent to record a specific insight and confirm that the agent called the **Record important insights** tool and that the information added to the input table matches the insight mentioned in the agent conversation.
* **Was the answer factually accurate and faithful to the data?**: Prompt the agent to respond to a factual question with a known answer in the data, such as the most common traffic referrer over a given period, then confirm the accuracy with a deterministic check against the known correct value in the data and validating that the database was queried directly to produce the response.
* **What is the reasoning quality of the agent?**: Prompt the agent to consider whether a given traffic pattern is anomalous and should be investigated, then assess the quality of the judgment.

For more on designing agent evaluations (also called evals), see the Anthropic guide on [Demystifying evals for AI agents](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents), the OpenAI guide on [Evaluate agent workflows](https://developers.openai.com/api/docs/guides/agent-evals), and the Langfuse guide on [AI agent evaluation](https://langfuse.com/resources/engineering/ai-agent-evaluation).

## Run your evaluation cases

Depending on your preferences, you can run evaluation cases entirely in the Sigma UI, or programmatically with the Sigma REST API.

> **Tip**
>
> Run evaluation cases repeatedly and regularly to establish consistency and reliability of the behavior in general and over time.

### Run agent evaluations in the Sigma UI

You can run your agent evaluations in the Sigma UI using [input tables](/docs/intro-to-input-tables) to define and record the evaluations.

Before you start, consider your document access and tagging strategy. Evaluations are tests, and your testing details shouldn't be visible to users or interfere with production agent use cases. Use this approach only when production users see a specific [tagged version](/docs/add-version-tags-to-workbooks-and-data-models) of the workbook and agent, and the latest published version is visible only to users performing evaluations.

To configure your evaluation set in a workbook, do the following:

1. In the workbook, add a page for this agent's evaluation details. Name the page **Evals** and make it visible only to yourself and any other users who run eval cases for this agent.
2. On the **Evals** page, [add an input table](/docs/create-new-input-tables) named **Eval cases**.

   > **Tip**
   >
   > Create the input table on the same connection used for the [`AI_USAGE` table](/docs/ai-usage), which records agent inputs and outputs, so that you can join the data in the scoring phase. If you take this approach, consider creating an `Evals` schema in the same database and adding it as a **Write destination** for the connection.
3. In the input table, add text columns for *Eval case*, *Prompt*, and *Expected output*. For example, for the **Traffic Overview Analyst** agent:

   | Eval case            | Prompt                                                                                   | Expected output                                                           |
   | :------------------- | :--------------------------------------------------------------------------------------- | :------------------------------------------------------------------------ |
   | 001-tool-call-check  | Record an important insight.                                                             | Calls **Record important insights**.                                      |
   | 002-tool-call-valid  | Record an insight for the % that organic traffic grew month-over-month.                  | Calls **Record important insights** and stores the % value.               |
   | 003-answer-valid     | What was the most common traffic referrer in the last 30 days?                           | Matches the known top referrer in the *EVENTS* table for the time period. |
   | 004-answer-reasoning | What was the most valuable paid ad campaign for free trial sign-ups in the last 90 days? | Matches the ad campaign with the most referrers to the free trial page.   |

When you run agent evaluations in the Sigma UI, how you run them depends on whether you want to run each case individually or run the whole set automatically:

#### [Run cases individually](#run-eval-cases-individually-in-the-sigma-ui)

Use a chat element, input table, and text area control with actions to run one case at a time and watch the agent respond.

#### [Run the full set on a schedule](#run-the-full-eval-set-on-a-schedule-in-the-sigma-ui)

Schedule an automated action sequence to call an agent and insert the response into an input table, so the whole set runs without human supervision.

#### Run eval cases individually in the Sigma UI

> **Info**
>
> To run agent evaluations in the Sigma UI using this process, you must modify the agent instructions to populate the eval cases.

1. Add a [chat element](/docs/chat-with-agent) to the **Evals** page and select the agent that you want to test. For example, **Traffic Overview Analyst**.

2. Add a [**Text area** control](/docs/create-and-manage-a-control-element) and set the control ID to `agent-input`.

3. Create a child table for the **Eval cases** input table and delete the *Expected output* column, then add a column named *Run test* to use as the button that runs each case.

4. To store a record of manual evaluation runs, create an input table on the same connection as the `AI_USAGE` table. Name the input table **Manual eval history** and add two columns, one **Text** column called *Eval case* and one **Date** column called *Timestamp*.

5. Add an [action sequence](/docs/create-and-manage-action-sequences) on the child table of the **Eval cases** input table:

   * Specify an **On select** trigger to the *Run test* column.
   * Add an action to [**Set control value**](/docs/create-actions-that-manage-control-values), then choose the `agent-input` control. For **Set value as**, select **Formula** and use a formula of `[Selection/Prompt]`.
   * Add an action to [**Clear chat element messages**](/docs/create-actions-that-interact-with-agents), then choose the chat element on your hidden **Evals** page.
   * Add an action to [**Insert row(s)**](/docs/create-actions-that-modify-input-table-data), then choose the **Manual eval history** input table. For the *Eval case* column, set the value with a **Column** and choose the **Eval case** column from the **Eval cases** table. For the *Timestamp* column, set the value with a **Formula** and set the formula to `Now()`.

6. Update the agent to use a **Greeting** and set the prompt for first message to reference the `agent-input` control value with text like the following:

   ```text
   Respond to the following prompt and include details about your actions or a table that reflects the data that you discovered. `=[agent-input]`
   ```

   ![The Greeting settings for the Traffic Overview Analyst agent, with the prompt for first message set to the agent-input control ID.](/_fern-img/bb0e2ef6b15b25cfa4f46a90784e3ebfe5e207aaada900167f3da43115ba3956.webp)

7. Publish the workbook, and optionally [tag](/docs/add-version-tags-to-workbooks-and-data-models) the latest published version with a **Testing** tag, to isolate test activity from the latest published version of the workbook.

8. Run the full set of evaluations through the agent, row-by-row:

9. For each row in the input table, select **Run test**.

10. Review the output.

![A workbook page with a chat element next to an Eval cases for agent table with a Run test button for each row, and a Manual eval history table logging each eval case with a timestamp.](/_fern-img/8b4ff9eb7d5bb5c5d739576032ce2aacebbd28081c5d055cf559f5b24c3b63ba.webp)

After you finish running the eval suite prompts through the chat element, [review and score the results](#review-and-score-the-results).

#### Run the full eval set on a schedule in the Sigma UI

This approach requires two actions per evaluation case, so the sequence grows with the number of evaluations. Use this approach for an eval suite of ten cases or fewer. For anything larger, [run your evaluations with the Sigma REST API](#run-agent-evaluations-with-the-sigma-rest-api) instead.

1. Add a *Status* column to the **Eval cases** input table.
2. Add an [automated action sequence](/docs/configure-action-sequences-to-run-automatically) to the workbook.
3. Add a schedule at which you want to run the evaluation suite.
4. Add two actions to the sequence for each evaluation case:

   1. Add a [**Call agent**](/docs/create-actions-that-interact-with-agents) action and set the prompt to the following formula:

      ```sigma wordWrap
      Switch(True, StartsWith([Eval cases/Eval case], "001"), [Eval cases/Prompt])
      ```

      Leave the agent response as the default action variable.
   2. Add an [**Update row**](/docs/create-actions-that-modify-input-table-data) action:

      1. Choose the **Eval cases** input table.
      2. For **Update rows by**, select **Column value matches criteria** and specify the *Eval case* column with criteria of **Text starts with** and `"001"`.
      3. For **Update with value(s)**, select **Status**, then select **Action variable** and choose the default `Agent-response` action variable from the **Call agent** action.
5. For each additional row in your **Eval cases** input table, duplicate the two actions, updating the prompt formula and update row criteria to reference the relevant eval case row.

   ![A Call agent action configured with a Switch formula that matches eval case 001 to its prompt, alongside an action sequence that alternates Call 'Traffic Overview Analyst' and Update row(s) in 'Eval cases' steps for each eval case.](/_fern-img/69b8780bf309e8c6964086c87214d86127377bf1d499f69ca304f7abc9af36c0.webp)
6. Publish the workbook. The action automatically runs on the latest published version of the workbook at the scheduled time.

> **Tip**
>
> This guidance uses the agent response in the *Status* column as an indication that the evaluation was run successfully. Use the agent output in the `AI_USAGE` table to score the output when assessing the performance of the agent.

After the eval suite finishes running, [review and score the results](#review-and-score-the-results).

### Run agent evaluations with the Sigma REST API

Running agent evaluations with the Sigma REST API is recommended if you want to run large evaluation suites with custom metadata for tracing and scoring purposes.

1. Define a set of evaluation cases following the guidance in [Define evaluation cases](#define-evaluation-cases).

   > **Tip**
   >
   > Store your eval cases in a dedicated file referenced by the script or build automation that runs the evaluation job, versioned alongside your other tests and code.
2. Write a script that [calls the agent with the API](/docs/call-agents-with-the-api) for each case, attaching `metadata` so you can identify the run later.

   For example, for a given agent in a workbook, use your API credentials to loop over a set of cases and tag each request with the eval suite version, time, and unique identifier for the eval case:

   ```python
   import requests

   # workbook_id and agent_id come from the List agents endpoint
   # access_token comes from your API client credentials
   eval_cases = [
       {"id": "001-tool-call-check", "prompt": "Record an important insight."},
       {"id": "002-tool-call-valid", "prompt": "Record an insight for the % that organic traffic grew month-over-month."},
       {"id": "003-answer-valid", "prompt":"What was the most common traffic referrer in the last 30 days?"},
       {"id": "004-answer-reasoning", "prompt":"What was the most valuable paid ad campaign for free trial sign-ups in the last 90 days?" }
   ]

   # use the relevant API base URL for your Sigma organization
   for case in eval_cases:
       requests.post(
           f"https://api.sigmacomputing.com/v2/workbooks/{workbook_id}/agents/{agent_id}",
           headers={"Authorization": f"Bearer {access_token}"},
           json={
               "messages": [{"role": "user", "content": case["prompt"]}],
               "metadata": {
                   "evalRun": True,
                   "evalSuiteVersion": "2026-09-18",
                   "evalCase": case["id"],
               },
           },
       )
   ```

   > **Tip**
   >
   > When you run iterative evaluation suites, consider the following:
   >
   > * Resolve the agent for a tagged version of the workbook by passing the `versionTagName` query parameter to the [List agents](/reference/list-workbook-agents) endpoint, then record that tag name in your own `metadata` so each set of results is associated with a specific version of the workbook (and agent instructions).
   > * Increment the `evalSuiteVersion` after adding new eval cases or improving existing questions, to make sure you compare results from the same versions of the same eval suite.

After the eval suite finishes running, [review and score the results](#review-and-score-the-results).

> **Info**
>
> If you perform evaluations programmatically, you can't test agent behavior with an approval prompt. An approval prompt is shown only when chatting with an agent in a chat element; otherwise, action approval is skipped.

## Review and score the results

After you run your evaluation cases, use the `AI_USAGE` table to review each agent response and score it against the expected output.

* [Prepare to score the results with AI_USAGE](#prepare-to-score-the-results-with-ai_usage)
* [Score the results with a relevant method](#score-the-results-with-a-relevant-method)

### Prepare to score the results with AI_USAGE

To set up reviewing and scoring in Sigma, prepare the `AI_USAGE` table to review the responses for one agent:

1. In the connection browser, locate the [`AI_USAGE` table](/docs/ai-usage#ai-usage-table-in-connection-browser) and click **Explore** to open the table in an exploration. Save the exploration as a workbook, naming it something like **Agent evaluation scoring**.

   > **Tip**
   >
   > If you ran your agent evaluations in the Sigma UI, add the AI_USAGE table to the **Evals** page of the workbook that contains your agent, instead of a new workbook, so that you can evaluate the response quality alongside the eval cases directly.
2. In the workbook, shape the table for analysis of the agent responses. Group the table by *Timestamp* to sort conversations by time. Truncate the date to **Hour**.
3. Add another grouping by *Trace Id* to associate conversation inputs and outputs together.
4. In the grouping, add a calculated column with the formula `Max([Agent Id])` and optionally another with the formula `Max([Agent Name])` to identify the agent associated with the conversation.
5. Filter the table to your agent using the *Max of Agent Id* column.
6. Extract relevant metadata columns for your analysis. For the *Metadata* column, open the column menu and select **Extract columns...**. Depending on how you performed your evaluations, select the following relevant columns:

   * For all eval suite methods, extract the `upstreamCaller` key as a column. Sigma populates this key automatically, and it identifies whether the agent was called from a chat element, the API, an automated action, or the home page.
   * If you ran your eval suite programmatically, extract the *`evalRun`*, *`evalSuiteVersion`* and *`evalCase`* metadata keys that you included in the script as columns.
   * If you ran your eval suite in the Sigma UI, extract the *`score`*, *`score_name`*, and *`score_comment`* keys as columns. These keys hold the rating that a user gives a conversation in the chat element, so they're populated only for cases that someone rated.
7. If you plan to score evaluation suites run with different methods at the same time, add the *upstreamCaller* column to the *Trace Id* grouping.
8. Make the agent output easier to review. Add a new column called *Output text* and use the formula `Coalesce(Text([Output].text), Text([Output].output), Text([Output]))` to extract the `text` key and the `output` key for different types of agent responses from the *Output* column.
9. (Optional) Hide the *Max of Agent Id* and *Max of Agent Name* columns because all rows are for the same agent.
10. In the base table, move the following columns to the top of the column shelf to make performing the scoring easier: *Timestamp*, *Trace Name*, *Input*, and *Output text*. Hide other columns, but keep *Span Id* available, because you need it as the unique key if you score with a human grader.

    > **Tip**
    >
    > If you ran your eval suite programmatically, also move the *evalCase*, *evalSuiteVersion*, and *evalRun* columns.

After making the updates to the table, you end up with a table like the following:

![The AI\_USAGE table grouped by Hour of Timestamp and Trace Id, with Trace Name, Input, and Output text moved to the front of the column shelf and the extracted upstream caller, eval case, and eval suite version metadata columns visible.](/_fern-img/41e7ede9c60e29e0f7652b1295a3154c600273ac0dbb07f0228b72c0ada53938.webp)

### Score the results with a relevant method

You can use different methods to score agent accuracy and performance against your evaluation suite. Depending on what you're evaluating and how large of an evaluation suite you're reviewing, choose the relevant scoring approach:

#### [Human grader](#score-responses-with-a-human-grader)

A person reviews the output against the expected answer. Most reliable for nuanced judgment calls, such as reasoning quality, but doesn't scale to large eval suites or frequent runs.

#### [LLM-as-a-judge](#score-responses-with-llm-as-a-judge)

Another agent compares the output of the agent you're evaluating with your desired output or behavior and returns a verdict. A non-deterministic approach, can be useful to review instruction-following checks at scale.

#### [AI column as classifier](#score-responses-using-a-classifier-model-in-an-ai-column)

Use an AI column to non-deterministically sort output into fixed categories, such as pass/fail or an issue type, rather than a free-form judgment. Useful when you want consistent buckets to consider for agent performance.

#### [A combination](#use-a-combination)

Use an LLM-as-a-judge approach, but route low-confidence or flagged cases to a human grader.

#### Score responses with a human grader

To score responses with a human grader, use an input table:

1. On the grouped `AI_USAGE` table in your workbook, create a child [linked input table](/docs/use-linked-columns-in-workbooks). Use the *Span Id* column as the unique key, and include the relevant columns from the `AI_USAGE` table.
2. Rename the linked input table **Manual review cases**.
3. Sort the table by *Trace Id* to keep related messages together.
4. Rename the auto-created *Text* column to *Score* and add a second **Text** column named *Rationale*.
5. Open the column menu for the *Input* column and select **Transform** > **Convert to Text**.
6. Add the expected outputs from the **Eval cases** table to the **Manual review cases** table:

   1. Add a column to the review table via lookup.
   2. Select the **Eval cases** input table. It must use the same connection as the `AI_USAGE` table to perform the lookup.
   3. For **Columns to add**, select **Expected output**. If you ran your eval suite in the Sigma UI, also select **Prompt** and **Eval case**.
   4. For **Map two elements**, choose the matching columns depending on how you performed your evaluations:

      * If you ran your eval suite programmatically, select **evalCase** from the `AI_USAGE` table and **Eval case** from the `Eval cases` table.
      * If you ran your eval suite with a scheduled action sequence, select **Input** from the `AI_USAGE` table and **Prompt** from the `Eval cases` table.
      * If you ran your eval cases individually with the chat element, first do a lookup with the **Manual eval history** table, matching on *Timestamp*, then match the *Eval case* column from that lookup with the *Eval case* column from the `Eval cases` table.
7. Review the *Input*, *Output text*, and *Expected output* columns, note your judgment in the *Score* column, and provide your rationale in the *Rationale* column. For example, `Pass`, `Fail`, or `Partial Fail`, with an explanation such as `Tool call was performed but insight was not accurate.`

![The Manual review cases - API linked input table with columns for Input, Output text, and the looked-up Expected output, and empty Score and Rationale columns ready for a reviewer to fill in.](/_fern-img/9dffd3511215d7e289f91f79e94b647f51193528b9d1dc5e0e4c2daf6fc0dc60.webp)

#### Score responses with LLM-as-a-judge

To ask ad hoc questions about your eval results, such as which cases regressed after your last instructions change, or to automate using a large language model (LLM) as a judge, add a Sigma agent to the **Agent evaluation scoring** workbook and add the `AI_USAGE` table and your eval cases table (uploaded as a CSV if you used a local file programmatically, or the input table if you used a UI-based approach) as data sources for the agent.

When you use an LLM-as-a-judge approach, it's important to consider the following:

* An LLM is non-deterministic and relies on the given context and capabilities of the chosen model to review the quality of the output. If you want to perform a deterministic review of evaluation performance, consider writing a script instead.
* The prompt for the agent used to perform LLM-as-a-judge reviews is important. For example, instruct the agent to act as a subject matter expert.
* The quality of the LLM-as-a-judge agent scores also needs to be evaluated and calibrated on scores that you are confident in or results that you know the answer to, also referred to as a ground truth or gold test suite.

For more details on setting up an LLM-as-a-judge, see [LLM-as-a-judge: a complete guide to using LLMs for evaluations](https://www.evidentlyai.com/llm-guide/llm-as-a-judge) from Evidently AI.

![The LLM-as-a-judge eval review table listing a Score and Rationale for each eval case, next to a chat conversation with the LLM as a judge agent reporting a score of 9 out of 10 for eval case 004-answer-reasoning.](/_fern-img/c6de8fe7857e8ea4baa4b90b6f2c6d3da749eafe404439fbe05c98baa5734876.webp)

To set up a Sigma agent as an LLM-as-a-judge to review the output of your evaluations:

1. Create an input table titled **LLM-as-a-judge eval review** to store the score and rationale from the LLM-as-a-judge. Add the following columns:

   * Text column named *Eval case*
   * Text column named *Trace Id*
   * Text column named *Score*
   * Text column named *Rationale*
2. Create a Sigma agent to review the `AI_USAGE` table and eval cases and update the input table with the score and rationale for each reviewed case and trace ID. For example:

   * Name the agent **LLM as a judge**.
   * Add the **Eval cases** table and the **AI_USAGE** table filtered to the relevant eval cases to review.
   * Use **agent assistant** to write instructions and an action tool, or use the following suggestions:

     ```text
     * When a user asks you to judge a specific eval case, compare the output text in AI_USAGE with the expected output for the eval case in Eval cases.
     * Provide a concise score and rationale and record it with the Record eval score action tool.
     * Base every judgment only on the retrieved case data; do not invent missing context.
     * If multiple cases match, ask the user to choose or summarize each separately. Use the trace Id to assemble a full agent session.
     ```

     In addition to functional guidance, provide the agent with scoring guidance in the instructions:

     ```text
     Scoring Guidance:

     * Add a score as an accuracy judgment of how well the output answers the input prompt.
     * Explain the main reasons for the score, including omissions, inaccuracies, or strong alignment.
     * If an existing Score is present in the data, you may reference it, but still provide your own rationale based on the input and output.
     ```

     Add an action tool to store reviews in the **LLM-as-a-judge eval review** table:

     1. For **Tools**, click **+** (**Add tool**), then select **Action**.
     2. For **Name**, enter **Record eval score**.
     3. For **Steps**, click **Add step** (**+**), then for **Action**, select **Insert row(s)**.
     4. For **Insert type**, select **New data**.
     5. For **Into**, select the **LLM-as-a-judge eval review** input table.
     6. For **Set column values**, select **Agent input** for each column and provide guidance for the agent:

        * For the *Eval case* column, enter **Eval case number**.
        * For the *Trace Id* column, enter **Trace Id for the reviewed session**.
        * For the *Score* column, enter **Score for the eval performance**.
        * For the *Rationale* column, enter **Rationale for the score**.

     ![The Record eval score action tool with an Insert row(s) step into the LLM-as-a-judge eval review table, showing the Eval case, Trace Id, Score, and Rationale columns each set to Agent input with guidance text.](/_fern-img/05f8dbbfc8c7bef2323b8143f733546dfc6e3989ddec43a423c2f69257917ba2.webp)
3. Either add a chat element for the **LLM as a judge**, or set up an automated action to call the agent on a schedule.

#### Score responses using a classifier model in an AI column

To score responses without a person reading each one, [add an AI column](/docs/create-ai-columns) that judges each agent response against the expected output for its evaluation case.

1. On the `AI_USAGE` table that you prepared, create a child table and name it **AI column review cases**.
2. Add the expected outputs from the **Eval cases** table to the **AI column review cases** table:

   1. Add a column to the review table via lookup.
   2. Select the **Eval cases** input table. It must use the same connection as the `AI_USAGE` table to perform the lookup.
   3. For **Columns to add**, select **Expected output**. If you ran your eval suite in the Sigma UI, also select **Prompt** and **Eval case**.
   4. For **Map two elements**, choose the matching columns depending on how you performed your evaluations:

      * If you ran your eval suite programmatically, select **evalCase** from the `AI_USAGE` table and **Eval case** from the `Eval cases` table.
      * If you ran your eval suite with a scheduled action sequence, select **Input** from the `AI_USAGE` table and **Prompt** from the `Eval cases` table.
      * If you ran your eval cases individually with the chat element, first do a lookup with the **Manual eval history** table, matching on *Timestamp*, then match the *Eval case* column from that lookup with the *Eval case* column from the `Eval cases` table.
3. Follow the steps to [create an AI column](/docs/create-ai-columns):

   * Name the column **LLM classifier score**.
   * Add a **Prompt**. For example, `For each _evalCase_ in a session (identified by _Trace Id_), evaluate the _Output text_ compared to the input prompt in _Input_ for correctness.`
   * Select the **Don't run automatically** checkbox to run the scoring classifier only when requested.
   * (Optional) Select **JSON format** and turn on the **Format as JSON** toggle to get structured output from the classifier model, such as a `Passed` logical value, a `Score` number, and a `Rationale` text field alongside the raw output from the model.

   ![The Add AI column dialog for LLM classifier Score, with Format as JSON enabled and Passed, Score, and Rationale fields defined, and Don't run automatically selected.](/_fern-img/7ab8719e817582728c14fe231ea78ce4c53cf91d16ea87b0eadf228d393c0fed.webp)
4. After you configure the AI column, click **Create**, then **Confirm**.
5. To run the AI column, right-click a cell and choose **Run AI column**.

![The LLM-as-a-judge review table with the LLM classifier score column running for each row, showing completed JSON output for one row and loading placeholders for the others.](/_fern-img/d13638848d74c86ad9c174ac360f0bf1e4aa4d92abd4b0570cf6a4cfeaf11fc8.webp)

#### Use a combination

You can also combine the LLM-as-a-judge, AI column classification, and human grader approaches so that automated scoring assesses high-volume straightforward cases and a person intervenes to handle more challenging cases and to periodically review the quality of the LLM-as-a-judge output.

For example:

* Escalate flagged cases from LLM-as-a-judge to a human grader. Add an action tool that lets the LLM-as-a-judge agent insert flagged rows into a **Manual review cases** table.
* Escalate cases from the AI column classifier to a second opinion. Add an action to the **AI column review cases** table that sends selected values from a row to either a **Manual review cases** table for a person to grade, or to a table that the LLM-as-a-judge agent can read.

  > **Tip**
  >
  > When you send a case for a second opinion, don't give the LLM-as-a-judge agent the AI column's verdict. Have a human reviewer compare the two independent scores instead.

## Review individual conversations

Building an eval suite helps you test known cases systematically, but you can also review real conversations with an agent directly in the Administration portal.

If you're an admin in Sigma, open the agent details page for an agent and review the **Execution log** tab to review conversations, including tokens, latency, and execution path for each. For a given conversation, select **View details** to review the number of turns, tool calls, and each turn's input and output.

You can use the execution log to spot-check flagged runs, follow up on a low conversation rating, or trace exactly what actions an agent performed in a conversation, turn by turn.

For more details, see [Manage Sigma agents for your organization](/docs/manage-agents-for-your-organization).

## Related resources

* [About Sigma agents](/docs/sigma-agents)
* [Build Sigma agents](/docs/build-agents)
* [Call Sigma agents with the API](/docs/call-agents-with-the-api)
* [AI usage dashboard](/docs/ai-usage)
* [Manage Sigma agents for your organization](/docs/manage-agents-for-your-organization)
* [Example use cases for Sigma agents](/docs/example-agent-implementations)
* [Create AI columns](/docs/create-ai-columns)
* [Create actions that interact with agents](/docs/create-actions-that-interact-with-agents)