Interpreting Run Results

View notebook on GitHub

Introduction

The output of an ARKIT flow is a RunResult object. This guide will help you to work with the run results effectively.

A run result stores the output of an ARTKIT flow as a list of dictionaries, each representing distinct path through the flow. The dictionaries are nested, with the outer keys corresponding to the names of steps, and the values corresponding to attribute-value mappings produced by each step.

The easiest way to get comfortable working with RunResult objects is to roll up your sleeves and work with an example.

Working with run results

The data stored within a RunResult object can be accessed in the following ways:

• As an iterable of dictionaries, where each dictionary represents the output of the flow for a single input.

• By index, where the index corresponds to the path through the flow, and the value is a list of outputs along that path for each input.

• By transforming to a pandas DataFrame

Let’s create a simple workflow that returns a RunResult:

import pandas as pd
pd.set_option("display.max_colwidth", None)

import artkit.api as ak

# Define an increment function
def increment(a: int, by: int):
    return dict(by=by, a=a + by)

# Construct and run the flow
input = [dict(a=1, b=2), dict(a=2, b=1)]
steps = ak.parallel(
    ak.step("increment", increment, by=1),
    ak.step("increment", increment, by=2)
)
result = ak.run(input=input, steps=steps)

# Display the RunResult
result

RunResult(
    [
        {'input': {'a': 1, 'b': 2}, 'increment': {'by': 1, 'a': 2}},
        {'input': {'a': 2, 'b': 1}, 'increment': {'by': 1, 'a': 3}}
    ],
    [
        {'input': {'a': 1, 'b': 2}, 'increment': {'by': 2, 'a': 3}},
        {'input': {'a': 2, 'b': 1}, 'increment': {'by': 2, 'a': 4}}
    ]
)

Since our flow consisted of two inputs with two paths, the RunResult has four entries. We can iterate overall all results by calling get_outputs, or iterate over the results of a path by calling get_outputs_per_paths:

for output in result.get_outputs():
    print(output)

{'input': {'a': 1, 'b': 2}, 'increment': {'by': 1, 'a': 2}}
{'input': {'a': 2, 'b': 1}, 'increment': {'by': 1, 'a': 3}}
{'input': {'a': 1, 'b': 2}, 'increment': {'by': 2, 'a': 3}}
{'input': {'a': 2, 'b': 1}, 'increment': {'by': 2, 'a': 4}}

for path_output in result.get_outputs_per_path():
    print([output for output in path_output])

[{'input': {'a': 1, 'b': 2}, 'increment': {'by': 1, 'a': 2}}, {'input': {'a': 2, 'b': 1}, 'increment': {'by': 1, 'a': 3}}]
[{'input': {'a': 1, 'b': 2}, 'increment': {'by': 2, 'a': 3}}, {'input': {'a': 2, 'b': 1}, 'increment': {'by': 2, 'a': 4}}]

Finally, we can convert the results into a pandas DataFrame by calling to_frame. The DataFrame index corresponds to the input index, and the columns correspond to the step names and outputs:

result_df = result.to_frame()
result_df

	input		increment
	a	b	by	a
item
0	1	2	1	2
1	2	1	1	3
0	1	2	2	3
1	2	1	2	4

Note that this dataframe contains a 2-level column, with the first column level corresponding to the step name, and the second level corresponding to the outputs of each step.

Working with multi-level columns is a bit different from the usual single-level column dataframe, but Google or your preferred Gen AI chatbot can help you figure out how to perform your favorite pandas operations with two-level columns.

If you don’t like the two level columns, you can drop the top level with df.droplevel(0, axis=1). Alternatively, if you prefer to concatenate the first and second level column names into a single-level column, you can do something like this:

result_df.columns = ['_'.join(col) for col in result_df.columns]
result_df

	input_a	input_b	increment_by	increment_a
item
0	1	2	1	2
1	2	1	1	3
0	1	2	2	3
1	2	1	2	4

Concluding Remarks

Working with ARTKIT run results takes some practice, but we have found that the RunResult object is an incredibly useful way to structure the outputs of ARTKIT pipelines. We converged on this design after considerable practical experience.

The Examples section of the ARTKIT documentation contains many useful illustrations of how to work with run results. We particularly recommend checking out Single-Turn Attacks: Augmenting Beavertails, which is full of useful patterns and code snippets.