​
Image Insights Engine Inputs

Image Insights Configuration

The Image Insights Engine Configuration has four parameters that take in values:

  • instruction: optional. A string used to prompt the Agent during job execution.
  • image: required. The image to extract from.
  • model: required. The model to use for extraction.
  • output_schema: optional. Defines the exact structure of the JSON output that the extracted data will populate. Follows the standard JSON schema specification.

See Template Strings for dynamic parameter configuration.

​
Image Insights Output

The output will always be a JSON value of the structure specified in the output_schema (if you defined it).

​
Image Insights Example

Let’s run through an example using this engine together.

1

Create an Agent

Click on the “Add Agent” button in the top right corner of the Agents page.

Enter a name and an optional description of your Agent.

2

Select the Image Insights Engine

3

Configure the engine as follows

$ starts a template string

  • instruction: $instruction

  • image: $image

  • model: $model

  • output_schema: Copy and paste the JSON schema below (hit Use Text).

{
  "type": "array",
  "description": "An array of line items",
  "items": {
    "type": "object",
    "properties": {
      "item_name": {
        "type": "string",
        "description": "Name of the item"
      },
      "unit_price": {
        "type": "number",
        "description": "Price per unit item"
      },
      "final_price": {
        "type": "number",
        "description": "Final total price of the line item"
      }
    },
    "description": "Invoice line item"
  }
}

You can click Use Widget to then view the JSON schema in the UI.

4

Create the Agent

Hit the Create button. Now, let’s run it on an image through the UI.

5

View the Agent you just created

6

Create a new Agent job

7

Fill in the Agent inputs

Leave instruction empty.

image: Download and use this image

model: gpt-4o

Sometimes, you need to experiment with the output_schema configuration and the prompts you pass in to the instruction to get the results you want.

Here are the filled-in Agent inputs:

8

Run the job

Hit the Create button at the bottom to start the image Insights job.

9

View the Results

Click View of the respective job to view its status and results.

Scroll down the Agent Job Details page and you’ll see the job outputs.

Notice that the JSON output will be in the structure that you defined in the output_schema. In our case, we defined our output to be a JSON Object with certain properties to be filled in by the Agent.