​
Video Insights Engine Inputs

Video Insights Configuration

The Video Insights Engine Configuration has three parameters that take in values:
  • instruction: optional. A string used to prompt the Agent during job execution.
  • video: required. The video to extract from.
  • output_schema: optional. Defines the exact structure of the JSON output that the extracted data will populate. Follows the standard JSON schema specification.
See Template Strings for dynamic parameter configuration.

​
Video Insights Output

The output will always be a JSON value of the structure specified in the output_schema (if you defined it).

​
Video Insights Example

Let’s run through an example using this engine together.
1

Create an Agent

Click on the “Add Agent” button in the top right corner of the Agents page.
Enter a name and an optional description of your Agent.
2

Select the Video Insights Engine

3

Configure the engine as follows

$ starts a template string
  • instruction: $instruction
  • video: $video
  • output_schema: Copy and paste the JSON schema below (hit Use Text). 
{
  "type": "object",
  "properties": {
    "visual_elements": {
      "type": "array",
      "description": "The visual elements appeared in this video, e.g superhero, fairytale, family etc. Give at least 5.",
      "items": {
        "type": "string",
        "description": "Visual elements appeared in this video"
      }
    },
    "music_style": {
      "type": "string",
      "description": "The styles of the music e.g JAZZ, POP etc. "
    },
    "scenes": {
      "type": "array",
      "items": {
        "type": "string",
        "description": "A description of one scene"
      },
      "description": "A list of scenes in the video"
    },
    "summary": {
      "type": "string",
      "description": "detailed description of the video"
    }
  },
  "description": "Video Insights result"
}
You can click Use Widget to then view the JSON schema in the UI.
4

Create the Agent

Hit the Create button. Now, let’s run it on a video through the UI.
5

View the Agent you just created

6

Create a new Agent job

7

Fill in the Agent inputs

Use this for the instruction:
Escape or do not include quotes in string fields of the output json
Sometimes, you need to experiment with the output_schema configuration and the prompts you pass in to the instruction (like in our case) to get the results you want.
Download and use this video in the video input field:Here are the filled-in Agent inputs:
8

Run the job

Hit the Create button at the bottom to start the Video Insights job.
9

View the Results

Click View of the respective job to view its status and results.
Scroll down the Agent Job Details page and you’ll see the job outputs.
Notice that the JSON output will be in the structure that you defined in the output_schema. In our case, we defined our output to be a JSON Object with certain properties to be filled in by the Agent.