Build a VLM Flow

A pipeline that combines Object Detection and VLM with a logic layer. It first gates candidates with cheaper signals, before passing to the VLM, cutting token usage while improving precision.

What is a VLM Flow?

A VLM Flow is a composable detection pipeline that allows you to together Object Detection, Time Gaps, VLM, and a logic layer. The idea is to use more lightweight signals first (like classic detectors), and only send confirmed signal cases to the more heavy VLM. This way you save VLM tokens and still get higher-quality alerts.

The VLM flow utilizes the following building blocks:

• Object Detection (blue): Allows for real time, frame-by-frame detections.

• Time gap (green): A buffer window (e.g., 10s) to wait before firing the VLM detection.

• VLM (brown): Semantic reasoning over a single or multiple key frames.

• Logic (gray): Involves object detection logic that can be explored in custom detection events such as object and zone overlap, count thresholds, and more. This can also include logic resulted from the VLM keys, such as boolean results, Numbers, and a triggered selection out of a number of choices. All the logic options simply define how events should work.

• Event Trigger: fires when all the previous logic conditions are met.

Detection Combinations & Example Use Cases

1. Object Detection → logic → Event

Simple, detector-only setup

• What it does Pure Object Detection. As soon as the detector finds something and passes the logic rules, it triggers an event.

• Behavior

  • Very fast: typically within ~1 second.

  • Low cost: no VLM usage.

  • Good when you just need “something entered / exited this area” and don’t care about deeper context.

Example

• Detect when a person enters a restricted zone.

• Count vehicles crossing a line.

2. Object Detection → logic → VLM → logic → Event

Detector + VLM for smarter, cleaner alerts

• What it does Object Detection acts as a gate. Only when the detector sees something suspicious do we send frames to the VLM for semantic confirmation.

• Behavior

  • Big token savings: VLM only runs after the detector says “this might matter”.

  • Higher quality real-time alerts: fewer false positives from simple bounding boxes.

  • Ideal when you already have a good detector, but need semantic understanding to clean up the results.

Example

• Fire detection that understands “how serious” it is

  • Step 1: Object Detection spots “fire/flame”.

  • Step 2: VLM checks if it’s a real hazard (e.g., a big fire) or something harmless (like a candle or stovetop).

• Reduce false alarms in “sensitive” alerts by letting VLM double-check the situation.

3. VLM → logic → Event

VLM-only, semantics-first detection

• What it does No classic detector involved. VLM periodically looks at frames to understand the overall situation, not just single objects.

• Behavior

  • Usually runs on a fixed interval (every few minutes), not every frame.

  • Slower and heavier than simple detectors, but much richer in meaning.

  • Good when you care about “what’s going on here overall?” instead of just “is there a person or car?”.

Example

• Check risk level or disaster signs:

  • Is there flooding?

  • Does the scene look like a traffic jam or accident?

• Describe the current scene.

4. VLM → logic → Time Gap → VLM → logic → Event

Two-stage VLM: coarse check → fine check / double confirm

(Visually it’s VLM → Time Gap → VLM, with logic before/after.)

• What it does

  1. First VLM pass (coarse) Run a cheaper, simpler check: “Is there any potential danger here?”

  2. Time Gap If yes, collect more frames over a short time window to get more context.

  3. Second VLM pass (fine / detailed) Run a more detailed prompt to answer deeper questions:

     • How big is the affected area?

     • What kind of damage is happening?

     • Are emergency workers already present?

     • Do we need to escalate the alert level?

• Behavior

  • Most expensive pipeline in terms of compute and tokens.

  • Best used for few but critical events where you want very high confidence and rich information.

  • Great when you want “coarse filter → deep analysis” instead of running expensive VLM on everything.

Example

• High-value safety or incident monitoring:

  • First stage: detect “possible hazard”.

  • Second stage: only for those cases, analyze impact, severity, people involved, etc.

• Double confirm:

  • Use multiple frames / time points to reduce wrong decisions from a single unlucky frame.

Creating a VLM Flow

To create a VLM Flow, go to Vision Language Model on the navigator and select VLM Flow Template. Here, you'll be able to see all the VLM Flows that your team has made in the past.

Afterwards, click + VLM Flow Template to open the editor and create a new flow. In this canvas, drag and drop the components you need. Outlined below are all the components or nodes utilized in this editor.

1. Entry Node

The flow always starts with an Entry Node, which can either be based on Object Detection or VLM. This entry node is the first logic check for your flow. Click on the three dots to choose between Object Detection or VLM as your entry logic.

Having Object Detection as the entry node offers additional choices to overlay warning bounding boxes and event zones for the VLM inference. This is useful for highlighting specific focus areas when there are multiple objects in the camera. For example, when looking for cases of loitering, the bounding box can help differentiate someone loitering vs someone passing by. The color of this overlayed outline can also be specified.

Once satisfied with the selection, click Confirm. This returns you to the editor. You'll notice a slight difference in the appearance of a VLM entry node vs an OD entry node, as the former displays non-string keys in the VLM template used. Each of these keys must connect to an eventual event trigger.

Each key can also be clicked into to toggle its condition.

2. Timer (Optional)

Between the Entry node and the next Sub VLM, an optional timer can be added. Timers are useful when the key frames need to be after the initial finding. For example, a trespassing detection event can run the vlm 5 seconds after the initial OD trigger of a person, ensuring that the initial trigger wasn't a false positive. Click on the three dots in the timer node to adjust the duration of the timer before triggering the next node.

3. Sub VLM

The Sub VLM is the subsequent VLM to be ran right before the Entry Node (or the timer, if you decide to include one). Add this node and connect it to the corresponding upstream node, and click the three dots to modify the VLM template you wish to run.

Clicking the three dots on the Sub VLM node lets you choose the VLM Template to run, as well as the trigger condition. By default, this is set to always trigger. This can be modified to only trigger when a specific key is true/false, above a certain numerical threshold, or a selection out of multiple options. This condition is valuable in its use of the VLM as an additional gate to reduce false positives that the previous entry node may have captured. Multiple trigger conditions can also be added to make the rules stricter. Once satisfied, click Confirm.

4. Event Trigger

The final component of the flow is the Event Trigger. Once the event passes all the previous nodes, the event trigger node outlines the name of the event that appears, as well as a corresponding description. After filling this in, click Confirm and name the VLM Flow Template. Once all the details are confirmed, click ✓ Create on the upper right hand side of the editor to create your flow.

Reminder:

Before you build a Flow, make sure you already created at least one VLM Template. The VLM node in the Flow will reference these templates.

Build a VLM Template

How to Use a VLM Flow in a Task

1. Open the Individual Task page for the task you want to configure.

2. Go to the Event Type Settings.

3. Click + Detection Event to add a new event rule.

4. In the VLM Detection section, choose one of the existing VLM Flows you created.

5. Save and adjust the event. The task will now use that VLM Flow to decide when to trigger events.

VLM Detection Result

After detecting a predefined event, the system automatically runs the VLM for further analysis.

Integrated Event Results:

The VLM analysis results are appended to the detected event for enriched reporting.

• Example Results:

  • Weather: Sunny, partly cloudy.

  • Alert Level: 2.

  • Vehicle Types: Firetruck, car.

  • Scene Description: Downtown street with a firetruck stopped.

Example Use Case:

If a fire is detected, the VLM analyzes the scene to identify emergency personnel, vehicles, or abnormal conditions.

Summary

VLM Flow gives you a flexible way to mix fast detectors with smart VLM reasoning, so you only spend tokens when it really matters. You can start simple with detector-only Flows, then gradually add VLM, Time Gaps, or even two-stage VLM when you need deeper understanding or fewer false alarms.

In practice, the setup is straightforward:

• Build your VLM Templates

• Design a VLM Flow by wiring together Object Detection, Time Gap, VLM, and logic

• Attach that Flow to a task’s Event settings.

Once that’s done, your tasks can move from basic “did something enter this zone?” alerts to rich, context-aware event detection that understands what’s actually happening in the scene.