System Management

VisionAI Observ can help you manage current machine resources and understand their utilization, effectively leveraging existing resources to maximize computation.

Reviewing Machine Resource Usage

Enter the System Management tab to view the current status of computing machine resources. This section of our guide mainly focuses on viewing existing compute. For adding new compute targets, please refer to this tab for adding a new inference node under the Deployment section.

Opening the tab defaults to showing the user the compute node. A compute node is a single machine equipped with processors (CPUs/GPUs) and memory, responsible for running computational tasks.

Each node is categorized using the following:

• Type: The machine type, which currently supports general x86 servers, cloud servers (Azure, AWS, GCP, etc.), and Orin NX type AIoT computing resources.

• Node Name: The machine name as inputted by the user.

• Cluster: The cluster to which the machine belongs. Currently, the Kubernetes system architecture automatically allocates the computational resources needed for tasks within a cluster.

• IP: The machine's IP address.

• Status: The current connection status of the machine.

• CPU / Memory / Disk: Monitors usage levels. If utilization exceeds 85%, exercise caution, as this may lead to system instability or performance degradation.

• Deployed Model: Lists the models currently running on this machine. Use the "Config" button to switch models; please ensure the new model is compatible with your hardware specifications.

Toggling to the icon to the right opens up the compute clusters. A cluster is the collective group of these interconnected compute nodes, working together as a single, powerful system to handle various workloads.

Here, each cluster is categorized using the following:

• Type: Identical to the requirements in the node.

• Cluster Name: The cluster name as inputted by the user.

• ITS (Integrated Traffic System): The current operational or connectivity status of the entire cluster. Read more on Smart Traffic Systems here.

  • Signifies that all systems needed to run ITS have been installed. This is necessary, as ITS runs on a different streaming pipeline.

  • Signifies that the systems needed to run ITS have not been installed.

• Cluster CPU: The aggregate CPU utilization across all nodes within the cluster.

• Cluster Memory: The collective memory usage across all nodes within the cluster.

Server Machine Configuration

Clicking Config under each compute node allows you to choose the AI models to run on your server machines (either on-premise x86 servers or cloud machines). The setting displays the name of the GPU on the current machine and its GPU Memory or Utility usage. If the machine has multiple GPUs, multiple fields will be shown (as in the following example). You can decide to deploy the AI models you need on any GPU of the machine. Depending on the computing power of different types of machines, up to 20 AI models can be placed.

Click Edit on your model to choose the version and type to be deployed, and select Update once finalized.

Once satisfied with all the models deployed, click Apply on the lower right of the window to confirm all changes and apply the new AI model settings.

Tip: Applying and deploying a new AI model takes time, possibly several minutes or more depending on network speed. During this time, tasks using this AI Model will be paused until the new AI model setup is complete. Please be patient and wait for all settings to be properly finalized.

Note: If the GPU Memory is fully loaded, adding new AI models to run may fail. You can solve this problem by shutting down some AI models that are not in use.