OpenCV 5.0 brings LLMs to the Computer Vision Library

Contents

With OpenCV 5.0, a new major version of the widely used computer vision library has been released. At the core of the release is a completely redeveloped deep learning engine (DNN). It supports significantly more ONNX models than before, executes modern transformer architectures more efficiently, and for the first time also processes language models (LLMs) and vision-language models (VLMs) directly in OpenCV. Furthermore, the developers are modernizing the library's core, expanding hardware acceleration, and enhancing 3D functionalities.

OpenCV (Open Source Computer Vision Library) is one of the most important open-source libraries for image processing and computer vision. It is used in robotics, industrial automation, medical technology, AR/VR applications, and embedded systems, among others. The library offers numerous algorithms for image recognition, object detection, calibration, tracking, and 3D reconstruction.

New DNN Engine with Significantly Improved ONNX Support

The most significant innovation is the revised DNN engine. According to the project, support for ONNX operators is increasing from around 22 percent in the 4.x series to over 80 percent. ONNX (Open Neural Network Exchange) has established itself as a common exchange format for AI models. Previously, importing modern models into OpenCV often failed due to missing operators or limitations with dynamic input sizes.

The new engine relies on graph-based execution: it no longer processes models as a simple sequence of layers but analyzes them as a computation graph. This allows for optimizations such as shape inference, constant folding, and operator fusion. Also new are support for dynamic shapes, control flow constructs like If and Loop blocks, and quantization graphs.

Attention Fusion is particularly relevant for current AI models: the engine recognizes typical transformer patterns and combines multiple operations into a single, optimized computation. This is intended to accelerate modern transformer models and reduce memory requirements. The project describes details about the new engine in the Überblick zu OpenCV 5 auf der Projektseite.

Language and Vision-Language Models directly in OpenCV

Additionally, there is the integration of language and multimodal models. For this, OpenCV 5 includes its own tokenizer and a KV cache for autoregressive text generation. It supports model families such as Qwen 2.5, Gemma 3, and PaliGemma (partially). Thus, OpenCV no longer covers only classic image processing but also vision-language scenarios – for example, when a model analyzes an image and then describes it in natural language.

To facilitate the transition for existing applications, the previous DNN engine is being retained. OpenCV 5 thus provides three execution variants: the new engine, the classic engine, and optionally ONNX Runtime. Applications can switch between variants as needed without modifying their DNN API. Which engine is used can be controlled when loading a model via a parameter from the enum cv::dnn::EngineType; by default, ENGINE_AUTO automatically selects the appropriate variant.

Feature Matching via Deep Learning

OpenCV is also increasingly relying on deep learning for feature matching. The new Features module replaces the previous Features2D and complements classic methods like SIFT or ORB with neural alternatives, including ALIKED, DISK, and LightGlueMatcher. Such methods are used, for example, in assembling panoramas, for Visual SLAM, or in 3D reconstructions.

LightGlue uses attention mechanisms to match image features more robustly than classic methods. The classic detectors remain available, allowing the new deep learning path and established methods to be combined depending on the use case.

Modernized Core and Streamlined API

The developers have also modernized the library's core. OpenCV now supports FP16 and BF16 data types, which are widely used in current AI accelerators, as well as Bool and other integer variants. The matrix class cv::Mat can map true 0D and 1D structures for the first time and now supports broadcasting as well as other N-dimensional operations. This is intended to save many workarounds and conversions.

Regarding interfaces, the project is gradually shedding legacy components; the historical C API is now officially considered deprecated. For Python, OpenCV 5 supports NumPy 2.x and integrates named parameters more strongly, allowing functions to be called more readably – for example, cv.someAlgorithm(threshold=0.5) instead of a purely position-based argument passing.

Hardware Acceleration via Revised HAL

Another key topic is hardware acceleration. The developers have fundamentally revised the Hardware Abstraction Layer (HAL) to more easily integrate optimized implementations from various hardware manufacturers. The project mentions Intel IPP, Arm KleidiCV, Qualcomm FastCV, and support for the vector extensions of modern RISC-V processors, among others.

This allows applications to benefit from acceleration on different processor architectures without modifications. This is made possible, among other things, by a unified vector codebase that addresses various instruction set extensions such as SSE, AVX, NEON, SVE, and RVV through a common interface.

Expanded 3D Stack

The 3D functionalities have been significantly expanded. The previous calib3d module will be split into three modules: 3d, calib, and stereo. New functions are added for calibrating multiple cameras, importing and exporting point clouds and meshes, and methods for 3D reconstruction based on TSDF volumes. Modern estimation methods like MAGSAC are also being incorporated into OpenCV. These extensions are primarily aimed at developers in robotics, autonomous systems, and industrial 3D measurement.

Further innovations are found in image processing; the documentation will henceforth rely on a combination of Sphinx and Doxygen. The project provides the source code in the GitHub repository; installation via pip is also planned.

(fo)