Run LLMs locally on the command line with Docker Desktop 4.40
With the Docker Model Runner, AI models can be started directly from Docker Desktop – with GPU acceleration and packaged as an OCI artifact.
(Image: Peshkova/ Shutterstock.com)
The company Docker, Inc. has released version 4.40 of Docker Desktop. An important new feature is the Docker Model Runner, which allows developers to run Large Language Models (LLMs) locally directly on the command line. The Docker AI Agent will also use the Model Context Protocol (MCP) in the future and can therefore also be integrated into workflows that go beyond Docker.
Even without containers: working locally with AI models
The Docker Model Runner is now integrated as a plug-in in Docker Desktop and is activated by default – but initially still in a beta version. According to the announcement in the Docker blog, the declared aim is to provide developers with a simple way to work quickly and easily with AI models locally on their system – even outside of a container. The models can be downloaded from Docker Hub via docker model pull and saved locally.
Execution and interaction with the LLMs is – similar to Ollama – via the command line, either by prompt or in chat mode. For inference, Docker has integrated the open-source library llama.cpp, including access via the OpenAI API. In addition, models are packaged as OCI artifacts so that they can be integrated into established CI/CD workflows as well as into trusted registries.
During the beta phase, Docker Model Runner is initially only available for macOS and uses the GPU acceleration of Apple Silicon processors on these platforms. Support for Windows and other operating systems will follow. In addition, Docker also wants to enable developers to adapt and publish their AI models in the future.
Videos by heise
AI-supported help beyond Docker
The Docker AI Agent has long been available to developers as an intelligent helper. With the release of Docker Desktop 4.40, however, the agent is set to gain even more capabilities, including downloading and managing local files as well as executing Git operations and shell commands. The Docker AI Agent now also offers full support for the Model Context Protocol (MCP), which can be used to connect AI agents and models with external data and tools. The Docker AI Agent can act both as an MCP client and provide its functions as an MCP server. In this way, the agent's support can also be used directly in Docker Desktop (GUI and CLI) and other environments such as the Cursor IDE or Claude Desktop.
Other new features include Desktop Settings Management, which was introduced in version 4.36 and now also includes compliance reporting for Docker Business customers. A complete overview of all changes in version 4.40 can be found on the Docker blog and in the release notes.
(map)
