Skip to content

Type a term, e.g. API, Shopware or SEO.

    ← All AI tools
    For developersOpen sourceLast checked

    Inference Manager

    Command-line tool from Canonical that installs local AI models on Ubuntu and serves them through a single OpenAI-compatible endpoint.

    What it does

    Inference Manager is a command-line tool from Canonical for Ubuntu and other Linux systems with Snap. You use it to find local AI models, install them as so-called inference snaps, remove them again and check their status. Each snap bundles a model with its runtime and picks a variant optimised for your hardware – CPU, GPU or NPU from vendors such as Intel, NVIDIA, AMD or Ampere. The catalogue includes Gemma, Qwen, DeepSeek R1, Nemotron, Phi 4 and an embedding model. All installed models are reachable through one shared endpoint on localhost:8400, which you address like the OpenAI API.

    Who it suits

    Developer and IT teams that run language models on their own Ubuntu machines or servers and want to connect existing OpenAI-compatible applications locally without setting up runtimes and driver optimisations themselves.

    Data protection for businesses

    The models run on your own hardware; requests are not meant to go through a cloud service of the vendor. You therefore decide where data is stored. The vendor does not mention a data processing agreement for the tool. The vendor does not describe any use of your inputs for training. Canonical does not name business plans or dedicated support for Inference Manager.

    Limits

    The tool requires Linux with Snap; on Windows, the documentation lists only one model (Gemma3) as usable via WSL 2. It is operated almost entirely from the command line, and after installation you have to connect the snapd-control interface manually. Many models are specifically optimised only for NVIDIA graphics cards. Version 1.0 was released only in November 2026.

    For businesses

    Looking for help with integration?

    Choosing a tool is the easy part. We bring AI into your existing workflows – with interfaces to CRM, ERP and shop, sound data protection and an onboarding so your team actually uses it.

    Ask for advice →

    Your AI tool is missing here? Submit an AI tool →

    Newsletter

    News from the vona workshop.

    AI tools we actually use, our weekly AI review & insights from our projects – short, practical and without spam.