ChatGpt is too expensive? The solution: Your own Ollama server in Docker

Artificial intelligence and large language models (LLMs) are revolutionizing the way data is handled. However, many companies are faced with the challenge of providing a cost-efficient and flexible infrastructure for the use of LLMs. This is where the portable Ollama Server comes into play - operated in Docker. We show you how you can use it directly.

Advantages of the Ollama server

• Lower costs: Compared to the use of cloud-based services such as ChatGPT, there are no ongoing API costs with a local server.
• Larger model selection: The Ollama server supports various LLMs and enables the use of models that are optimally tailored to individual requirements.
• Data sovereignty: The data remains local, which is a decisive advantage, especially for sensitive company data.
• PortabilityContainerization in Docker makes the Ollama Server easy to port and run both locally and in many cloud environments.

Setting up the Docker-based infrastructure

The installation of the Ollama server in a Docker environment offers a flexible and portable solution that can be easily integrated into existing systems.

We will show you the right config so that you can use the server immediately.

Docker-Compose Config

version: '3.8'

services:
  ollama:
    build:
      context: .
      dockerfile: Dockerfile
    container_name: ollama
    ports:
      - "11434:11434"
    restart: always

Dockerfile for the Ollama server

FROM ollama/ollama:latest

COPY run.sh /run.sh
RUN chmod +x /run.sh

ENTRYPOINT ["/run.sh"]

Start script

#!/bin/bash

echo "Starting Ollama server..."
ollama serve &
SERVE_PID=$!

echo "Waiting for Ollama server to be active..."
while ! ollama list | grep -q 'NAME'; do
  sleep 1
done

ollama pull llama3.2

wait $SERVE_PID

This configuration ensures that the Ollama server is started and automatically downloads the desired model (e.g. llama3.2).

Use of an LLM

To interact with the LLM, you can connect to the LLM via CMD in the container as soon as the container has been started. The use of an LLM is possible via:

docker exec -it <CONTAINER_ID> ollama run llama3.2

Alternatively, HTTP requests can be sent to the Ollama server via port 11434. The API doc for this is available in the Ollama GitHub Project to find.

Potential for more performance

The Docker container presented here uses the CPU to process LLM operations. To further improve the processing speed, the container can be adapted to use the GPU, provided a correspondingly high-performance GPU is available on the host.

This requires the following adjustments to the infrastructure and the container:

• Install the NVIDIA Container Toolkit: Make sure that the host system is set up correctly by running the NVIDIA Container Toolkit install and configure.

• Use a CUDA-compatible Docker image: Use an NVIDIA CUDA base image as the basis in the Dockerfile to ensure GPU support, such as the ollama/ollama Image from the example above.
• Start the container with GPU access: When starting the container, use the option --gpus allto make the GPU of the host available for the Ollama server.