Building a Private Cloud-Native AI Platform: Deploying Ollama & Open WebUI on Kubernetes
Running Large Language Models (LLMs) locally on personal hardware often leads to memory bottlenecks, high latency, and thermal throttling when processing larger context windows. Deploying your inference engine to a managed cloud Kubernetes cluster resolves these hardware constraints, providing compute scalability, high availability, and isolated resource management.
This step-by-step guide walks you through building a private, production-grade AI inference platform using Ollama, Open WebUI, and NGINX Ingress hosted on Linode Kubernetes Engine (LKE)
🏛️ Architecture Overview
The system architecture decouples the frontend UI from the backend LLM inference engine, keeping inter-service traffic isolated within the internal cluster network
- Ollama Pods (Llama Logos): LLM inference service executing Llama 3.1 models inside Kubernetes pods.
- Open WebUI (Icon “OI”): Web interface for interacting with Ollama, communicating directly over internal cluster network routes (
http://ollama). - NGINX Ingress (Hexagonal “N”): Manages external traffic routing, load balancing, and TLS (HTTPS) termination.
- External Endpoints:
ollama.service: HTTPS endpoint for raw API access, secured by an HTTP Basic Authentication lock. webui.service: HTTPS endpoint for WebUI browser access.
2. Step-by-Step Implementation Guide
Step 1: Cluster Configuration & Authentication ( kubeconfig )
- Obtain the cluster configuration file (
kubeconfig.yaml) from the Linode Cloud Manager interface. - Create and paste the
kubeconfig.yamlfile into your local administration terminal (e.g., Kali Linux):
- Export the configuration and verify worker node status:
export KUBECONFIG=kubeconfig.yaml
kubectl get nodes
kubectl cluster-info
Step 2: Create the Dedicated Namespace
Isolate all AI workload resources inside a single Kubernetes namespace:
kubectl create ns ollamaStep 3: Configure & Deploy the Ollama Engine ( ollama-deployment.yaml )
Create a deployment and service manifest incorporating preloading, memory retention, and persistent host storage
Key Environment Parameters:
PRELOAD_MODELS="llama3:1b"(or"llama3.1"): Triggers automatic model downloading via apostStartlifecycle container hook[6].OLLAMA_KEEP_ALIVE="12h": Retains loaded model weights in RAM for 12 hours to eliminate disk reload latencies[6].- Storage Persistence: Mounts host directory
/opt/ollamaas ahostPathvolume (ollama-storage) to container path/root/.ollama[6]. - Resource Allocations: Requests set to
2GiMemory /1CPU; limits capped at4GiMemory /2CPU[5].
- Apply Manifest & Verify Deployment:
kubectl apply -f ollama-deployment.yaml
Check Ollama Deployment Status
Exec into the poc to interact with ollama
Step 4: Configure & Deploy Open WebUI ( webui-deployment.yaml )
Next, create the deployment and service endpoint for WebUI using the YAML file below. The data storage /app/backend/data is mounted to a host folder. The WebUI pod is fixed to a node by nodeName: ollama-worker-1 to keep the data stored, as it contains the login data. It connects to Ollama through the service endpoint http://ollamawithin the cluster.
- Key Deployment Parameters:
OLLAMA_BASE_URLS="http://ollama": Points WebUI directly to the internal Kubernetes service on port 80[7].nodeName: ollama-worker-1: Pinning the pod to a specific worker node ensures data saved at/app/backend/data(mapped to host directory/opt/webui) persists user login credentials across container cycles[7].
- Verify Pod Status:
kubectl get pods -n ollama
Step 5: Edge Ingress, HTTPS TLS & Basic Authentication Setup
We will expose the Ollama API and WebUI via Nginx ingress using HTTPS for security. The WebUI has its own login, so we will enable HTTP basic authentication for the Ollama API.
Get Gmptg’s stories in your inbox
Join Medium for free to get updates from this writer.
Nginx Ingress Controller
Install the Nginx Controller in your cluster. Follow the instructions for your specific cluster variety.
Secure public access points using NGINX Ingress:
- Install NGINX Ingress Controller: Manages external routing and load balancing.
- Configure Security Controls:
- ollama.service (API Endpoint): Secured with HTTPS TLS termination and enforced HTTP Basic Authentication.
- webui.service (Web Endpoint): Secured with HTTPS TLS termination while allowing WebUI to handle user login natively.
Local Certificate Installation:
If you have a certificate issued by a well-known CA, you can skip this step. Otherwise, create a self-signed certificate for testing purposes. The command below generates a certificate with two Subject Alternative Names (SAN): ollama.service and webui.service.
As we need to enable HTTP Basic Authentication, follow Basic Authentic to create a secret basic-auth with the username and password.
Then create the Ollama ingress with the YAML file below
Apply that configuration to Kubernetes
The WebUI Ingress
Similar settings for WebUI ingress, but without basic authentication.
next apply the yaml file to the kubernity like we did with ingress
we have a Load Balancer, we can expose the endpoints via port 443.
Configurations on Your Host
1. Update Your Host File
To add the entries for ollama.service and webui.service, follow these steps:
- Open the /etc/hosts file in a text editor with root privileges:
Update Your Trust Store
Since we are using a self-signed certificate, the certificate verification will fail by default. While you can skip verification with options like curl -k, this may not always be available in SDKs and API calls. It’s better to add the certificate to the trust store of the tools you are using.
Since we use a self-signed certificate, the certificate verification will fail. We can skip the verification with some options, like curl -k, but not always available in SDK and API calls. Better to add the certificate to the Truststore of whatever tools you are using.
For example, if you are using Python, you can use the certifi library to manage your trust store.
Step 6: Verification & Web Interface Testing
- Certificate Verification:
- Shows SSL/TLS certificate details issued to
example.com/webui.servicewith SHA-256 fingerprints[8].
- Admin Account Setup:
- Interactive Inference Testing:
