Grace
All projects
Personal project2026

Private LLM platform

My own platform for open-source language models: a K3s cluster, a remote NVIDIA GPU joined over a private network, and full observability.

Why I built it

Curiosity. Public AI assistants send every prompt to someone else's servers, and I wanted to understand what it really takes to run the whole chain myself, from the GPU to the chat interface, without exposing a single service to the internet. So I built it, to test, practise and learn.

Architecture

Architecture: a K3s control-plane node running Open WebUI, the vLLM router, the K3s server, Prometheus and Grafana, linked over Tailscale to a remote GPU worker node running the K3s agent, the NVIDIA device plugin, the vLLM engine and the DCGM exporter.
Two nodes. The control plane runs the cluster, the chat interface, the model router and monitoring. The remote GPU worker runs the model engine and exports GPU metrics. Tailscale links them over WireGuard.

What I built

  • Built a K3s cluster with a remote NVIDIA RTX 3090 worker node, joined over Tailscale so that no service is exposed to the internet.
  • Deployed vLLM with Helm to serve open models, such as Qwen3-8B, through an OpenAI-compatible API.
  • Deployed Open WebUI as the chat and administration interface for the models.
  • Set up observability with Prometheus, Grafana and the NVIDIA DCGM exporter, covering Kubernetes, the GPU and vLLM.
  • Created Grafana dashboards to follow GPU performance, cluster health and model serving.
  • Wrote a Python MCP server that brings Kubernetes administration, GPU monitoring and vLLM metrics into Visual Studio Code.

Outcome

A private platform that works end to end. Open models run on the remote GPU, the chat interface is only reachable inside my Tailscale network, and dashboards show the GPU, the cluster and the model serving in real time. I documented the whole build in an article on Grace Your Brain.