How to Self-Host Your Own Private AI Stack

26 thoughts on “How to Self-Host Your Own Private AI Stack”

• This is everything I want to do! Many thanks!!! 🙏

• You could run all the AI stuff in an LXC. You can pass the GPUs through to the LXC. The way that I figured out how to do it is mapping the devices in the LXC config and then install (the same!!) drivers on the host and in the LXC. You can actually share the GPUs this way if you wanted.

• Yeah I run 6x RTX A6000 (Ampere generation, same as the 30 series consumer cards) in 2 GPU nodes in my homelab. I don't train, but I do have a bunch of agents, and some automations that run a lot, so parallel compute of AI models is important enough to have spent a mid-sized car's worth on GPUs. EDIT: I'd also suggest gemma2:27b from ollama, it's a great model, better than llama3:8b in my testing (and in some, better than llama3:70b … i can run both).

• Well there's not a chance that I'll recreate this beast of a configuration but it was a fun watch! Thank you

• ollama/ollama:rocm docker container works great on my RX6700 XT with any model that fits within VRAM that I tested so far

• Cool and all, but still something it looks like I can't do with the hardware available to me. My best GPU is in my Gaming PC, a 6700XT…a 3 year old card. My server downstairs is from 11 years ago. Works great for the tasks I currently use it for.

• Thanks for covering this so in-depth Tim. I've been working on a similar project based on a Dell R730 with 2x Tesla P40 cards…much more power hungry than your setup, but will spit out answers from 70b and complex Stable Diffusion workloads as fast as the small models generate on a 3090. I was running into some issues with getting things integrated in the same stack, so great to see how you've put this all together….I have some rebuilding to do.

• What about VirGL? Would it still work with docker?

• Great video! How do you update the stack? git pull and docker compose pull or?

• I did a similar setup with Debian and rocm containers for AMD APU

• Thank you for the video, a very nice guide.

  One potential thing to play around with is quantization of the models, one can find one that is less quantized but still fits in the memory. For example with gemma2 27b once gets `q4` that takes 16 GB by default but one could get `27b-instruct-q6_K ` that takes ~21 GB and perhaps gives slightly better results. Of course then one has less space to space to host the models for other services like stable diffusion or whisper. One needs to click on tags when picking the model size on ollama website to see the full list.

  Another nice potential addition to the stack could be `openedia-speech` to handle text-to-speech. It can be integrated with Open Web UI. Not a must have but complements the stack nicely IMHO.

• Could this run with multi gpu? Like 4x P40s

• i'm considering redeploying my private ai using docker stack 😩thanks a lot, tim! 🤓

• This is too cool! Thanks for doing this!!!

• I ran into an error with starting Open WebUI

  HFValidationError: Repo id must be in the form 'repo_name' or 'namespace/repo_name'

  I solved it through some Googling by setting the environment variable RAG_EMBEDDING_MODEL_AUTO_UPDATE=True. Youtube doesn't like links so I can't seem to share the GitHub issue I found it under

  Whishper also doesn't seem to work, but I will probably just stand up my own API for faster-whisper. Other than the one bug with open webui, everything ran great on a 2060 w/ 12GB of VRAM.

• is it possible to run this in windows and with self signed certificate for local use only?

• Awesome video. I am trying to run ollama in kubernetes but now i think it will be easier to run as docker swarn

• An overview of what a local ai can actually do would be great lol. Because I don't want to jump in it without knowing the final goals 😂

• I'm always surprised when everything just works… Kudos & thanks for the support notes… Great job

• Do you think you'll be adding RouteLLM? I just watched a few videos about it. Apparently it has accuracy up to 95% of GPT-4 with an 85% cost reduction

• @TechnoTimTinkers Permission denied (tailscale) help fixed

• Oh baby! Time to put some of my Tesla GPUs to work 🙂

• I have a 4090 for gaming local LAN stream gaming and have considered using it as a AI workhorse but the energy it takes to run it on an always on server is keeping me from doing so.

• This guy just knows all ! I am watching all of his videos for almost two years and finally networking and hypervisor are getting easy. I just needed this detailed discussion about personalised AI. ❤

• Thank you so much for this video!
  At 31:37, this is how mine runs because I have a small GPU.

• Every time toy called the RTX 3090 "GTX 3090" it just hurt a bit inside. But otherwise cool ideas, thank you.

Comments are closed.