All videos
0:00 / 0:00
ai

Open Source AI Crash Course: Go Local in 45 Minutes

Mark Kashef22 July 2026Watch on YouTube

What you'll learn

  • You can run open source AI models on your own hardware, with quantization as the key technique to compress large models and make them faster without significant quality loss.
  • The 5-layer stack (model selection, runtime, API layer, mesh networking, user interface) forms the foundation for a functioning local AI system.
  • You don't have to choose between closed and open source models: you can use Claude or Codex to set up and maintain your own open source stack.
  • With tools like Hugging Face, llama.cpp, LiteLLM and Tailscale, you can build a local AI command center in a short time that gives you full control.
  • KV cache, tokens per second and model parameters are critical concepts to understand when choosing the right setup for your machine.

Frequently asked questions

What is quantization and why is it important for local AI?
Quantization compresses AI models by using fewer bits per number, similar to JPEG compression for images. This makes large models smaller and faster without significant quality loss, allowing them to run on your own hardware.
How do you choose the right open source model for your computer?
The crash course shows how to explore models via OpenRouter and Hugging Face, and use a calculator to check if your computer can run them. You assess model size (parameters) and processing speed (tokens per second) to find a good match.
What is the 5-layer stack and why is it important?
The 5-layer stack consists of model selection, runtime environment, API layer, mesh networking and user interface. This is the fundamental architecture needed to set up a functioning local AI system.
Can you use closed models like Claude in your open source setup?
Yes, the crash course shows how you can use Claude and Codex to set up and maintain your open source stack. This way you get the best of both worlds without being dependent on a single source.

Topics

open source ailocal inferencellm deploymentquantizationself-hosted

Description from the channel

Master AI: https://www.skool.com/earlyaidopters/about Get the kit FREE: https://markkashef.gumroad.com/l/open-source-ai-crash-course-kit Work With Us: https://www.promptadvisers.com Over the past few months we lost access to two frontier models overnight. That was the wake-up call. In this crash course, I break down everything you need to run open source AI on your own hardware, from the key concepts and jargon (quantization, GGUF vs MLX, KV cache, tokens per second) to picking the right model for your machine, to the path of least resistance for building your own local AI command center. The twist is that you don't have to pick sides. I'll show you how to use closed source models like Claude and Codex to set up, build, and maintain your entire open source stack, so you get the best of both worlds. The free kit (second link above) includes the 21-page visual crash course guide plus the full interactive Local AI Engineering Guide with a built-in can-I-run-this calculator. --- 0:00 - the two AI scares 1:42 - the 3 sections of this course 2:01 - my local AI command center (demo) 5:00 - the fake war between closed and open source 5:56 - 4 reasons to run AI locally 8:42 - "but local models were toys" 9:39 - the 5-layer stack (most important diagram) 12:52 - find the right models with openrouter 15:45 - what can YOUR computer run 18:20 - downloading models from hugging face 21:32 - same workflow on a mac mini over SSH 22:05 - llama.cpp vs vLLM in plain english 23:13 - litellm, the switchboard 24:24 - tailscale, your private mesh network 26:33 - the best open source models right now 28:26 - using claude code's harness with open models 31:55 - parameters, what the B actually means 33:38 - quantization, JPEG for brains 34:56 - GGUF vs MLX 35:57 - KV cache, why chats slow down 37:49 - sparse attention 38:20 - tokens per second 39:13 - harnesses and pi.dev 40:47 - build your command center in one command (ODS) 43:34 - wrap-up + your free resources #opensourceai #localai #ollama #llm #claudecode #huggingface #localllm #aitutorial #machinelearning #selfhosted #quantization #tailscale #aicrashcourse #opensource