Build a Local LLM Setup: All 30 Parts from Windows to RAG and Operations

This is a 30-part series covering everything from the first answer of a small model to document search, RAG, tool call, and operation and restoration. The basic environment is Windows PC and Ollama. Hermes, OpenClaw, and OpenRouter, which connect external models, can be viewed by selecting the parts you need.

Base environment: Windows 11 x64, local Ollama, base model qwen3:4b-instruct. Python labs focus on 3.12 and the standard library. The model tag/official document confirmation date is October 18, 2026 and does not represent the latest performance ranking.

If you’re short on time, read them in this order:

goal	Recommended route
Talk to your local AI today	01 → 02 → 03 → 04
Answer based on my document	08 → 13 → 16 → 17 → 18 → 29 → 30
Understanding agent structure and tools	19 → 20 → 21 → 22 → 23
Compare with external models	24 → 25, check external transmission and charging conditions

You don’t have to install the whole program from scratch. You can choose WSL·Docker·LM Studio and agent according to the functions you want to use. Docker Ollama in Part 12 has a different repository from Windows Ollama, and CUDA learning in Part 28 is an optional exercise in a separate environment.

First run and model selection

1. Things to check before starting local LLM
   When starting local LLM, it is enough to run a small model on your current PC. Just add a short note and see if it summarizes the content properly. This series starts from the first execution and continues through document search and operation.
2. Install Ollama on Windows 11 and get your first answer
   All you need to get your first answer on Windows is Ollama and a small conversation model. Install Python or Docker when using it behind the scenes. First, let’s briefly follow the steps from downloading, running, and checking the response.
3. How to choose a recommended model: Look at tasks and memory before the name
   When choosing a model, it is best to decide on one task first. Korean memo summaries, code modifications, and photo descriptions require different skills. After selecting candidates of a size that can be run on a PC, compare them by adding actual questions.
4. GPU·CPU and VRAM, how to read PC specifications
   When looking at PC specifications for local LLM, first check whether the model fits into memory. Even if the GPU is fast, if VRAM is insufficient, some of the weight must be moved to system RAM. It is difficult to determine the bottleneck that occurs at that time based on the game performance table alone.
5. Token·Context·Quantization and Memory Usage
   If you missed the previous content in the long conversation, take a look at the execution context first. The set limit may be small, the input may be truncated, or you may not be able to find what you need in a long document. If you know the token and context, you can see this phenomenon along with memory usage.
6. Move the model storage location and check offline execution
   If you receive several models, the storage space decreases faster than expected. Let’s first share where the Ollama program is installed and where the model files are stored. Knowing the path makes moving disks or re-preparing your PC less confusing.
7. Using the GGUF model and API in LM Studio
   If you want to select the model file and quantization type on the screen, you can use LM Studio. Let’s follow the downloading, loading, and response confirmation that we did in Ollama on another screen. You don’t have to keep using both, so just look at the differences in management methods and choose whichever you are more comfortable with.

Order of setting up a practice environment

8. Installing Python and creating a virtual environment for each project
   If it says the package is installed but the import fails, check the running Python first. The Python you put the package in and the Python you ran the code in may be different. Let’s create a virtual environment in the project folder and check the path directly.
9. Node.js·npm Check the installation and execution path
   Node.js runs JavaScript outside the browser, and npm is used to install the necessary packages. OpenClaw, which we will cover later, also operates as a Node. Here, we will check which version and path is actually used after completing the installation.
10. WSL2 installation and file/network distinction
    WSL2 installation allows you to use Linux tools on your Windows PC. When going back and forth between terminals, it’s easy to think you’re looking at the same file or localhost. Let’s start by breaking down where the files are stored and what environment the server runs in.
11. Docker Desktop and web chat screens such as Container·Volume·Port
    Open WebUI can be run as a Docker container. Preparation is easy because you can bundle and launch the necessary programs. However, you must first know where the dialog and model will remain when the container is deleted.
12. Set up web chat with Open WebUI and Ollama
    Add Open WebUI to keep conversation history and choose a model in the browser. This exercise puts WebUI and Ollama on the same Docker network and runs separately from the Ollama installation already prepared on Windows.

From API to My Document Q&A

13. Call Ollama API with Python
    You can also send questions asked in the chat window through Python code. This is a necessary starting point when automating file summarization or inspection memo organization. Let’s create a small function to send the request as JSON and retrieve a sentence from the response.
14. Prompt that clearly conveys the original text and conditions
    When requesting a summary, it is easier to check the results if you provide the original text and necessary items together. Write down how you will indicate if you cannot find the person in charge and how long the response will be. Let’s change the instructions slightly on the same note and compare the results.
15. JSON Schema and response verification
    To pass the answer to the next program, human-readable sentences alone are not enough. Even if there is a greeting in front of JSON, the parser can stop. Let’s check whether the format was followed and the values ​​contained within are correct.
16. Search documents with embeddings and cosine similarity
    A question about how many days to keep a backup may need to find a passage using the phrase retention period. Embedding search helps bridge wording differences by representing text as vectors. Inspect the retrieved passages before adding answer generation.
17. Build RAG answers and retain supporting sources
    Send the retrieved document passages with the question so the chat model can use them as evidence. RAG retrieves relevant information for each request without retraining the model’s weights on the documents.
18. Build an evaluation sheet for correctness, evidence and failures
    Keep a fixed question set for comparisons after a model change or update. Include successful questions, previous failures and questions the documents cannot answer. A small evaluation sheet can assess retrieval separately from answer generation.

Associate models with tools

19. Hermes Model and Chat Template
    You will encounter both model and agent programs under the name of Hermes. It’s easier to read the installation instructions if you separate them first. This article covers Nous Research’s conversational model, and the next article links to the Hermes Agent.
20. Hermes Agent connection and tool/memory settings
    The agent runs the tool based on the model’s answer and passes the results back to the model. You can use it to reduce repetitive tasks, but you must first decide which materials and tools you want to access. Start by connecting the Hermes Agent to your local model, but with a narrow read and execute scope.
21. Build a tool-call loop: the model proposes, the program executes
    Connect an example equipment inventory lookup tool to see where model decisions end and application responsibility begins. Follow the request, argument validation and result handling without allowing file edits or shell execution.
22. OpenClaw installation and gateway confirmation
    OpenClaw is a program that connects the conversation screen with model providers, tools, and messengers. It operates separately from Ollama, which is in charge of model inference. After installation, run the CLI, start the Gateway, and check the model response in that order.
23. Check OpenClaw’s Ollama address and tool call
    If you can chat but cannot call the tool, check the provider settings and API path first. Currently, OpenClaw’s Ollama provider uses a native API. If you copy the address used in Hermes, the path may not be correct.
24. OpenRouter API and check rates/transmission routes
    OpenRouter allows you to make requests to multiple external model providers with a common API. Since the model runs externally, you need to know where the data is going and how much it costs. Let’s walk through the call process with a hypothetical question.
25. Standard for dividing local and cloud requests
    You can use the local model as the default and send only some questions to the external model. In this case, the user must be able to confirm the transmission destination. Let’s create a branch that doesn’t automatically hand over to the cloud if the local execution fails.

Expanding to performance, learning, and operations

26. Find the delay: measure first response, generation speed and memory
    Separate time spent reading model files, processing the input and generating tokens. Repeat the same question while recording each stage so that loading delay is not confused with generation speed.
27. Run GGUF with llama.cpp and inspect inference options
    Choose the model file and run options directly, first on CPU and then with GPU offloading requested. Compare the logs and identify which settings you now manage yourself rather than leaving them to Ollama.
28. LoRA·QLoRA and the structure of small learning code
    If you want to reflect the new document in your answer, first see if it can be solved with RAG. If you want to change the repetitive answer format or classification behavior, you can consider fine-tuning. Here, we read the principles of LoRA·QLoRA and check the connection structure of a small learning code.
29. Review API, log and tool permissions
    Local execution still requires checking listen addresses, source text retained in logs and external tool and file access. Map the data boundaries in the configuration built throughout the series.
30. Back up and restore the document Q&A tool
    Plan how documents and scripts are updated and how the tool can be restored from backups. Test starting again in a new directory and record the startup, update and recovery procedure.

Practice files and verification scope

Get Python·PowerShell·Compose practice file ZIP. Includes fictitious documents, blank scorecards and operating record forms. It does not include model weights, account keys, or actual business data. Do not run the entire file at once, but follow the prerequisites in that section.

The verification scope of the practice file includes Python grammar, mock HTTP requests, and search flow. Windows installation and GPU inference and CUDA learning must be verified on the actual device. The speed and quality of the model are checked using the scorecards and measurement codes provided.

If you are starting out for the first time, write down your PC specifications and tasks in Part 1 and then proceed to running Ollama in Part 2. If you already have a model in use, you can select the article you need from the series list below or the subtopics on the left.