Running a Local LLM with Ollama and Building RAG

Why run it locally

Cloud LLM APIs are convenient, but there are moments where you stop at “is it acceptable to send this data to an external API”. That is exactly why I went near local LLMs.

• Local LLM — running the model on your own machine rather than through a cloud API. The biggest benefit is that data does not leave
• Ollama — a tool that pulls and runs a local LLM in one command. Swapping models is easy
• RAG (retrieval-augmented generation) — rather than making the model memorise everything, search for relevant documents, have it read them, then answer. Specialist knowledge lives as a knowledge base outside the model

Why Qwen

Qwen is an LLM published under an open licence, and it pulls straight down through Ollama.

What was good in use was that it has few constraints and is straightforward. On the premise of running locally, that ease matters. I could concentrate on the body of the RAG system (the join between retrieval and generation) rather than wrestling with heavy configuration.

The setup I built

A plain RAG system: documents from a specialist field as the knowledge base, a question comes in, relevant passages are retrieved, and Qwen reads them and answers.

In a field demanding accuracy and rich in primary sources, having RAG pull the supporting documents is sounder than relying on the model’s memory. Being local, being able to try it without the data leaving was reassuring too.

What you have to accept

To avoid writing an advertisement, honestly.

Do not expect the intelligence of a top-tier cloud model — running it on your own machine means it is not on the same footing as a vast model. It is something you use on the premise of a narrowed purpose.

A RAG system’s quality is decided by retrieval — more than the model, it is the retrieval design that pulls the right documents that matters. Weak there and a good model still answers wide of the mark.

Where the time should go

The second point was the essential one. When you feel like swapping the model, it is usually a retrieval problem.

Trying a local LLM tempts you into spending your first hours on model selection. But what was actually deciding answer quality was which documents get cut out how, and how they are retrieved.

The more model options there are, the more the structure lets the fun of choosing hide the essential thing. Being able to try a specialised setup without the data leaving is worth a lot, so on that basis it is worth not getting the allocation of time wrong.