What Are Open-Source LLMs? Run One Locally With Ollama
What open-source LLMs are, why people run them locally, and how Ollama lets you download and use one with a few commands.
Short answer: Open-source LLMs are language models whose weights you can download and run yourself, instead of calling a company's hosted service. Ollama is a free tool that downloads and runs such models on your own computer with a couple of commands, and it also exposes a local API your code can call.
What are open-source LLMs?
A large language model is a neural network trained on text (see What is a large language model). With a hosted model you send your prompt to a provider over the internet. With an open model the trained weights are published, so you can run them on your own hardware.
Strictly speaking, many of these are "open-weight": the weights are available, but the training data and code may not be. Licences also differ, and some limit commercial use, so always read the licence of the model you pick.
Why run an LLM locally?
Reason | What it means |
|---|---|
Privacy | Your prompts stay on your machine, which helps with sensitive text. Check your own rules and laws. |
Cost control | No per-token bill. You pay with your own hardware and electricity instead. |
Offline use | Works without an internet connection once the model is downloaded. |
Learning and control | You can try different models and settings freely. |
The trade-offs are real. Models that fit on a laptop are usually smaller and less capable than the largest hosted ones, and bigger models need more memory and a good GPU to run quickly.
What is Ollama?
Ollama is a tool that handles downloading, storing and running open models. You install it from the official Ollama website, then use the terminal. Model names come from the Ollama library, so browse it and choose one that fits your computer's memory. Smaller models run on modest hardware.
ollama pull your-model-name # download a model from the Ollama library
ollama run your-model-name # chat with it in the terminal
ollama list # show models on your computerReplace your-model-name with a real name from the library. Ollama itself may update its commands over time, so check its documentation if something differs.
How do you use a local model from code?
While Ollama is running, it serves an HTTP API on your own computer, normally at port 11434. This Python script calls it with only the standard library:
import json
import urllib.request
payload = {
"model": "your-model-name", # a model you pulled, shown by `ollama list`
"prompt": "Explain what an API is in two sentences.",
"stream": False, # return one JSON object instead of a stream
}
request = urllib.request.Request(
"http://localhost:11434/api/generate",
data=json.dumps(payload).encode("utf-8"),
headers={"Content-Type": "application/json"},
)
with urllib.request.urlopen(request) as response:
print(json.load(response)["response"])Because it is a normal HTTP API, the same ideas apply as for any other (see What is an API). To compare local and hosted options, see ChatGPT vs Gemini vs Claude.
Common mistakes
Downloading a model too big for your memory, so it runs painfully slowly or fails.
Assuming "open" means any use is allowed. Read the licence.
Expecting a small local model to match the largest hosted models on hard tasks. Test it on your own task.
Exposing the local API to the internet without protection.
Key Takeaway
Open models let you run an LLM on your own machine, and Ollama makes that a few commands. Choose a model that fits your hardware, read its licence, and test it on your real task.
Ready to start? Learn how LLMs work and build AI features with them, including provider-neutral clients you can point at hosted or local models. Begin with the free course AI Engineering with LLMs: Beginner.
Related articles
More in LLM
What Are Top-k and Top-p in LLMs? How Do They Control AI Output?
Top-k and Top-p are sampling techniques that control which candidate tokens an LLM can choose from during generation. Understanding them is important for controlling creativity, diversity, and consistency in AI applications.
2 min readAug 30, 2026What Are Tokens and Tokenization in LLMs? Why Are They Important?
LLMs don't process text exactly like humans do. They process text as tokens. Understanding tokenization is essential for understanding context windows, API costs, model limits, and LLM performance.
2 min readAug 29, 2026What Is a Context Window in LLMs? Why It Matters
The context window is how many tokens an LLM can consider in one request: instructions, chat history, documents and its own answer. Learn why it matters and how to manage it.
3 min readAug 29, 2026Writing Your CV and Cover Letter With ChatGPT
How to use ChatGPT to improve your CV and draft a cover letter without inventing facts: working prompts, a safety checklist and common mistakes.
4 min readNov 16, 2026What Is AI Automation? Real Examples
AI automation puts a language model inside a workflow to handle messy, text-heavy steps. See real examples, a small code pattern and the risks to watch.
4 min readNov 14, 20267 AI Project Ideas for Your Portfolio
Seven small, finishable AI project ideas for a beginner portfolio, with the skill each one teaches and how to present it so others trust your work.
4 min readNov 14, 2026
Comments
Sign in to join the discussion.


