Top Open Source AI Models in 2025: LLaMA 3, Mistral, and Beyond

Open-weight AI models have closed much of the gap with proprietary APIs. Models you can download and run yourself now handle reasoning, coding, and multilingual tasks that required a frontier API not long ago — and the tooling to deploy them has gone from research code to one-line installs.

This guide covers the significant open models, how to deploy them, how to fine-tune them, and — importantly — when self-hosting is genuinely the wrong choice.

Why Self-Host at All?

The case for running your own models rests on four things, only one of which is cost.

1. Data privacy and compliance Your data never leaves your infrastructure. For GDPR, HIPAA, or SOC 2 scopes, removing a third-party processor from the diagram is often worth more than any performance consideration. There's no data processing agreement to negotiate and no vendor to audit.

2. Cost at sustained high volume Self-hosting trades a per-token cost for a fixed infrastructure cost. That's a good trade only above a utilization threshold — see the honest cost section below, because this is where most self-hosting business cases quietly fall apart.

3. Customization You can fine-tune on proprietary data, adjust behaviour beyond what prompting allows, and maintain task-specific variants. No content policy sits between you and the model.

4. Control and stability No rate limits. No deprecation notices retiring the model your prompts were tuned against. No provider changing the model underneath you between releases.

The Models

1. Meta LLaMA 3.1

Parameters: 8B, 70B, 405B License: Llama 3.1 Community License (not OSI open source — has an acceptable-use policy and a monthly-active-user threshold above which you need a separate license from Meta) Context: 128K tokens Released: July 2024

LLaMA 3.1 is the reference point for open-weight models. The 405B variant was the first open release positioned as broadly competitive with frontier proprietary models; the 70B and 8B variants are where most production deployments actually land.

Capabilities:

• Strong general reasoning and instruction following
• Official multilingual support for eight languages (English, German, French, Italian, Portuguese, Hindi, Spanish, Thai)
• 128K context, useful for document analysis and codebase work
• Well-supported by every inference framework worth using — often the safest default purely on ecosystem grounds

Hardware, roughly:

| Variant | Typical GPU requirement (FP16) | Notes | |---------|-------------------------------|-------| | 405B | 8x A100/H100 80GB | Multi-GPU only; substantial infrastructure | | 70B | 2x A100 80GB | The common production sweet spot | | 8B | 1x consumer GPU (e.g. RTX 4090) | Runs comfortably on a workstation |

Quantization changes this picture significantly. At 4-bit, a 70B model fits on a single A100 or a pair of consumer GPUs, and 8B runs on a well-specced laptop. Published evaluations of 4-bit quantization generally report modest quality degradation — but "modest" varies by task, and you should measure it on yours rather than trusting a general claim.

Deploying with vLLM:

from vllm import LLM, SamplingParams

llm = LLM(
    model="meta-llama/Meta-Llama-3.1-70B-Instruct",
    tensor_parallel_size=2,  # Split across 2 GPUs
    dtype="float16",
    max_model_len=8192,
    gpu_memory_utilization=0.95
)

sampling_params = SamplingParams(
    temperature=0.7,
    top_p=0.9,
    max_tokens=2048
)

prompts = [
    "Analyze this financial report and extract key risks:",
    "Summarize the following legal document:"
]

outputs = llm.generate(prompts, sampling_params)

for output in outputs:
    print(f"Response: {output.outputs[0].text}")

2. Mistral Large 2

Parameters: 123B License: Mistral Research License — research and non-commercial use only. Commercial deployment requires a separate commercial license from Mistral. Context: 128K tokens Released: July 2024

Mistral Large 2 has a strong reputation for coding and instruction following relative to its parameter count, and includes native function calling — which matters if you're building agents and don't want to hand-roll a tool-calling layer.

Function calling:

import requests

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get current weather for a location",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": "City name"
                    }
                },
                "required": ["location"]
            }
        }
    }
]

response = requests.post(
    "http://localhost:8000/v1/chat/completions",
    json={
        "model": "mistral-large-2",
        "messages": [
            {"role": "user", "content": "What's the weather in Paris?"}
        ],
        "tools": tools,
        "tool_choice": "auto"
    }
)

tool_call = response.json()["choices"][0]["message"]["tool_calls"][0]
print(f"Function: {tool_call['function']['name']}")
print(f"Args: {tool_call['function']['arguments']}")

3. Qwen 2.5

Parameters: 0.5B through 72B License: Apache 2.0 for most sizes — but not all. Some variants (including 72B) ship under the separate Qwen License. Check the specific model card. Context: 32K–128K depending on variant Released: September 2024

Alibaba's Qwen series is the strongest option for multilingual work, particularly Chinese, Japanese, and Korean, where Western-trained models tend to underperform on idiom and cultural context. It's also competitive on math and coding, and the range of sizes makes it easy to find one that fits your hardware.

The Qwen2-VL variants add vision capabilities if you need multimodal input.

4. Mixtral 8x22B

Parameters: ~141B total, ~39B active per token License: Apache 2.0 — genuinely open source, commercial use included Context: 64K tokens Released: April 2024

Mixtral uses a Mixture of Experts architecture: only a subset of the network activates for any given token.

Why MoE matters:

• Compute scales with active parameters (~39B), not total (~141B)
• Memory scales with total parameters — you must hold all of them in VRAM
• The result is inference speed closer to a 39B model with capacity closer to a much larger one

That memory/compute asymmetry is the crucial thing to understand about MoE. It's fast, but it is not cheap to host — you need VRAM for the whole model even though you only compute through a fraction of it.

5. Phi-3.5-mini

Parameters: 3.8B License: MIT — the most permissive on this list Context: 128K tokens Released: August 2024

Microsoft's Phi series makes the case that training data quality can substitute for scale. Phi-3.5-mini is trained on curated, textbook-style, and synthetic data rather than bulk web scraping, and punches well above its parameter count on reasoning benchmarks.

Why it's interesting: it runs on hardware that can't touch the other models here. Quantized to 4-bit it's small enough for a Raspberry Pi, a phone, or a modest laptop — which makes genuinely offline AI viable.

Edge deployment:

from llama_cpp import Llama

# Quantized model — small enough for CPU-only inference
llm = Llama(
    model_path="phi-3.5-mini-instruct.Q4_K_M.gguf",
    n_ctx=4096,
    n_threads=4,
    n_gpu_layers=0  # CPU only
)

response = llm(
    "Summarize the key points of this text:",
    max_tokens=512,
    temperature=0.3
)

print(response['choices'][0]['text'])

6. DeepSeek-V2

Parameters: 236B total, ~21B active per token License: MIT for the code; the model weights are under the separate DeepSeek Model License — check the terms for commercial use Context: 128K tokens Released: 2024

DeepSeek-V2's contribution is architectural. Multi-head Latent Attention (MLA) compresses the KV cache dramatically — the DeepSeek-V2 paper reports a reduction of over 90% versus standard attention.

That matters more than it sounds. The KV cache, not the weights, is usually what limits how many concurrent requests you can serve at long context. Shrinking it is what makes long-context serving affordable, and it's why this architecture influenced a lot of what came after.

Combined with a fine-grained MoE design activating only ~9% of parameters per token, DeepSeek-V2 is one of the more efficient architectures available in open weights.

On Benchmarks

You'll find tables comparing these models on MMLU, GSM8K, HumanEval, and MATH in a lot of articles, including earlier versions of this one. We've deliberately left them out, because they're less useful than they look:

• Published scores drift. Numbers get reported from different harnesses with different prompting and few-shot configurations, then copied between articles until the provenance is gone entirely.
• Contamination is a real and unsolved problem. Popular benchmarks leak into training data. A high MMLU score may partly measure memorisation.
• Benchmarks don't measure your task. A model that tops HumanEval may be worse than a smaller one at your specific extraction, classification, or summarization workload.

If you want current numbers, go to sources that show their methodology: the Open LLM Leaderboard, LMSYS Chatbot Arena for human preference data, and the model cards themselves.

Better still: build a small evaluation set from your own data — even 50 representative examples with known good answers will tell you more than any leaderboard.

Deployment Options

Option 1: Ollama (Easiest)

Best for: development, testing, personal use

# Install
curl -fsSL https://ollama.com/install.sh | sh

# Run a model — downloads on first use
ollama run llama3.1:70b
ollama run qwen2.5:72b

# HTTP API
curl http://localhost:11434/api/generate -d '{
  "model": "llama3.1:70b",
  "prompt": "Explain quantum computing"
}'

Pros: one-line install, automatic downloads, simple API, model management built in. Cons: not built for production throughput — limited batching and concurrency.

Option 2: vLLM (Production)

Best for: high-throughput serving

pip install vllm

# OpenAI-compatible server
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Meta-Llama-3.1-70B-Instruct \
  --tensor-parallel-size 2 \
  --dtype float16 \
  --max-model-len 8192 \
  --port 8000

It speaks the OpenAI API format, so existing client code often needs only a base URL change:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Meta-Llama-3.1-70B-Instruct",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

Tuning for throughput:

from vllm import LLM

llm = LLM(
    model="meta-llama/Meta-Llama-3.1-70B-Instruct",
    tensor_parallel_size=4,
    dtype="float16",
    max_model_len=16384,
    gpu_memory_utilization=0.95,
    max_num_batched_tokens=16384,
    max_num_seqs=256,           # Concurrent sequences
    enable_prefix_caching=True, # Reuses shared prompt prefixes
)

vLLM's core contribution is PagedAttention, which manages the KV cache like virtual memory rather than pre-allocating per sequence. Continuous batching means new requests join an in-flight batch instead of waiting for it to drain — which is where most of the throughput difference against a naive implementation comes from.

enable_prefix_caching is worth knowing about specifically: if every request shares a long system prompt, this caches its computation across requests.

Option 3: LM Studio (GUI)

Best for: non-technical users and experimentation

Download from lmstudio.ai, browse the model library, download with one click, chat immediately, and export a local API server when you want to build against it.

Option 4: Text Generation WebUI

Best for: power users, researchers, fine-tuners

git clone https://github.com/oobabooga/text-generation-webui
cd text-generation-webui
pip install -r requirements.txt
python server.py --model meta-llama/Llama-3.1-70B-Instruct

Multiple model loaders, an extensions system, training interfaces, and fine-grained sampling control.

Where to Run It

GPU pricing changes constantly and varies by region and availability, so specific rates go stale quickly. As of writing, the rough landscape:

RunPod — transparent per-hour pricing, instant deployment, wide GPU selection. A reasonable default for on-demand inference.

Vast.ai — a peer-to-peer marketplace, typically the cheapest option. The trade-off is variable availability and reliability, since you're renting someone else's hardware. Fine for batch jobs and experimentation; harder to justify for production SLAs.

Lambda Labs — on-demand GPUs, ML-optimized stack, no spot interruptions. Priced accordingly.

Hugging Face Inference Endpoints — managed, auto-scaling, no DevOps:

from huggingface_hub import create_inference_endpoint

endpoint = create_inference_endpoint(
    name="my-llama-endpoint",
    repository="meta-llama/Meta-Llama-3.1-70B-Instruct",
    framework="pytorch",
    task="text-generation",
    accelerator="gpu",
    instance_size="2xlarge",
    instance_type="nvidia-a100-80gb",
    region="us-east-1",
    vendor="aws",
    min_replica=0,  # Scale to zero when idle
    max_replica=10
)

min_replica=0 is the interesting parameter — scale-to-zero means you're not paying for an idle GPU, at the cost of cold starts. For bursty workloads that's often the difference between self-hosting making sense and not.

Always check current pricing directly with the provider. Also check GPU availability before you plan around it — the constraint is frequently supply, not price.

The Honest Cost Picture

This is where most self-hosting business cases go wrong, so it's worth being precise.

API pricing is per token. Self-hosting is per hour. That's the whole thing. An API bill scales with what you use and drops to zero when you don't. A GPU costs the same whether you send it a million requests or none.

The break-even therefore depends almost entirely on utilization, not volume:

• A GPU running at 80% utilization around the clock is usually far cheaper per token than an API.
• The same GPU serving bursty daytime traffic at 5% average utilization is frequently more expensive than the API — you're paying for idle silicon.

Costs that get left out of the comparison:

• Engineering time to deploy, monitor, and maintain the stack — usually the largest line item, and always the one that's omitted
• Redundancy: one GPU is a single point of failure, so a real deployment needs at least two
• Idle capacity provisioned for peak load
• Model upgrades — migrating to a new model is a project, not an API parameter change
• On-call burden for infrastructure a provider previously handled

Do the calculation like this:

1. Measure your actual token volume over a representative period — not your peak, your average
2. Price that volume at current API rates
3. Price the GPU capacity needed to serve your peak with acceptable latency, running 24/7
4. Add engineering time at a realistic rate
5. Compare — and be honest about the utilization number in step 3

Fine-Tuning

When It's Worth It

Fine-tune when:

• You have 1,000+ quality examples
• You need domain-specific vocabulary or a consistent output format
• Prompting has genuinely plateaued
• Brand voice or style matching is critical

Don't fine-tune when:

• Few-shot prompting works — try this first, always
• Retrieval (RAG) would solve it — fine-tuning teaches behaviour, not facts
• You have limited or low-quality training data
• Requirements change frequently — every change means retraining

The most common mistake is fine-tuning to inject knowledge. If you want the model to know your documentation, retrieve it into context. Fine-tuning is for teaching format, style, and task behaviour.

QLoRA

QLoRA loads the base model in 4-bit and trains small adapter matrices, which is what makes fine-tuning a 70B model possible without a research cluster:

from transformers import (
    AutoModelForCausalLM,
    AutoTokenizer,
    TrainingArguments
)
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
from trl import SFTTrainer
import torch

# Load in 4-bit
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Meta-Llama-3.1-70B-Instruct",
    load_in_4bit=True,
    device_map="auto",
    torch_dtype=torch.bfloat16,
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4"
)

model = prepare_model_for_kbit_training(model)

# Only the adapter weights train — the base model stays frozen
lora_config = LoraConfig(
    r=16,           # Rank — higher means more capacity and more memory
    lora_alpha=32,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM"
)

model = get_peft_model(model, lora_config)

training_args = TrainingArguments(
    output_dir="./llama-finetuned",
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    learning_rate=2e-4,
    num_train_epochs=3,
    logging_steps=10,
    save_steps=100,
    optim="paged_adamw_8bit"
)

trainer = SFTTrainer(
    model=model,
    args=training_args,
    train_dataset=dataset,
    dataset_text_field="text",
    max_seq_length=2048
)

trainer.train()

Because only adapters train, the output is a small file you can swap per task rather than a full model copy — which means one base model can serve many fine-tunes.

Tooling

Axolotl — configuration-driven fine-tuning:

base_model: meta-llama/Meta-Llama-3.1-70B-Instruct
model_type: LlamaForCausalLM
tokenizer_type: AutoTokenizer

load_in_4bit: true
adapter: qlora
lora_r: 16
lora_alpha: 32

datasets:
  - path: my_dataset.jsonl
    type: completion

num_epochs: 3
micro_batch_size: 2
gradient_accumulation_steps: 8

Unsloth — optimized kernels; markedly faster training and lower memory than a stock Transformers loop.

LLaMA Factory — web UI for training, with dataset management and experiment tracking.

Troubleshooting

Out of Memory

CUDA out of memory. Tried to allocate 4.50 GiB

# 1. Reduce context length — KV cache scales with it
llm = LLM(
    model="meta-llama/Meta-Llama-3.1-70B-Instruct",
    max_model_len=4096,
    gpu_memory_utilization=0.90
)

# 2. Quantize
llm = LLM(
    model="meta-llama/Meta-Llama-3.1-70B-Instruct",
    quantization="awq",  # or "gptq"
    dtype="float16"
)

# 3. Offload to CPU (slow — a last resort)
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Meta-Llama-3.1-70B-Instruct",
    device_map="auto",
    offload_folder="offload"
)

Reducing max_model_len is usually the first thing to try. The KV cache grows with context length and concurrency, and it's typically what actually exhausted your VRAM — not the weights.

Slow Inference

# Use all available GPUs
llm = LLM(
    model="meta-llama/Meta-Llama-3.1-70B-Instruct",
    tensor_parallel_size=4,
)

# Flash Attention 2
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Meta-Llama-3.1-70B-Instruct",
    attn_implementation="flash_attention_2"
)

If throughput is the problem, check batching before anything else. A server processing one request at a time leaves most of the GPU idle — the fix is continuous batching, not a bigger GPU.

Poor Quality Output

sampling_params = SamplingParams(
    temperature=0.3,        # Lower for focused, deterministic output
    top_p=0.9,
    top_k=50,
    repetition_penalty=1.1
)

Before tuning sampling, check two things:

1. Are you using the -Instruct variant? Base models are not instruction-tuned and will happily continue your prompt as text rather than answering it. This is the single most common cause of "the model is broken."
2. Are you applying the correct chat template? Each model family has its own. Getting it wrong degrades quality in ways that look like a bad model.

Best Practices

Choosing a Model

• Maximum capability: LLaMA 3.1 405B — if you can afford the hardware
• Best all-round balance: LLaMA 3.1 70B
• Efficiency: Mixtral 8x22B or DeepSeek-V2
• Multilingual, especially Asian languages: Qwen 2.5
• Edge and offline: Phi-3.5
• Getting started cheaply: LLaMA 3.1 8B or Qwen 2.5 7B
• Unambiguous commercial licensing: Mixtral 8x22B (Apache 2.0) or Phi-3.5 (MIT)

That last line deserves more weight than it usually gets. A model that's marginally better but needs a commercial license negotiation can be the worse engineering choice.

Quantization

Lower precision trades quality for memory. The commonly reported pattern is that 8-bit is nearly lossless, 4-bit costs a little quality for a large memory saving, and below 4-bit degradation becomes noticeable. 4-bit (GPTQ or AWQ) is the usual production default.

The important caveat: quality loss from quantization is task-dependent. Published averages hide the fact that reasoning and code generation tend to degrade more than summarization or classification. Measure on your task.

Monitoring

from prometheus_client import Counter, Histogram
import logging

request_counter = Counter('llm_requests_total', 'Total requests')
latency_histogram = Histogram('llm_latency_seconds', 'Request latency')

@latency_histogram.time()
def generate_response(prompt):
    request_counter.inc()
    response = llm.generate(prompt)
    logging.info(f"Generated {len(response)} tokens")
    return response

Track time-to-first-token separately from total latency — they're different user experiences and different bottlenecks. Also track GPU utilization: it's the number that tells you whether your cost model is working.

Security

• Rate limiting — inference is expensive; an unmetered endpoint is a bill waiting to happen
• Input validation — cap prompt length; long inputs are a denial-of-service vector
• Output filtering — check for sensitive data in responses, particularly with fine-tuned models
• Access control — authenticate every request
• Prompt injection awareness — if the model sees untrusted input and has tool access, that input can direct its behaviour

Where Things Are Heading

Mixture of Experts is becoming standard. Mixtral and DeepSeek showed that sparse activation delivers large-model capacity at mid-size compute. Most significant new architectures use some form of it.

Multimodal is arriving in open weights. Qwen2-VL, Phi-3.5-Vision, and the LLaMA 3.2 vision variants brought image understanding to models you can self-host.

Quantization keeps improving. Each generation of techniques pushes usable quality further down the bit-width curve, which steadily lowers the hardware bar.

Domain-specific models are proliferating. Code, math, biomedical, and legal specialists increasingly outperform general models of similar size within their domain — while being far cheaper to run than a frontier generalist.

Tooling is maturing. vLLM, Ollama, and the deployment platform ecosystem have turned self-hosting from a research project into an afternoon. This is arguably the biggest change of all: the models were usable before the tooling was.

We're deliberately not predicting specific future releases or dates. That's the part of these articles that ages worst, and there's enough speculation about it elsewhere.

Resources

Model discovery

• Hugging Face Models — model cards, licenses, and weights
• Open LLM Leaderboard — reproducible benchmark methodology
• LMSYS Chatbot Arena — human preference rankings

Deployment

• vLLM — production inference
• Ollama — local development
• llama.cpp — CPU and edge inference
• LM Studio — GUI

Fine-tuning

• PEFT — parameter-efficient methods
• Axolotl — config-driven training
• Unsloth — optimized training

Our Take

The pragmatic path: start with an API and build the thing. Measure your real token volume and utilization. If your data can't leave your infrastructure, self-host from day one and accept the engineering cost as the price of the requirement. If it's a cost question, wait until you have a bill and a utilization number to reason about — then run LLaMA 3.1 8B or Qwen 2.5 7B locally to learn the stack before committing to production.

And read the license before you build. Several of the best models here aren't open source in the sense the word usually implies, and finding that out after you've shipped is a much worse conversation than finding out now.

For AI tooling on the development side rather than the infrastructure side, see our guide to AI coding assistants.