When OpenAI released gpt-oss, a pair of open-weight reasoning models, the move marked a clear signal to enterprises and developers. Instead of restricting access through cloud-only APIs, OpenAI made the weights of gpt-oss-20b and gpt-oss-120b available under the permissive Apache 2.0 license. This decision opens the door to secure, private deployments of advanced reasoning systems on infrastructure organizations control, while also allowing cost optimization at scale.
The models are designed for one primary purpose: enabling reliable reasoning in agents that can run privately, with cost levers that companies can adjust depending on hardware, workload, and accuracy requirements.
Inside the Models
gpt-oss comes in two configurations, 20 billion and 120 billion parameters. Both are built on a mixture-of-experts architecture with 32 experts, of which only two are active per token. This structure keeps the models efficient while maintaining reasoning quality.
The models use MXFP4 quantization, which reduces memory requirements significantly. The 20b checkpoint is about 12.8 GiB and can fit on consumer GPUs with 16 GB of memory. The 120b checkpoint is about 60.8 GiB and runs on 80 GB class GPUs. Both models have a context window of 131,072 tokens, thanks to YaRN extensions. This means they can process massive amounts of context in a single pass, a key feature for private retrieval-augmented generation (RAG) setups.
Why gpt-oss Matters
Most organizations are wary of sending sensitive data to external APIs. By hosting gpt-oss models on-premises or within their own VPCs, companies can build agents that keep prompts, retrieved documents, and logs within their own environment.
The Apache 2.0 license further reduces dependency on a single vendor. Organizations are free to choose their own infrastructure, optimize deployment, and scale based on cost targets. MXFP4 quantization means inference can run efficiently, even on consumer-grade hardware for lighter workloads.
In short, gpt-oss makes it possible to balance privacy, cost, and performance without trade-offs that usually come with closed API models.
How the Models Reason
Both models were trained to produce structured conversations in OpenAI’s “harmony” format. This format separates different parts of reasoning: chain of thought, tool calls, and final outputs. Harmony also allows developers to set reasoning levels to low, medium, or high. In practice, this lets organizations trade off speed and cost against reasoning depth.
The models are capable of using tools, such as search or Python execution, during their reasoning process. When a tool is called, the raw chain of thought must be passed back to the model on the next turn. For reliability, OpenAI recommends serving responses in the “Responses API” format, which packages reasoning, tool calls, and outputs together in a single envelope.
It is important to note that the raw chain of thought is not safety-aligned for end users. It should only be used internally for agent orchestration and never shown directly to customers.
Choosing Between 20b and 120b
A major decision in deploying gpt-oss is choosing the right model size.
When to choose gpt-oss-120b:
- Complex, multi-step reasoning tasks
- Workloads that require tool integration and long planning sequences
- Organizations with access to 80 GB GPUs that can prioritize accuracy and reasoning depth over raw concurrency
When to choose gpt-oss-20b:
- Lightweight classification or extraction tasks
- High-concurrency applications running on 16 GB GPUs or strong consumer devices
- First-pass triage or routing tasks where a larger model can be invoked only when necessary
OpenAI’s evaluations show that 120b is substantially stronger on challenging reasoning benchmarks, while 20b remains competitive on simpler workloads. The trade-off is clear: depth and precision versus speed and scale.
Deployment Options
OpenAI designed gpt-oss to be flexible in deployment. Options include:
- Self-hosted engines: vLLM, TensorRT-LLM on NVIDIA, llama.cpp, Ollama, and LM Studio
- Managed platforms: AWS, Azure, Databricks, Baseten, Vercel, Cloudflare, and OpenRouter
- Edge and device runtimes: Apple Silicon via MLX and Windows ONNX runtimes with the Windows AI Toolkit
When hosting the models yourself, it is crucial to format prompts correctly with the harmony renderer libraries in Python or Rust. If harmony formatting is ignored, model quality drops and tool use becomes unreliable.
Building Private RAG with gpt-oss
Private retrieval-augmented generation is one of the strongest use cases for these models. The large 131k token context allows entire batches of documents to be packed into a single prompt.
Patterns that work well include:
- Hosting your own embeddings, vector index, and inference pipeline inside a VPC for end-to-end privacy
- Structuring prompts into clear sections: instructions, query, retrieved passages, and schema for outputs
- Exposing private search as a tool, allowing the model to retrieve snippets as part of its reasoning process
- Forcing structured outputs, such as JSON or citation formats, to ensure traceability and reliability
This design ensures that no sensitive document leaves organizational boundaries while still allowing rich reasoning over proprietary data.
Cost and Operational Levers
Operating reasoning models can be expensive, but gpt-oss provides ways to optimize.
- Model size: Choose 20b when concurrency and cost are primary concerns, and 120b when reasoning quality is critical.
- Reasoning level: Lower reasoning levels reduce token generation length and latency.
- Engine choice: TensorRT-LLM yields maximum throughput on NVIDIA GPUs, while vLLM provides strong performance with easier setup.
- Response shaping: Summarize reasoning traces instead of returning full chains when tool calls are not required.
- Evaluation: Run structured evaluations on your own tasks to fine-tune reasoning depth and context usage before production rollout.
Safety and Governance
With open-weight reasoning models, responsibility for governance shifts to the operator. While the models come with safety measures for final outputs, the chain of thought is not aligned for public display. Enterprises need to ensure internal use only.
Because the weights are licensed under Apache 2.0, organizations can build commercial services without additional restrictions. Security best practices still apply: encrypt logs, isolate vector stores, and monitor tool outputs for consistency.
The Bottom Line
OpenAI’s gpt-oss models are not just another pair of open weights. They represent a turning point where advanced reasoning can be hosted privately and optimized for cost. With a choice between the efficient 20b and the powerful 120b, organizations can decide how to balance scale and depth.
When combined with private retrieval pipelines, structured harmony formatting, and agent orchestration through the Responses API, these models enable enterprises to run their own secure, reasoning-powered AI agents.
For companies that want to use the power of AI reasoning without sacrificing privacy or cost control, gpt-oss provides a foundation that is both flexible and future-ready.
Click here to read this article on Dave’s Demystify Data and AI LinkedIn newsletter.