OpenAI introduced GPT-4o mini on Thursday, July 18, a new, smaller AI model designed to be cheaper and faster than previous models.
Starting today, ChatGPT users on Free, Plus, and Team plans can use GPT-4o mini, with enterprise users gaining access next week. Developers can also access GPT-4o mini via the API, while GPT-3.5 Turbo will remain available for a limited time before it is eventually retired.
GPT-4o Mini’s Performance and Benchmark Scores
OpenAI claims that GPT-4o mini outperforms leading small AI models in reasoning tasks involving text and vision.
OpenAI did not disclose the exact size of GPT-4o mini but indicated that it is comparable to other small AI models like Llama 3 8b, Claude Haiku, and Gemini 1.5 Flash. However, OpenAI claims GPT-4o mini is faster, more cost-efficient, and smarter than these models. Before its launch, GPT-4o mini was tested on the LMSYS.org chatbot arena to evaluate its competitiveness.
According to data from Artificial Analysis, the model scored 82% on the Measuring Massive Multitask Language Understanding (MMLU) benchmark, which includes 16,000 multiple-choice questions across 57 academic subjects. This score is higher than Gemini 1.5 Flash’s 79% and Claude 3 Haiku’s 75%, though slightly lower than GPT-4o’s 88.7%.
| Model | MMLU Score (%) |
|---|---|
| GPT-4o Mini | 82 |
| Gemini 1.5 Flash | 79 |
| Claude 3 Haiku | 75 |
| GPT-4o | 88.7 |
Despite these scores, AI experts caution against relying solely on benchmarks like MMLU due to potential variations in administration and the risk of models being trained on the benchmark questions, as reported by The New York Times.
Growing Popularity of Smaller AI Models
Smaller AI models are becoming more popular among developers due to their speed and cost efficiencies compared to larger models like GPT-4 Omni or Claude 3.5 Sonnet. These models are useful for high-volume, simple tasks that developers frequently require.
OpenAI says GPT-4o mini is over 60% cheaper to run than GPT-3.5 Turbo. The new model currently supports text and vision in the API, with plans to include video and audio capabilities in the future.
Olivier Godemont, OpenAI’s Head of Product API told The Verge, “If we want AI to benefit every corner of the world, every industry, every application, we have to make AI much more affordable.” GPT-4o mini is priced at 15 cents per million input tokens and 60 cents per million output tokens, and it has a context window of 128,000 tokens and a knowledge cutoff of October 2023.
Practical Applications of GPT-4o Mini
The release of GPT-4o mini addresses the high costs associated with using larger models, making AI more accessible for developers.
For instance, Ramp, a financial technology startup, used GPT-4o mini to develop a tool that extracts expense data from receipts. Superhuman, an email client, employed the model to create an auto-suggestion feature for email responses.
OpenAI’s development of GPT-4o mini was driven by market demand and the need to create more affordable AI models. As smaller models like GPT-4o mini gain popularity, they provide a viable option for developers who need efficient performance without the high costs associated with larger models.
Featured Image courtesy of Gabby Jones/Bloomberg via Getty Images
Follow us for more updates on OpenAI and other tech news.