Mistral 7B (2023)

Mistral 7B is a 7-billion-parameter model that outperforms the 13B LLaMA 2 on all benchmarks and matches or beats the 34B LLaMA on many tasks. It proved that architectural innovations can deliver outsized performance gains at small scale.

Mistral 7B introduces two key architectural innovations: Grouped-Query Attention (GQA), which reduces memory usage and speeds up inference by sharing key-value heads across query heads, and Sliding Window Attention (SWA), which allows the model to process longer sequences efficiently by limiting attention to a local window while maintaining information flow through layers.

Combined with careful training and data curation, these techniques produce a 7B model that punches far above its weight class.

Mistral 7B proved that small, well-designed models can compete with models many times their size. This has huge practical implications: a 7B model can run on a single consumer GPU, respond faster, cost less to serve, and be deployed in more environments (including edge devices and laptops).

It inspired a wave of efficient model development and challenged the assumption that you always need bigger models for better performance. Mistral followed up with Mixtral (a mixture-of-experts model) and commercial offerings that continue to push the efficiency frontier.

Mistral AI, the company, became one of the most significant European AI startups largely on the strength of this paper.

Authors: Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, and 8 others (Mistral AI).

Key result: 7B parameters outperforming 13B LLaMA 2 across all tested benchmarks.

Link to paper: https://arxiv.org/abs/2310.06825

Full paper: https://arxiv.org/abs/2310.06825

Mistral AI official site - https://mistral.ai/

Hugging Face: Mistral models - https://huggingface.co/mistralai