Chinese AI startup DeepSeek has launched an impressive new ultra-large open-source AI model, DeepSeek-V3, which outperforms leading models including Meta’s Llama 3.1-405B and Qwen on various benchmarks. This innovative model features 671 billion parameters and leverages a mixture-of-experts architecture to optimize performance by activating only select parameters for given tasks. As a result, DeepSeek-V3 achieves an impressive balance of accuracy and efficiency, closely matching the performance of closed models from prominent players like Anthropic and OpenAI.
The launch of DeepSeek-V3 signifies another major step towards bridging the gap between open and closed-source AI, which could ultimately lead to advancements in artificial general intelligence (AGI). AGI aims to develop models capable of understanding and learning any intellectual task that a human can perform. DeepSeek emerged from Chinese quantitative hedge fund High-Flyer Capital Management and continues to push the boundaries of AI development.
Innovative Architecture and Key Features
Mirroring the basic architecture of its predecessor, DeepSeek-V2, the new model boasts multi-head latent attention (MLA) and the DeepSeekMoE system. This architecture promotes efficient training and inference, with specialized and shared experts within the larger model activating 37 billion parameters for each token out of 671 billion total parameters. These advancements enable the model to allocate its extensive computational resources effectively, ensuring optimal performance across a broad spectrum of tasks.
Notable advancements in DeepSeek-V3 include two groundbreaking innovations. The first innovation is an auxiliary loss-free load-balancing strategy, which dynamically monitors and adjusts the load on experts to ensure a balanced utilization without sacrificing overall model performance. The second innovation, multi-token prediction (MTP), allows the model to predict multiple future tokens simultaneously, significantly enhancing training efficiency and enabling the model to perform three times faster, generating 60 tokens per second. These innovations collectively help in optimizing the performance of DeepSeek-V3, making it a highly efficient and accurate AI model.
Training and Cost Efficiency
In a technical paper detailing the new model, DeepSeek revealed that they pre-trained DeepSeek-V3 on 14.8 trillion high-quality and diverse tokens. This rigorous pre-training phase ensures that the model has a comprehensive understanding of various language patterns and nuances. Subsequently, the company conducted a two-stage context length extension, initially extending to 32,000 tokens and then to 128,000. This was followed by post-training, which included supervised fine-tuning (SFT) and reinforcement learning (RL) on the base model of DeepSeek-V3 to align it with human preferences and unlock its full potential. Throughout this process, DeepSeek distilled reasoning capability from the DeepSeekR1 series of models while maintaining a balance between accuracy and generation length.
During the training phase, DeepSeek employed multiple hardware and algorithmic optimizations, such as the FP8 mixed precision training framework and the DualPipe algorithm for pipeline parallelism. These optimizations significantly reduced the costs associated with the training process. According to the company, the complete training of DeepSeek-V3 required approximately 2,788,000 H800 GPU hours, costing around $5.57 million with a rental price of $2 per GPU hour. This cost-efficiency stands in stark contrast to the hundreds of millions typically spent on pre-training large language models, such as Llama-3.1, which is estimated to have been trained with an investment exceeding $500 million. Such economic efficiency further showcases the innovation behind DeepSeek-V3’s development.
Benchmark Performance and Comparisons
Despite the economical training process, DeepSeek-V3 has emerged as the strongest open-source model currently available. The company conducted multiple benchmarks to evaluate the performance of DeepSeek-V3 compared to leading open models like Llama-3.1-405B and Qwen 2.5-72B. The results showed that DeepSeek-V3 convincingly outperformed these models and even surpassed the closed-source GPT-4 on most benchmarks, with exceptions in English-focused SimpleQA and FRAMES, where OpenAI’s model maintained higher scores. This places DeepSeek-V3 in an elite category of AI models capable of rivaling even the most advanced closed-source counterparts.
DeepSeek-V3 particularly excelled in Chinese and math-centric benchmarks. In the Math-500 test, for instance, it achieved a score of 90.2, while Qwen’s score stood at 80, the next best. The few instances in which DeepSeek-V3 was challenged were by Anthropic’s Claude 3.5 Sonnet, which outperformed it in benchmarks like MMLU-Pro, IF-Eval, GPQA-Diamond, SWE Verified, and Aider-Edit. These results underscore the model’s robustness and capability in handling complex tasks, especially in specific domains such as mathematics and Chinese language processing.
Implications for the AI Industry
DeepSeek’s new model, DeepSeek-V3, underwent rigorous pre-training on 14.8 trillion high-quality, diverse tokens. This extensive pre-training helps the model grasp a wide array of language patterns and intricacies. The development involved a two-stage context length expansion, initially stretching to 32,000 tokens and later to 128,000. Following this, post-training processes like supervised fine-tuning (SFT) and reinforcement learning (RL) were applied to synchronize the model with human preferences and fully optimize its capabilities. During this phase, the team distilled reasoning skills from their DeepSeekR1 models while keeping a balance between accuracy and length of generation.
To achieve efficient training, DeepSeek used advanced hardware and algorithmic optimizations, including the FP8 mixed precision training framework and the DualPipe algorithm for pipeline parallelism. These strategies markedly cut training costs. Training DeepSeek-V3 took approximately 2,788,000 H800 GPU hours, amounting to about $5.57 million at $2 per GPU hour. This is significantly more economical than the estimated $500 million often spent on pre-training large language models like Llama-3.1, highlighting DeepSeek-V3’s cost-effectiveness and innovative development.