DeepSeek-V2 Unpacked

Ben Lorica

2 years ago

In the same week that China’s DeepSeek-V2, a powerful open language model, was released, some US tech leaders continue to underestimate China’s progress in AI. Former Google CEO Eric Schmidt opined that the US is “way ahead of China” in AI, citing factors such as chip shortages, less Chinese training material, reduced funding, and a focus on the wrong areas. However, the release of DeepSeek-V2 showcases China’s advancements in large language models and foundation models, challenging the notion that the US maintains a significant lead in this field.

What is DeepSeek-V2 and why is it significant?

DeepSeek-V2 is a strong, open-source Mixture-of-Experts (MoE) language model that stands out for its economical training, efficient inference, and top-tier performance across various benchmarks. The model comprises 236 billion total parameters, with only 21 billion activated for each token, and supports an extended context length of 128K tokens. The significance of DeepSeek-V2 lies in its ability to deliver strong performance while being cost-effective and efficient.

What are the key features and capabilities of DeepSeek-V2?

Large MoE Language Model with Parameter Efficiency: DeepSeek-V2 has a total of 236 billion parameters, but only activates 21 billion parameters for each token. This allows for more efficient computation while maintaining high performance, demonstrated through top-tier results on various benchmarks.

Innovative Architectures for Efficient Training and Inference:

Multi-Head Latent Attention (MLA): This novel attention mechanism compresses the Key-Value (KV) cache into a latent vector, which significantly reduces the size of the KV cache during inference, improving efficiency.
Mixture-of-Expert (MoE) Architecture (DeepSeekMoE): This architecture facilitates training powerful models economically. It uses fine-grained expert segmentation and shared expert isolation to achieve high expert specialization and reduce knowledge redundancy, respectively.

Economical Training and Efficient Inference: Compared to its predecessor, DeepSeek-V2 reduces training costs by 42.5%, reduces the KV cache size by 93.3%, and increases maximum generation throughput by 5.76 times.

Extended Context Length Support: It supports a context length of up to 128,000 tokens, enabling it to handle long-term dependencies more effectively than many other models.

Advanced Pre-training and Fine-Tuning: DeepSeek-V2 was pre-trained on a high-quality, multi-source corpus of 8.1 trillion tokens, and it underwent Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) to enhance its alignment with human preferences and performance on specific tasks.

Robust Evaluation Across Languages: It was evaluated on benchmarks in both English and Chinese, indicating its versatility and robust multilingual capabilities.

Strong Performance: DeepSeek-V2 achieves top-tier performance among open-source models and becomes the strongest open-source MoE language model, outperforming its predecessor DeepSeek 67B while saving on training costs.

Alignment with Human Preferences: DeepSeek-V2 is aligned with human preferences using online Reinforcement Learning (RL) framework, which significantly outperforms the offline approach, and Supervised Fine-Tuning (SFT), achieving top-tier performance on open-ended conversation benchmarks.

How does DeepSeek-V2 compare to its predecessor and other competing models?

Comparison with Other Models:

Qwen1.5 72B: DeepSeek-V2 demonstrates overwhelming advantages on most English, code, and math benchmarks, and is comparable or better on Chinese benchmarks.
Mixtral 8x22B: DeepSeek-V2 achieves comparable or better English performance, except for a few specific benchmarks, and outperforms Mixtral 8x22B on MMLU and Chinese benchmarks.
LLaMA3 70B: Despite being trained on fewer English tokens, DeepSeek-V2 exhibits a slight gap in basic English capabilities but demonstrates comparable code and math capabilities, and significantly better performance on Chinese benchmarks.
Chat Models: DeepSeek-V2 Chat (SFT) and (RL) surpass Qwen1.5 72B Chat on most English, math, and code benchmarks. They also exhibit competitive performance against LLaMA3 70B Instruct and Mistral 8x22B Instruct in these areas, while outperforming them on Chinese benchmarks.

Comparison with the previous version of DeepSeek:

Performance: DeepSeek-V2 outperforms DeepSeek 67B on almost all benchmarks, achieving stronger performance while saving on training costs, reducing the KV cache, and increasing the maximum generation throughput.
Economical Training: Training DeepSeek-V2 costs 42.5% less than training DeepSeek 67B, attributed to its innovative architecture that includes a sparse activation approach, reducing the total computational demand during training.
Efficient Inference: DeepSeek-V2 reduces the Key-Value (KV) cache by 93.3%, enhancing inference efficiency. This is achieved through the introduction of Multi-head Latent Attention (MLA), which compresses the KV cache significantly. The maximum generation throughput of DeepSeek-V2 is 5.76 times that of DeepSeek 67B, demonstrating its superior capability to handle larger volumes of data more efficiently.
Architectural Innovations: DeepSeek-V2 incorporates novel architectural features like MLA for attention and DeepSeekMoE for handling Feed-Forward Networks (FFNs), both of which contribute to its improved efficiency and effectiveness in training strong models at lower costs.
Data and Pre-training: DeepSeek-V2 is pretrained on a more diverse and larger corpus (8.1 trillion tokens) compared to DeepSeek 67B, enhancing its robustness and accuracy across various domains, including extended support for Chinese language data.
Fine-Tuning and Reinforcement Learning: The model further undergoes Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) to tailor its responses more closely to human preferences, enhancing its performance particularly in conversational AI applications.

Overall, DeepSeek-V2 demonstrates superior or comparable performance compared to other open-source models, making it a leading model in the open-source landscape, even with only 21B activated parameters. It becomes the strongest open-source MoE language model, showcasing top-tier performance among open-source models, particularly in the realms of economical training, efficient inference, and performance scalability.

What makes DeepSeek-V2 an “open model”?

DeepSeek-V2 is considered an “open model” because its model checkpoints, code repository, and other resources are freely accessible and available for public use, research, and further development. Furthermore, the code repository for DeepSeek-V2 is licensed under the MIT License, which is a permissive open-source license. This means that the model’s code and architecture are publicly available, and anyone can use, modify, and distribute them freely, subject to the terms of the MIT License.

How can teams leverage DeepSeek-V2 for building applications and solutions?

Teams can leverage DeepSeek-V2 for building applications and solutions in several ways:

DeepSeek’s Official Chat Website: Teams can easily explore and test DeepSeek-V2’s capabilities by interacting with the model directly on DeepSeek’s official website, chat.deepseek.com. This provides a readily available interface without requiring any setup, making it ideal for initial testing and exploration of the model’s potential.
OpenAI-Compatible API: DeepSeek Platform offers an OpenAI-Compatible API at platform.deepseek.com. This API allows teams to seamlessly integrate DeepSeek-V2 into their existing applications, especially those already utilizing OpenAI’s API. The platform provides millions of free tokens and a pay-as-you-go option at a competitive price, making it accessible and budget-friendly for teams of various sizes and needs.
Local Inference: For teams with more technical expertise and resources, running DeepSeek-V2 locally for inference is an option. This requires 80GB*8 GPUs to handle the model in BF16 format. Local deployment offers greater control and customization over the model and its integration into the team’s specific applications and solutions.
Hugging Face Transformers: Teams can directly employ Hugging Face Transformers for model inference. This widely-used library provides a convenient and familiar interface for interacting with DeepSeek-V2, enabling teams to leverage their existing knowledge and experience with Hugging Face Transformers.
LangChain Integration: Due to DeepSeek-V2’s compatibility with OpenAI, teams can easily integrate the model with LangChain. LangChain is a popular framework for building applications powered by language models, and DeepSeek-V2’s compatibility ensures a smooth integration process, allowing teams to develop more sophisticated language-based applications and solutions.

What are some early reactions from developers?

DeepSeek-V2’s Coding Capabilities: Users report positive experiences with DeepSeek-V2’s code generation abilities, particularly for Python. The model demonstrates strong zero-shot generation of complete, functional programs for games (Snake, chase game) and a basic MP3 player UI.
Cost Efficiency and Affordability: DeepSeek-V2 offers significant cost reductions compared to previous models and competitors like OpenAI. The API’s low cost is a major point of discussion, making it a compelling alternative for various projects. Cost efficiency is crucial for AI teams, especially startups and those with budget constraints, as it allows more room for experimentation and scaling. Affordable API access enables wider adoption and deployment of AI solutions.
Efficient Inference and Accessibility: DeepSeek-V2’s MoE architecture enables efficient CPU inference with only 21B parameters active per token, making it feasible to run on consumer CPUs with sufficient RAM. This accessibility expands the potential user base for the model. Efficiency in inference is vital for AI applications as it impacts real-time performance and responsiveness. The ability to run large models on more readily available hardware makes DeepSeek-V2 an attractive option for teams without extensive GPU resources.
Performance Improvements: DeepSeek-V2 achieves stronger performance metrics than its predecessors, notably with a reduced number of activated parameters per token, enhancing its efficiency. The model scores 80 on the HumanEval benchmark, signifying its strong coding abilities. This is critical for AI applications that require robust and accurate language processing capabilities. The HumanEval score offers concrete evidence of the model’s coding prowess, giving teams confidence in its ability to handle complex programming tasks.
Lack of Transparency Regarding Training Data and Bias Mitigation: The paper lacks detailed information about the training data used for DeepSeek-V2 and the extent of bias mitigation efforts. Transparency about training data and bias mitigation is crucial for building trust and understanding potential limitations. Lack of information can hinder ethical considerations and responsible AI development.
Censorship and Alignment with Socialist Values: DeepSeek-V2’s system prompt reveals an alignment with “socialist core values,” leading to discussions about censorship and potential biases. The model tends to self-censor when responding to prompts related to sensitive topics concerning China. Teams need to be aware of potential censorship and biases ingrained in the model’s training data. This is crucial for applications requiring neutrality and unbiased information.