Team: Zhihui Xie*, Jiacheng Ye*, Lin Zheng*, Jiahui Gao*, Jingwei Dong, Zirui Wu, Xueliang Zhao, Shansan Gong, Xin Jiang, Zhenguo Li†, and Lingpeng Kong†

Affiliations: The University of Hong Kong, Huawei Noah's Ark Lab

(*Equal contribution. †Core advising. )

Introducing Dream-Coder 7B

In a joint effort with Huawei Noah's Ark Lab, we release Dream-Coder 7B, the first fully open-source diffusion LLM for code that provides complete transparency throughout its development pipeline, with all components publicly available -- data processing scripts, implementation code, and model weights.

Text diffusion models represent a fundamental shift away from autoregressive LLMs. This emerging direction has attracted significant attention across academia and industry (Ye et al., 2025; Nie et al., 2025; Khanna et al., 2025; DeepMind, 2025), with startups like Mercury pioneering diffusion LLMs for code generation. Compared to autoregressive models, diffusion-based approaches offer greater generation diversity, improved robustness, and better capture of complex, multi-modal code structures. As a diffusion-based language model demonstrating competitive performance with autoregressive code LLMs at the same scale, Dream-Coder 7B Instruct achieves 21.4% pass@1 on LiveCodeBench (2410-2505), a remarkable result given that it was trained exclusively on publicly available datasets.

👨‍💻 Github, 🤗 HuggingFace

Instruct model performance comparison on coding benchmarks. We mark models trained on open-source data with ✓, and those trained on in-house data with ✗. The best results among open-weight diffusion language models are bolded.

Figure 1. Sketch-First Generation (from LiveCodeBench)

Figure 2. Left-to-Right Generation (from BigCodeBench)

Figure 3. Interleaved Reasoning Generation (from CRUXEval)

Figure 4. Variable-Length Code Infilling I

Figure 5. Variable-Length Code Infilling II

Features

Flexible Code Generation

We observe Dream-Coder 7B exhibits emergent any-order generation that adaptively determines its decoding style based on the coding task. For example, Dream-Coder 7B Instruct displays patterns such as:

• Sketch-first generation: For problems that read inputs from standard input and write outputs to standard output (Figure 1), Dream-Coder 7B Instruct begins by sketching the entire entry-point scaffold, then works backward to implement and refine helper functions and core logic.
• Left-to-right generation: For single-function completions (Figure 2), Dream-Coder 7B Instruct writes almost linearly—starting at the def header and moving left-to-right.
• Interleaved reasoning generation: For code reasoning tasks that require predicting output from code and input (Figure 3), Dream-Coder 7B Instruct first echoes the given input, then walks through the program step by step, jotting down each calculation and filling in output lines as soon as it figures them out.

These demos were collected using consistent sampling parameters: temperature=0.1, diffusion_steps=512, max_new_tokens=512, alg="entropy", top_p=1.0, alg_temp=0.0, and pad_penalty=3.0.

Variable-Length Code Infilling

One of the biggest challenges for diffusion LLMs is their lack of natural capability to generate variable-length sequences. This limitation is particularly problematic for infilling—generating code that seamlessly fits between existing snippets. We introduce an infilling variant, DreamOn-7B , that naturally adjusts the length of masked spans during generation by introducing two special tokens, <|expand|> and <|delete|>, which dynamically expand or contract the mask region to match the required infill length (Figure 4 and Figure 5). This capability allows the model to handle variable-length code infilling tasks more effectively, without prior knowledge of the target sequence length.

For more details, please refer to our accompanying blog post for our variable-length generation method DreamOn.