About

I am a 3rd-year Computer Science Ph.D. student at University of California, Santa Barbara, supervised by Prof. Zheng Zhang, and is expected to graduate in 2028.

Before joining UCSB, I received my B.S. and M.S. from EIC@Huazhong University of Science and Technology, worked with Prof. Jun Sun, Prof. Xinggang Wang and Prof. Yingzhuang Liu on computer vision and interpretability of deep learning.

Research Interests

My research interests focus on efficient pre-training for LLMs and multimodal Models, AI for science, computer vision, and the mathematical & physical principles behind them. I have recently focused on

  • Pre-training VLAs/WAMs.
  • Low-rank pre-training foundation models.
  • Muon Optimizer.

Away from the desk I shoot street and travel photographs; a selection lives in the gallery.

News

[08/2026] One of our recent works has been accepted by COLM 2026. Congratulations to all collaborators!

[01/2026] One of our recent works has been accepted by MLSys 2026. Congratulations to all collaborators!

[01/2026] One of our recent works has been accepted by ICLR 2026. Congratulations to all collaborators!

[09/2025] One of our recent works has been accepted by NeurIPS 2025. Congratulations to all collaborators!

[08/2025] One of our recent works has been accepted by EMNLP 2025 (oral). Congratulations to all collaborators!

[03/2025] Open-Sora 2.0 is released! Our fully open-source, commercial-level video generation model has now surpassed 29k stars on GitHub.

[09/2024] Joined Computer Science@UCSB Ph.D Program.

[09/2024] One of our recent works has been accepted by NeurIPS 2024. Congratulations to all collaborators!

Publications

  • COLM 2026 MuonQ: Enhancing Low-Bit Muon Quantization via Directional Fidelity Optimization
    Yupeng Su, Ruijie Zhang, Ziyue Liu, Yequan Zhao, Zheng Zhang
    Conference on Language Modeling (COLM 2026)
    pdf code 
    @article{su2026muonq,
title={MuonQ: Enhancing Low-Bit Muon Quantization via Directional Fidelity Optimization},
author={Su, Yupeng and Zhang, Ruijie and Liu, Ziyue and Zhao, Yequan and Zhang, Zheng},
journal={arXiv preprint arXiv:2605.11396},
year={2026}
}
    The Muon optimizer has emerged as a compelling alternative to Adam for training large language models, achieving remarkable computational savings through gradient orthogonalization. However, Muon's optimizer state is more sensitive to quantization errors: because the orthogonalization discards the magnitudes of singular values and retains only directional information, even small quantization errors in singular vector directions are amplified in the update. In this work, we propose MuonQ, a low-bit Muon training framework built on the principle of directional fidelity optimization. First, we apply a pre-quantization normalization so that each step introduces quantization errors of the same magnitude, preventing the accumulated error from developing a preferred direction. Second, we introduce a structural decomposition that separately quantizes the dominant singular components via power iteration, ensuring that quantization errors perturb only singular value magnitudes rather than rotating singular vector directions. Third, we adopt the mu-law companding quantization to allocate higher resolution to densely packed momentum values, shifting the quantization objective from outlier preservation to dense-region distinguishability. Together, these techniques enable stable 4-bit quantization of Muon's optimizer states. Pre-training experiments on GPT-style and LLaMA-style models demonstrate that MuonQ at 4-bit precision recovers most of full-precision Muon's training loss and downstream accuracy while reducing optimizer-state memory by up to 7.3x, offering a favorable point on the accuracy-memory trade-off.
  • MLsys 2026 BOOST: BOttleneck-Optimized Scalable Training Framework for Low-Rank Large Language Models
    Zhengyang Wang*, Ziyue Liu*, Ruijie Zhang, Avinash Maurya, Paul Hovland, Bogdan Nicolae, Franck Cappello, Zheng Zhang(* Equal contributions)
    Ninth Annual Conference on Machine Learning and Systems (MLSys 2026)
    pdf 
    @article{wang2025boost,
title={BOOST: BOttleneck-Optimized Scalable Training Framework for Low-Rank Large Language Models},
author={Wang, Zhengyang and Liu, Ziyue and Zhang, Ruijie and Maurya, Avinash and Hovland, Paul and Nicolae, Bogdan and Cappello, Franck and Zhang, Zheng},
journal={arXiv preprint arXiv:2512.12131},
year={2025}
}, 
}
    The scale of transformer model pre-training is constrained by the increasing computation and communication cost. Low-rank bottleneck architectures offer a promising solution to significantly reduce the training time and memory footprint with minimum impact on accuracy. Despite algorithmic efficiency, bottleneck architectures scale poorly under standard tensor parallelism. Simply applying 3D parallelism designed for full-rank methods leads to excessive communication and poor GPU utilization. To address this limitation, we propose BOOST, an efficient training framework tailored for large-scale low-rank bottleneck architectures. BOOST introduces a novel Bottleneck-aware Tensor Parallelism, and combines optimizations such as online-RMSNorm, linear layer grouping, and low-rank activation checkpointing to achieve end-to-end training speedup. Evaluations on different low-rank bottleneck architectures demonstrate that BOOST achieves 1.46-1.91× speedup over full-rank model baselines and 1.87-2.27× speedup over low-rank model with naively integrated 3D parallelism, with improved GPU utilization and reduced communication overhead.
  • ICLR 2026 Rényi Sharpness: A Novel Sharpness that Strongly Correlates with Generalization
    Qiaozhe Zhang, Jun Sun, Ruijie Zhang, Yingzhuang Liu
    The Fourteenth International Conference on Learning Representations (ICLR 2026)
    pdf code 
    @misc{zhang2025renyisharpnessnovelsharpness,
    title={R\'enyi Sharpness: A Novel Sharpness that Strongly Correlates with Generalization}, 
    author={Qiaozhe Zhang and Jun Sun and Ruijie Zhang and Yingzhuang Liu},
    year={2025},
    eprint={2510.07758},
    archivePrefix={arXiv},
    primaryClass={cs.LG},
    url={https://arxiv.org/abs/2510.07758}, 
}
    Sharpness (of the loss minima) is a common measure to investigate the generalization of neural networks. Intuitively speaking, the flatter the landscape near the minima is, the better generalization might be. Unfortunately, the correlation between many existing sharpness measures and the generalization is usually not strong, sometimes even weak. To close the gap between the intuition and the reality, we propose a novel sharpness measure, i.e., \textit{Rényi sharpness}, which is defined as the negative Rényi entropy (a generalization of the classical Shannon entropy) of the loss Hessian. The main ideas are as follows: 1) we realize that \textit{uniform} (identical) eigenvalues of the loss Hessian is most desirable (while keeping the sum constant) to achieve good generalization; 2) we employ the \textit{Rényi entropy} to concisely characterize the extent of the spread of the eigenvalues of loss Hessian. Normally, the larger the spread, the smaller the (Rényi) entropy. To rigorously establish the relationship between generalization and (Rényi) sharpness, we provide several generalization bounds in terms of Rényi sharpness, by taking advantage of the reparametrization invariance property of Rényi sharpness, as well as the trick of translating the data discrepancy to the weight perturbation. Furthermore, extensive experiments are conducted to verify the strong correlation (in specific, Kendall rank correlation) between the Rényi sharpness and generalization. Moreover, we propose to use a variant of Rényi Sharpness as regularizer during training, i.e., Rényi Sharpness Aware Minimization (RSAM), which turns out to outperform all existing sharpness-aware minimization methods. It is worthy noting that the test accuracy gain of our proposed RSAM method could be as high as nearly 2.5%, compared against the classical SAM method.
  • NeurIPS 2025 LaX: Boosting Low-Rank Training of Foundation Models via Latent Crossing
    Ruijie Zhang*, Ziyue Liu*, Zhengyang Wang, Zheng Zhang (* Equal contributions)
    The 39th Annual Conference on Neural Information Processing Systems (NeurIPS), 2025
    pdf code 
    @article{zhang2025lax,
title={LaX: Boosting Low-Rank Training of Foundation Models via Latent Crossing},
author={Zhang, Ruijie and Liu, Ziyue and Wang, Zhengyang and Zhang, Zheng},
journal={arXiv preprint arXiv:2505.21732},
year={2025}
}
    Training foundation models such as ViTs and LLMs requires tremendous computing cost. Low-rank matrix or tensor factorization offers a parameter-efficient alternative, but often downgrades performance due to the restricted parameter space. In this work, we introduce {\textbf{Latent Crossing (LaX)}} -- a simple yet effective plug-and-play module that enhances the capacity of low-rank models by enabling information flow across low-rank subspaces. We extensively validate the benefits of LaX on pre-training tasks with ViT-Base/Large and LLaMA-like models ranging from 60M to 1B parameters. LaX boosts low-rank model performance to match or exceed the full-rank baselines while using 2-3 fewer parameters. When equipped with low-rank adapters (i.e., LoRA) for fine-tuning LLaMA-7/13B, LaX consistently improves performance on arithmetic and common sense reasoning tasks with negligible cost.
  • EMNLP 2025 CoLA: Compute-Efficient Pre-Training of LLMs via Low-Rank Activation (oral)
    Ziyue Liu*, Ruijie Zhang*, Zhengyang Wang*, Mingsong Yan, Zi Yang, Paul Hovland, Bogdan Nicolae, Franck Cappello, Sui Tang, Zheng Zhang (* Equal contributions)
    The 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP 2025)
    pdf code 
    @article{liu2025cola,
title={Cola: Compute-efficient pre-training of llms via low-rank activation},
author={Liu, Ziyue and Zhang, Ruijie and Wang, Zhengyang and Yan, Mingsong and Yang, Zi and Hovland, Paul and Nicolae, Bogdan and Cappello, Franck and Tang, Sui and Zhang, Zheng},
journal={arXiv preprint arXiv:2502.10940},
year={2025}
}
    The full-size MLPs and the projection layers in attention introduce tremendous model sizes of large language models (LLMs), consuming extensive computational resources in pre-training. We empirically observe that the activations of pre-trained LLMs exhibit low-rank property. Motivated by such observations, we propose CoLA and its memory-efficient implementation, CoLA-M, to replace these full-size layers with compute-efficient auto-encoders that naturally enforce low-rank activations throughout training. This fundamental architectural change eliminates the activation redundancy and significantly boosts model capacity and training efficiency. Experiments on LLaMA models with 60 million to 7 billion parameters show that CoLA reduces the computing cost by 2×× and improves training throughput by 1.86×× while maintaining full-rank level performance. CoLA-M further squeezes memory cost without sacrificing throughput, offering a pre-training approach with collectively superior parameter, computing, and memory efficiency. The LLMs produced are also 2×× smaller, enabling faster inference with lower memory cost on resource-constrained platforms.
  • NeurIPS 2024 How Sparse Can We Prune A Deep Network: A Fundamental Limit Perspective
    Qiaozhe Zhang, Ruijie Zhang, Jun Sun, Yingzhuang Liu
    The Thirty-eighth Annual Conference on Neural Information Processing Systems (NeurIPS), 2024
    pdf code NeurIPS 
    @article{zhang2024sparse,
    title={How sparse can we prune a deep network: A fundamental limit perspective},
    author={Zhang, Qiaozhe and Zhang, Ruijie and Sun, Jun and Liu, Yingzhuang},
    journal={Advances in Neural Information Processing Systems},
    volume={37},
    pages={91337--91372},
    year={2024}
}
    Network pruning is a commonly used measure to alleviate the storage and computational burden of deep neural networks. However, the fundamental limit of network pruning is still lacking. To close the gap, in this work we'll take a first-principles approach, i.e. we'll directly impose the sparsity constraint on the loss function and leverage the framework of statistical dimension in convex geometry, thus enabling us to characterize the sharp phase transition point, which can be regarded as the fundamental limit of the pruning ratio. Through this limit, we're able to identify two key factors that determine the pruning ratio limit, namely, weight magnitude and network sharpness. Generally speaking, the flatter the loss landscape or the smaller the weight magnitude, the smaller pruning ratio. Moreover, we provide efficient countermeasures to address the challenges in the computation of the pruning limit, which mainly involves the accurate spectrum estimation of a large-scale and non-positive Hessian matrix. Moreover, through the lens of the pruning ratio threshold, we can also provide rigorous interpretations on several heuristics in existing pruning algorithms. Extensive experiments are performed which demonstrate that our theoretical pruning ratio threshold coincides very well with the experiments.

Preprints

  • Preprint Muon+: Towards Better Muon via One Additional Normalization Step
    Ruijie Zhang, Yequan Zhao, Ziyue Liu, Zhengyang Wang, Zheng Zhang
    arXiv 2026
    pdf code 
    @article{zhang2026muonplus,
title={Muon+: Towards Better Muon via One Additional Normalization Step},
author={Zhang, Ruijie and Zhao, Yequan and Liu, Ziyue and Wang, Zhengyang and Zhang, Zheng},
journal={arXiv preprint arXiv:2602.21545},
year={2026}
}
    The Muon optimizer has demonstrated promising performance in pre-training large language models through gradient (or momentum) orthogonalization. In this work, we propose a simple yet effective enhancement to Muon, namely Muon+, which introduces an additional normalization step after orthogonalization. We demonstrate the effectiveness of Muon+ through extensive pre-training experiments across a wide range of model scales and architectures. Our evaluation includes GPT-style models ranging from 130M to 774M parameters and LLaMA-style models ranging from 60M to 1B parameters. We comprehensively evaluate the effectiveness of Muon+ in the compute-optimal training regime and further extend the token-to-parameter (T2P) ratio to an industrial level of about 200. Experimental results show that Muon+ provides a consistent boost on training and validation perplexity over Muon.
  • Preprint TEON: Tensorized Orthonormalization Beyond Layer-Wise Muon for Large Language Model Pre-Training
    Ruijie Zhang, Yequan Zhao, Ziyue Liu, Zhengyang Wang, Dongyang Li, Yupeng Su, Sijia Liu, Zheng Zhang
    arXiv 2026
    pdf 
    @article{zhang2026teon,
title={TEON: Tensorized Orthonormalization Beyond Layer-Wise Muon for Large Language Model Pre-Training},
author={Zhang, Ruijie and Zhao, Yequan and Liu, Ziyue and Wang, Zhengyang and Li, Dongyang and Su, Yupeng and Liu, Sijia and Zhang, Zheng},
journal={arXiv preprint arXiv:2601.23261},
year={2026}
}
    The Muon optimizer has demonstrated strong empirical performance in pre-training large language models by performing matrix-level gradient (or momentum) orthogonalization in each layer independently. In this work, we propose TEON, a principled generalization of Muon that extends orthogonalization beyond individual layers by modeling the gradients of a neural network as a structured higher-order tensor. We present TEON's improved convergence guarantee over layer-wise Muon, and further develop a practical instantiation of TEON based on the theoretical analysis with corresponding ablation. We evaluate our approach on two widely adopted architectures: GPT-style models, ranging from 130M to 774M parameters, and LLaMA-style models, ranging from 60M to 1B parameters. Experimental results show that TEON consistently improves training and validation perplexity across model scales and exhibits strong robustness under various approximate SVD schemes.
  • Preprint Multi-level Multiple Instance Learning with Transformer for Whole Slide Image Classification
    Ruijie Zhang, Qiaozhe Zhang, Yingzhuang Liu, Hao Xin, Yan Liu, Xinggang Wang
    arXiv 2023
    pdf code 
    @article{zhang2023multi,
      title={Multi-level multiple instance learning with transformer for whole slide image classification},
      author={Zhang, Ruijie and Zhang, Qiaozhe and Liu, Yingzhuang and Xin, Hao and Liu, Yan and Wang, Xinggang},
      journal={arXiv preprint arXiv:2306.05029},
      year={2023}
}
    Whole slide image (WSI) refers to a type of high-resolution scanned tissue image, which is extensively employed in computer-assisted diagnosis (CAD). The extremely high resolution and limited availability of region-level annotations make employing deep learning methods for WSI-based digital diagnosis challenging. Recently integrating multiple instance learning (MIL) and Transformer for WSI analysis shows very promising results. However, designing effective Transformers for this weakly-supervised high-resolution image analysis is an underexplored yet important problem. In this paper, we propose a Multi-level MIL (MMIL) scheme by introducing a hierarchical structure to MIL, which enables efficient handling of MIL tasks involving a large number of instances. Based on MMIL, we instantiated MMIL-Transformer, an efficient Transformer model with windowed exact self-attention for large-scale MIL tasks. To validate its effectiveness, we conducted a set of experiments on WSI classification tasks, where MMIL-Transformer demonstrate superior performance compared to existing state-of-the-art methods, i.e., 96.80% test AUC and 97.67% test accuracy on the CAMELYON16 dataset, 99.04% test AUC and 94.37% test accuracy on the TCGA-NSCLC dataset, respectively.
  • Preprint GRZO: Group-Relative Zeroth-Order Optimization for Large Language Model Fine-Tuning
    Liyan Tan, Yequan Zhao, Yifan Yang, Ruijie Zhang, Xinling Yu, Zheng Zhang
    arXiv 2026
    pdf 
    @article{tan2026grzo,
title={GRZO: Group-Relative Zeroth-Order Optimization for Large Language Model Fine-Tuning},
author={Tan, Liyan and Zhao, Yequan and Yang, Yifan and Zhang, Ruijie and Yu, Xinling and Zhang, Zheng},
journal={arXiv preprint arXiv:2606.02857},
year={2026}
}
    Zeroth-order (ZO) optimization is a memory-efficient alternative to backpropagation for fine-tuning large language models, but its deployment is limited by the high variance of gradient estimation. We propose GRZO, a Group-Relative Zeroth-Order optimizer that draws one pseudo-independent perturbation per minibatch example and aggregates the per-example losses through group-relative normalization, raising the effective gradient-direction count from one to the batch size at no additional forward cost while preserving inference-level memory. We prove that GRZO is directionally unbiased with variance shrinking proportionally to the batch size, yielding a tighter nonconvex convergence bound than MeZO. Across RoBERTa-large, Llama3-8B, and OPT-13B over multiple tasks, GRZO improves average accuracy on Llama 3-8B by +3.0 over MeZO at 23% lower peak GPU memory; as a drop-in replacement for the MeZO core, it lifts sparse, low-rank, and quantized ZO variants by +6.0 on average.
  • Preprint FuRA: Full-Rank Parameter-Efficient Fine-Tuning with Spectral Preconditioning
    Yequan Zhao, Ruijie Zhang, Liyan Tan, Niall Moran, Tong Qin, Zheng Zhang
    arXiv 2026
    pdf code 
    @article{zhao2026fura,
title={FuRA: Full-Rank Parameter-Efficient Fine-Tuning with Spectral Preconditioning},
author={Zhao, Yequan and Zhang, Ruijie and Tan, Liyan and Moran, Niall and Qin, Tong and Zhang, Zheng},
journal={arXiv preprint arXiv:2605.22869},
year={2026}
}
    Both full fine-tuning (Full FT) and parameter-efficient methods like LoRA add weight updates without regard to the spectral structure that pretraining has established. This allows noisy gradients from a small fine-tuning distribution to freely perturb the robust features learned through pretraining. We first identify spectral preconditioning as the key missing ingredient: reparameterizing each weight W through its full-rank SVD and freezing one singular basis confines every update to the pretrained column space, yielding a preconditioned optimizer that outperforms unconstrained Full FT at the same parameter count. To make this insight practical, we propose FuRA (Full-Rank Adaptation), which factorizes W via a block tensor-train decomposition W=LSR: the large core L is frozen at the pretrained block-wise SVD basis while only the small core R and per-block singular values S are trained. This single design choice simultaneously delivers full-rank spectral preconditioning, full-rank update capacity, and parameter, step time, memory efficiency on par with LoRA. FuRA outperforms Full FT on LLM fine-tuning (+1.37 on LLaMA-3-8B commonsense reasoning), LLM math reinforcement learning, and VLM visual instruction tuning. The 4-bit quantized version QFuRA also outperforms QLoRA.
  • Preprint ReCoVer: Resilient LLM Pre-Training System via Fault-Tolerant Collective and Versatile Workload
    Ziyue Liu, Zhengyang Wang, Ruijie Zhang, Avinash Maurya, Hui Zhou, Paul Hovland, Sheng Di, Franck Cappello, Bogdan Nicolae, Zheng Zhang
    arXiv 2026
    pdf 
    @article{liu2026recover,
title={ReCoVer: Resilient LLM Pre-Training System via Fault-Tolerant Collective and Versatile Workload},
author={Liu, Ziyue and Wang, Zhengyang and Zhang, Ruijie and Maurya, Avinash and Zhou, Hui and Hovland, Paul and Di, Sheng and Cappello, Franck and Nicolae, Bogdan and Zhang, Zheng},
journal={arXiv preprint arXiv:2605.11215},
year={2026}
}
    Pre-training large language models on massive GPU clusters has made hardware faults routine rather than rare, driving the need for resilient training systems. Yet existing frameworks either focus on specific parallelism schemes or risk drifting away from a failure-free training trajectory. We propose ReCoVer, a resilient LLM pre-training system that upholds a single invariant: each iteration keeps the number of microbatches constant, ensuring per-iteration gradients remain stochastically equivalent to a failure-free run. The framework is organized as three decoupled protocol layers: (1) Fault-tolerant collectives that isolate faults from propagating across replicas; (2) in-step fine-grained recovery that preserves intra-iteration progress and prevents gradient corruption; (3) versatile-workload policy that dynamically redistributes microbatch quotas across the survivors. The design is parallelism-agnostic, integrating directly with both 3D parallelism and Hybrid Sharded Data Parallel (HSDP) as a drop-in substrate. We evaluate our implementation on end-to-end pre-training tasks for up to 512 GPUs, ReCoVer successfully preserves the training trajectory from a failure-free reference despite of 256 GPUs lost spread across the run. For comparison with checkpoint-and-restart baselines, ReCoVer demonstrates 2.23x higher effective throughput after successive failures. This advantage results in ReCoVer processing 74.9% more tokens at 234 GPU-hours, with the gap widening as the training prolongs.
  • Preprint Muon2: Boosting Muon via Adaptive Second-Moment Preconditioning
    Ziyue Liu, Ruijie Zhang, Zhengyang Wang, Yequan Zhao, Yupeng Su, Zi Yang, Zheng Zhang
    arXiv 2026
    pdf 
    @article{liu2026muon2,
title={Muon$^2$: Boosting Muon via Adaptive Second-Moment Preconditioning},
author={Liu, Ziyue and Zhang, Ruijie and Wang, Zhengyang and Zhao, Yequan and Su, Yupeng and Yang, Zi and Zhang, Zheng},
journal={arXiv preprint arXiv:2604.09967},
year={2026}
}
    Muon has emerged as a promising optimizer for large-scale foundation model pre-training by exploiting the matrix structure of neural network updates through iterative orthogonalization. However, its practical efficiency is limited by the need for multiple Newton-Schulz (NS) iterations per optimization step, which introduces non-trivial computation and communication overhead. We propose Muon^2, an extension of Muon that applies Adam-style adaptive second-moment preconditioning before orthogonalization. Our key insight is that the core challenge of polar approximation in Muon lies in the ill-conditioned momentum matrix, of which the spectrum is substantially improved by Muon^2, leading to faster convergence toward a practically sufficient orthogonalization. We further characterize the practical orthogonalization quality via directional alignment, under which Muon^2 demonstrates dramatic improvement over Muon at each polar step. Across GPT and LLaMA pre-training experiments from 60M to 1.3B parameters, Muon^2 consistently outperforms Muon and recent Muon variants while reducing NS iterations by 40%. We further introduce Muon^2-F, a memory-efficient factorized variant that preserves most of the gains of Muon^2 with negligible memory overhead.
  • Preprint Rényi Entropy: A New Token Pruning Metric for Vision Transformers
    Wei-Yuan Su, Ruijie Zhang†, Zheng Zhang († Project supervision)
    arXiv 2026
    pdf 
    @article{su2026renyientropy,
title={R\'enyi Entropy: A New Token Pruning Metric for Vision Transformers},
author={Su, Wei-Yuan and Zhang, Ruijie and Zhang, Zheng},
journal={arXiv preprint arXiv:2603.27900},
year={2026}
}
    Vision Transformers (ViTs) achieve state-of-the-art performance but suffer from the O(N^2) complexity of self-attention, making inference costly for high-resolution inputs. To address this bottleneck, token pruning has emerged as a critical technique to accelerate inference. Most existing methods rely on the [CLS] token to estimate patch importance. However, we argue that [CLS] token can be unreliable in early layers where semantic representations are still immature. As a result, pruning in the early layer often leads to inaccurate importance estimation and unnecessary information loss. In this work, we propose a training-free token importance metric, namely Col-Ln, which is derived from Renyi entropy that enables the identification of informative tokens from the first layer of the network, thereby enabling more reliable pruning in token reduction. Extensive experiments on ViTs and Large Vision-Language Models (LVLMs) demonstrate that our approach consistently outperforms state-of-the-art pruning methods across diverse benchmarks.
  • Preprint Open-Sora 2.0: Training a Commercial-Level Video Generation Model in $200k
    Zangwei Zheng, Xiangyu Peng, Yuxuan Lou, Chenhui Shen, Tom Young, …, Ruijie Zhang, …, Yang You
    arXiv 2025
    pdf code 
    @article{zheng2025opensora2,
title={Open-Sora 2.0: Training a Commercial-Level Video Generation Model in $200k},
author={Zheng, Zangwei and Peng, Xiangyu and Lou, Yuxuan and Shen, Chenhui and Young, Tom and Guo, Xinying and Wang, Binluo and Xu, Hang and Liu, Hongxin and Jiang, Mingyan and Li, Wenjun and Wang, Yuhui and Ye, Anbang and Ren, Gang and Ma, Qianran and Liang, Wanying and Lian, Xiang and Wu, Xiwen and Zhong, Yuting and Li, Zhuangyan and Gong, Chaoyu and Lei, Guojun and Cheng, Leijun and Zhang, Limin and Li, Minghao and Zhang, Ruijie and Hu, Silan and Huang, Shijie and Wang, Xiaokang and Zhao, Yuanheng and Wang, Yuqi and Wei, Ziang and You, Yang},
journal={arXiv preprint arXiv:2503.09642},
year={2025}
}
    Video generation models have achieved remarkable progress in the past year. The quality of AI video continues to improve, but at the cost of larger model size, increased data quantity, and greater demand for training compute. In this report, we present Open-Sora 2.0, a commercial-level video generation model trained for only $200k. With this model, we demonstrate that the cost of training a top-performing video generation model is highly controllable. We detail all techniques that contribute to this efficiency breakthrough, including data curation, model architecture, training strategy, and system optimization. According to human evaluation results and VBench scores, Open-Sora 2.0 is comparable to global leading video generation models including the open-source HunyuanVideo and the closed-source Runway Gen-3 Alpha. By making Open-Sora 2.0 fully open-source, we aim to democratize access to advanced video generation technology, fostering broader innovation and creativity in content creation. All resources are publicly available at: https://github.com/hpcaitech/Open-Sora.
  • Preprint A Range-Null Space Decomposition Approach for Fast and Flexible Spectral Compressive Imaging
    Junyu Wang, Shijie Wang, Ruijie Zhang, Zengqiang Zheng, Wenyu Liu, Xinggang Wang
    arXiv 2023
    pdf code 
    @article{wang2023rndsci,
title={A Range-Null Space Decomposition Approach for Fast and Flexible Spectral Compressive Imaging},
author={Wang, Junyu and Wang, Shijie and Zhang, Ruijie and Zheng, Zengqiang and Liu, Wenyu and Wang, Xinggang},
journal={arXiv preprint arXiv:2305.09746},
year={2023}
}
    We present RND-SCI, a novel framework for compressive hyperspectral image (HSI) reconstruction. Our framework decomposes the reconstructed object into range-space and null-space components, where the range-space part ensures the solution conforms to the compression process, and the null-space term introduces a deep HSI prior to constraining the output to have satisfactory properties. RND-SCI is not only simple in design with strong interpretability but also can be easily adapted to various HSI reconstruction networks, improving the quality of HSIs with minimal computational overhead. RND-SCI significantly boosts the performance of HSI reconstruction networks in retraining, fine-tuning or plugging into a pre-trained off-the-shelf model. Based on the framework and SAUNet, we design an extremely fast HSI reconstruction network, RND-SAUNet, which achieves an astounding 91 frames per second while maintaining superior reconstruction accuracy compared to other less time-consuming methods.