Researchers at Multiverse Computing have published a new approach to large language model compression that applies insights from statistical physics to the problem of removing transformer blocks. Rather than treating block removal as a ranking problem—a common approach in the field—they reformulate it as a constrained binary optimization problem that maps directly onto an Ising glass, a concept from physics describing disordered spin systems. This reframing accounts for the interdependencies between blocks, which traditional mean-field methods overlook.
The practical advantage lies in replacing expensive model evaluations with cheap energy calculations. The method requires computing a Hessian matrix just once from a small calibration dataset, then reuses it to evaluate billions of potential configurations without running the actual model. For Llama-3.3-70B-Instruct at 50% compression, the technique achieves nearly 23 percentage points higher MMLU scores than existing block-removal methods—a significant margin in a deep-compression regime where performance degradation is typically severe.
Beyond exact solving via GPU brute-force for tractable cases, the team leverages the Ising glass formulation to apply quantum-inspired solvers for larger configuration spaces. The approach stacks neatly with quantization and low-rank compression, positioning block removal as a foundational step in multi-stage model efficiency pipelines.
Key Points
Block removal reformulated as constrained binary optimization problem mapped to an Ising glass model, accounting for block interdependencies
Achieves 23-percentage-point MMLU improvement over competing approaches at 50% compression on Llama-3.3-70B-Instruct
Hessian-based energy calculations evaluate billions of configurations without running the model, dramatically reducing computational cost
Integrates quantum and quantum-inspired solvers for larger problem instances beyond classical brute-force limits