GitHub

Sparsity-aware deep learning inference runtime for CPUs

We’re reaching out with heartfelt thanks and important news. Following Neural Magic’s acquisition by Red Hat in January 2025, we’ve shifted our focus to commercial and open-source offerings built ...
GitHub

LoRAX: Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs

LoRAX (LoRA eXchange) is a framework that allows users to serve thousands of fine-tuned models on a single GPU, dramatically reducing the cost of serving without compromising on throughput or latency.
IEEE

DeepSpeed- Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale

Abstract: The landscape of transformer model inference is increasingly diverse in model size, model characteristics, latency and throughput requirements, hardware requirements, etc. With such ...
IEEE

Measuring and Improving the Energy Efficiency of Large Language Models Inference

Abstract: Recent improvements in the accuracy of machine learning (ML) models in the language domain have propelled their use in a multitude of products and services, touching millions of lives daily.
BMJ

Causal inference is hard, and advanced statistical analysis is not enough

Health researchers need to fully understand the underlying assumptions to uncover cause and effect. Timothy Feeney and Paul Zivich explain Physicians ask, answer, and interpret myriad causal questions ...