MangoBoost Launches Mango LLMBoost™: AI Inference Optimization Software with Up to 12.6x Relative Performance Improvement and 92% Cost Savings

BELLEVUE, Wash.--(BUSINESS WIRE)--MangoBoost, a provider of cutting-edge system solutions designed to maximize AI data center efficiency, is announcing the launch of Mango LLMBoost™, system optimization software providing unparalleled performance improvement and efficiency for AI inference.

Through effective system schedule coordination, kernel optimization and its proprietary prefetching mechanism, reinforced by model quantization that fully takes advantage of modern GPUs, Mango LLMBoost™ achieves up to 12.6x boost in relative performance improvement and 92% cost savings compared to other popular LLM inference engines.

Mango LLMBoost™ is currently available through AWS Marketplace, with expansion to other major cloud service providers and support for on-prem deployments in the horizon.

Product Highlights:

• GPU Flexibility: Mango LLMBoost™ is compatible with all popular NVIDIA and AMD GPUs.
• Multi-Model Deployment and Management: Mango LLMBoost™ is validated across a diverse range of chat-based and multi-modal models, including Llama, Mixtral, Gemma, Qwen2, Llava, Phi3, Chameleon, MiniCPM, and GLM-v4, which can be deployed and managed on a single inference server with automated resource allocation.
• Hassle-Free Deployment: Mango LLMBoost™ provides end-to-end deployment option with MangoBoost’s web-serving and streaming APIs, and intelligently selects the best performing configuration given the GPU and the running models.
• OpenAI API Compatibility: Mango LLMBoost™ can be easily integrated into existing AI applications utilizing OpenAI’s API.

"The launch of Mango LLMBoost™ represents a significant step in MangoBoost's continued dedication to enhancing systems-level performance and efficiency. Our expertise in DPUs has been central to our mission of improving data center efficiency, and Mango LLMBoost™ expands that focus to deliver optimization at both hardware and software levels. By addressing the critical need for performance and efficiency in AI inference workloads, we’re enabling businesses to achieve more with their existing infrastructure," says Jangwoo Kim, CEO of MangoBoost.

For information on Mango LLMBoost™, visit the product webpage or reach out to [e-mail].

About MangoBoost

MangoBoost delivers cutting-edge, full-stack system solutions designed to maximize AI data center efficiency. At the heart of these solutions is the MangoBoost DPU, which ensures full compatibility with general-purpose GPUs, accelerators, and storage products, allowing for cost-efficient, standardized AI infrastructure. Founded in 2022 on a decade of research presented at top-tier computer systems conferences such as OSDI and ISCA, MangoBoost’s core DPU technology is protected by over 30 patents. The company has raised over $60 million in funding and is rapidly expanding its operations in the U.S., Canada, and Korea. MangoBoost’s team includes more than 100 experts, many of whom hold PhDs from world-class research institutions. For more information, visit MangoBoost’s website and LinkedIn page.