Blogs by Bud Ecosystem

Showing 56 results

Optimising Cost Efficiency in LLM Serving Using Heterogeneous Hardware Inferencing

Summary: The current industry practice of deploying GenAI-based solutions relies solely on high-end GPU infrastructure. However, several analyses have uncovered that this approach leads to resource wastage, as high-end GPUs are used for inference tasks that could be handled by a CPU or a commodity GPU at a much lower cost. Bud Runtime’s heterogeneous inference […]

Exploring Transformed Multi-Head Latent Attention for Cost-Effective Enterprise GenAI

Deepseek’s latest innovation, R1, marks a significant milestone in the GenAI market. The company has achieved performance comparable to OpenAI’s o1, yet claims to have done so at a much lower training cost—a major breakthrough for the industry. However, with 671 billion parameters, R1 remains too large for cost-effective enterprise deployment. While impressive, such massive […]

Efficient Expert Pruning for Sparse Mixture-of-Experts Language Models

As LLMs continue to grow, boasting billions to trillions of parameters, they offer unprecedented capabilities in natural language understanding and generation. However, their immense size also introduces major challenges related to memory usage, processing power, and energy consumption. To tackle these issues, researchers have turned to strategies like the Sparse Mixture-of-Experts (SMoE) architecture, which has […]

Fast yet Safe: Early-Exit Neural Networks with Risk Control for Optimal Performance

Large Language Models, with their increased parameter sizes, often achieve higher accuracy and better performance across a variety of tasks. However, this increased performance comes with a significant trade-off: inference, or the process of making predictions, becomes slower and more resource-intensive. For many practical applications, the time and computational resources required to get predictions from […]

LiveMind: Low-latency Large Language Models with Simultaneous Inference

In the rapidly evolving world of artificial intelligence, large language models (LLMs) are making headlines for their remarkable ability to understand and generate human-like text. These advanced models, built on sophisticated transformer architectures, have demonstrated extraordinary skills in tasks such as answering questions, drafting emails, and even composing essays. Their success is largely attributed to […]

Reducing LLM Ops Costs through Hybrid Inference with SLMs on Intel CPUs and Cloud LLMs

Despite the transformative potential of generative AI, its adoption in enterprises is lagging significantly. One major reason for this slow uptake is that many businesses are not seeing the expected ROI from their initiatives; in fact, recent research indicates that at least 30% of GenAI projects will be abandoned after proof of concept by the end of […]

Kangaroo: Lossless Self-Speculative Decoding via Double Early Exiting

In the research paper “Kangaroo: Lossless Self-Speculative Decoding via Double Early Exiting,” the authors introduce a new framework called Kangaroo designed to make large language models (LLMs) run faster. This framework enables the training of a smaller, lightweight model in a cost-effective way. This new framework is introduced to speedup the text generation process of […]

Bud Ecosystem