PyData Global 2023

With the release of Llama 2, the performance of open source models are closing the gap to the level of GPT-4. Remarkably, for tasks that require low level of reasoning, fine-tuned LLMs with smaller parameter sizes can significantly out perform their proprietary counterparts. In addition, running your own LLMs guarantees data privacy knowing that your sensitive data will never be leaked or used in training. Lastly, since you only pay for compute resources as opposed to tokens, a better economy of scale can be achieved by leveraging smaller models and efficient scaling of the underlying resources.

However, running your own open source LLMs comes with many challenges. We will deep dive into five categories:

• Operability: Can I deploy and serve the model reliably on hardware I have available?
• Scalability: Can I scale serving instances elastically while maintain high availability?
• Throughput: Can I respond to a large number of concurrent text generation efficiently?
• Latency: Can I respond with a reasonable delay to both APIs and human users?
• Cost: How much do I need to spend to run my open source LLMs?

With these challenges in mind, we want to help AI developers more efficiently deploy and operate their own LLMs. We started the open source project, OpenLLM, that aims to solve the above challenges by providing users the options to optimize LLM serving through techniques like quantization, continuous batching, model parallelism, kernel optimization, and token streaming. We aim to package all state of the art LLM serving optimizations in a single easy to use Python package. In a domain that is both fast-evolving and densely researched like LLM inference, OpenLLM facilitates AI developers to center their attention on core tasks without the need to stay abreast and integrate the latest optimizations continuously.