Petals

Petals is a decentralized platform that lets you run large language models like Llama 3.1 (up to 405B), Mixtral (8x22B), Falcon (40B+), or BLOOM (176B) on consumer-grade hardware by collaborating with others in a peer-to-peer network. Instead of loading the entire model into your GPU, you load just a part, while other parts are served by other participants. This BitTorrent-style approach enables single-batch inference at up to 6 tokens/second for Llama 2 70B and up to 4 tokens/second for Falcon 180B, making it suitable for chatbots and interactive applications. Petals is built for developers, researchers, and hobbyists who want the flexibility of running LLMs locally without needing expensive infrastructure. It supports fine-tuning, custom sampling methods, and access to hidden states, offering an API similar to classic LLM APIs but with the adaptability of PyTorch and Hugging Face Transformers. You can use it for research, experimentation, or building custom applications that require running LLMs outside centralized services. The platform operates over a network of volunteers who contribute their GPU resources. Users can contribute their own GPU to help others or just consume model capacity. Petals is free to use, with no pricing tiers or paid plans. It runs on a single machine or through Google Colab, with no need for cloud subscriptions or API keys. What makes Petals different is its decentralized architecture, which removes dependency on centralized cloud providers and allows unlimited experimentation. You can fine-tune models on private data, run custom inference pipelines, and see intermediate hidden states. The trade-off is variable latency and throughput depending on network availability, but it provides a unique option for privacy-conscious or cost-sensitive users.