IBL News | New York
Several startups, collectives, and academics have released a wave of new large language models (LLMs) as open source, trying to challenge the closed, proprietary AI systems such as OpenAI and Anthropic.
These private organizations, knowing that state-of-the-art LLMs require huge compute budgets — OpenAI reportedly used 10,000 Nvidia GPUs to train ChatGPT — and deep ML expertise have refused to open up their models. They rely on API distribution instead.
The data, source code, or deep learning programming, of the model weights, remain hidden from public scrutiny.
Open-source initiatives state that they are seeking to democratize access to LLMs.
Two weeks ago, Databricks announced the ChatGPT-type Dolly, which was inspired by Alpaca, another open-source LLM released by Stanford in mid-March.
Stanford’s Alpaca used the weights from Meta’s LLaMA model that was released in late February.
LLaMA was hailed for its superior performance over models such as GPT–3, despite having ten times fewer parameters.
Other open-source LLaMA-inspired models have been released in recent weeks, such as:
– Vicuna, a fine-tuned version of LLaMA that apparently matches GPT-4 performance;
– Koala, a model from Berkeley AI Research Institute;
Some of these open-source models have even been optimized to run on the lowest-powered devices, from a MacBook Pro down to a Raspberry Pi and an old iPhone.
However, none of these open-source LLMs are available yet for commercial use, as the LLaMA model is not released for commercial use.
In March, the free-software community Mozilla announced an open-source initiative for developing AI, saying they “intend to create a decentralized AI community that can serve as a ‘counterweight’ against the large profit-focused companies.”
– Base models are sourced from @OpenAI, @EleutherAI, @Cerebras, @HuggingFace, and more!
– You get a 71K dataset of instructions to use for fine-tuning your own instruction-following LLM.… pic.twitter.com/ieTgbrAMP1
— DataChazGPT 🤯 (not a bot) (@DataChaz) April 29, 2023
🔥 If you read one thing about AI this week, let it be Google’s leaked doc
“We have no moat, and neither does OpenAI” pic.twitter.com/qwjLiCRYNW
— Jelle Prins (@jelleprins) May 4, 2023