New Delhi: Elon Musk has revealed some details about the training of the next version of his AI chatbot, Grok, on X/Twitter. Responding to some posts, Musk talked about the problem of “models training on each other’s data,” comparing it to a “human centipede effect,” and admitted that a lot of work is needed to clean up large language models (LLMs) because most of the training data comes from the internet.
The governor is correct that this is a Dojo Supercomputer, but $500M, while obviously a large sum of money, is only equivalent to a 10k H100 system from Nvidia.
— Elon Musk (@elonmusk) January 26, 2024
Tesla will spend more than that on Nvidia hardware this year. The table stakes for being competitive in AI are at…
Musk promised that Grok 2, set to launch in August 2024, will be a big improvement, acknowledging the current flaws of the AI. He has previously claimed that Grok competes with AI models from companies like OpenAI, Meta, and Google. Looking ahead, Musk mentioned that Grok 3 would be “something special,” and it will be trained using 100,000 Nvidia H100 chips. He had earlier redirected a shipment of Nvidia chips intended for Tesla to support his ventures, X and xAI.
In an earlier tweet, Musk said that the xAI headquarters would soon have a 100,000 H100 liquid-cooled training cluster up and running. He had also previously stated in August 2023 that H100 chips are better than A100 chips. However, it is still unclear if Musk will go ahead with plans to buy chips from AMD.
