Commentary and a selection of the most important recent news, articles, and papers about AI.
Today’s Brief Commentary
The first link today is to an update from The Futurum Group about the advanced semiconductor AI accelerator market. Everything else is about Generative AI.
It’s easy to say that GenAI is over-hyped (it is), dominates the AI news (it does), and that many people believe it will solve all known problems (it won’t). Once we cut through the noise, you can see that people are asking important questions about the state of the technology.
Will today’s GenAI paradigm still dominate in five years? Meta’s Yann LeCun doesn’t think so. I certainly agree. While there is a lot of good first-generation engineering in what we are doing now, the systems are too monolithic, the training methods too brute force, and the models too slow to get updated and improved in real-time.
Is GenAI a hammer looking for every nail it can hit? On some days, it certainly seems so. We need to ask the fundamental question: why are we using GenAI for a given application? Ge Wang asks this based on his personal experience and expertise in music and computer science.
If GenAI is a hammer, then GPUs are certainly hammers looking for AI tasks to compute. This is why there is such technical and investor interest in specialized accelerators such as those from Cerebras and the others covered in the Futurum report.
Finally, do we need all this energy-consuming computational power to get good results? DeepSeek is saying no, and people are listening. Maybe the US should not hand out those $500 billion quite yet until we know how to spend it wisely, though that is but one of a hundred questions and issues with the Stargate program.
Contents
Semiconductor Chipsets and Infrastructure
Data Center Spending on AI Processors and Accelerators Reached a Record $26.7B in 3Q 2024 | The Futurum Group
https://futurumgroup.com/press-release/data-center-spending-reached-a-record-26-7b-in-3q-2024/
Author: Richard Gordon
Date: Thursday, January 16, 2025
Excerpt: The data center industry has witnessed a period of exponential growth over the past decade, primarily driven by the burgeoning cloud computing sector. Leading cloud providers, including Amazon, Google, Meta, Microsoft, Apple, Alibaba, and ByteDance, have significantly increased capital expenditures, fueling the expansion of hyperscale data centers on a global scale. This substantial investment trajectory is poised to exert a significant positive influence on the market for AI processors and accelerators in the foreseeable future.
Generative AI and Models
Introducing Phi-4: Microsoft’s Newest Small Language Model Specializing in Complex Reasoning
Author: Ece Kamar
Date: Thursday, December 12, 2024
Commentary: Don’t ignore Microsoft‘s generational AI efforts that do not involve OpenAI‘s products and models. Also see the technical report on arXiv that I link to in the Research section below.
Excerpt: Phi-4 outperforms comparable and larger models on math related reasoning due to advancements throughout the processes, including the use of high-quality synthetic datasets, curation of high-quality organic data, and post-training innovations. Phi-4 continues to push the frontier of size vs quality.
Perplexity launches Sonar API, taking aim at Google and OpenAI with real-time AI search | VentureBeat
Author: Michael Nuñez
Date: Tuesday, January 21, 2025
Commentary: I like Perplexity – a lot. I’ve had much more success with it for gathering information than OpenAI ChatGPT and especially Google Gemini. Microsoft Copilot is a good second choice.
Excerpt: Perplexity’s Sonar is a real-time AI search API challenging Google and OpenAI with disruptive pricing and superior performance.
Nobody in their right mind will use genAI, LLMs in the next 5 years: Meta chief AI scientist Yann LeCun | India Today
Author: India Today Tech
Date: Friday, January 24, 2025
Commentary: To clarify, the title means no one will use GenAI in five years, not during the next five years.
Excerpt: Meta’s chief AI scientist predicts that in the next three to five years, we will enter the decade of robotics.
Ge Wang: GenAI Art Is the Least Imaginative Use of AI Imaginable
https://hai.stanford.edu/news/ge-wang-genai-art-least-imaginative-use-ai-imaginable
Date: Friday, January 24, 2025
Excerpt: The prevailing public mindset that AI is only a labor-saving tool betrays a lack of understanding of why people create and a lack of imagination of this technology’s potential.
How Chinese AI Startup DeepSeek Made a Model that Rivals OpenAI | WIRED
https://www.wired.com/story/deepseek-china-model-ai/
Author: Zeyi Yang
Date: Saturday, January 25, 2025
Excerpt: When Chinese quant hedge fund founder Liang Wenfeng went into AI research, he took 10,000 Nvidia chips and assembled a team of young, ambitious talent. Two years later, DeepSeek exploded on the scene.
Research and Technical
[2412.08905] Phi-4 Technical Report
https://arxiv.org/abs/2412.08905
Authors: Abdin, Marah; Aneja, Jyoti; Behl, Harkirat; Bubeck, Sébastien; Eldan, Ronen; Gunasekar, Suriya; Harrison, Michael; Hewett, Russell J.; Javaheripi, Mojan; ; …; and Zhang, Yi
Date: Thursday, December 12, 2024
Commentary: This is a an ICYMI item, which I did.
Excerpt: We present phi-4, a 14-billion parameter language model developed with a training recipe that is centrally focused on data quality. Unlike most language models, where pre-training is based primarily on organic data sources such as web content or code, phi-4 strategically incorporates synthetic data throughout the training process. While previous models in the Phi family largely distill the capabilities of a teacher model (specifically GPT-4), phi-4 substantially surpasses its teacher model on STEM-focused QA capabilities, giving evidence that our data-generation and post-training techniques go beyond distillation. Despite minimal changes to the phi-3 architecture, phi-4 achieves strong performance relative to its size – especially on reasoning-focused benchmarks – due to improved data, training curriculum, and innovations in the post-training scheme.
Disclosures
Bob Sutor is a former employee of IBM and Infleqtion and holds equity positions or stock options in each company. He is a Non-Executive Director for Nu Quantum.
