How GenAI is making your smartphone more intelligent

With generative AI, which can create new context with the help of prompts in natural languages like English or Hindi, smartphones will only get smarter. For instance, when Samsung Galaxy S24 launches on 17 January, Canalys expects on-device AI features like Live-Transcribe, but adds that most new AI features and solutions will be rolled out in the future using software updates.

According to research firm Counterpoint, too, this year will be pivotal for Generative AI (GenAI) smartphones with preliminary data projecting their shipments to reach over 100 million units in 2024. By 2027, Counterpoint expects GenAI smartphone shipments to reach 522 million units, growing at a compounded annual growth rate (CAGR) of 83%. Effectively, over 1 billion GenAI smartphones are likely to be shipped cumulatively between CY2024 and CY2027.

The fact is that chipset makers including Qualcomm, Mediatek, Samsung and Google have been focusing on improving neural processing unit and tensor processing unit (NPU/TPU) performance over the past few years. Both NPUs and TPUs are broadly classified as AI accelerators. TPU, on their part, are Google’s custom-developed application-specific integrated circuits (ASICs) used to accelerate machine learning (ML) workloads.

Likewise, NPUs are specifically focused on handling AI/ML workloads. Smartphone vendors including Apple, Huawei, vivo, and Xiaomi, too, are implementing AI/ML algorithms locally to improve imaging quality, battery life, and typing experience.

However, integrating large language model (LLM) and other GenAI models on-device necessitates a different computing platform and software capabilities. According to Canalys, an AI-capable smartphone ‘s system-on-chip (SoC) should include a dedicated unit like MediaTek’s APU (AI processing unit) or a Google TPU to accelerate AI-related tasks. The smartphone should also be able to run LLMs and other generative AI models on-device.

Further, an LLM’s inference (allowing the model to generate responses that are more relevant and appropriate based on the context) performance on-device should be faster than the average adult’s reading speed, while image generation using on-device AI should be less than two seconds. According to Counterpoint, GenAI-enabled smartphones will run “size-optimized AI models natively and come with certain hardware specifications”.

Most new iPhones and iPads have a Neural Engine or NPU. In a research paper titled ‘LLM in a flash: Efficient Large Language Model Inference with Limited Memory’, Apple researchers claim to have tackled the challenge of running LLMs on devices with constrained memory capacities. The researchers outline two techniques: ‘windowing’ and ‘row-column bundling’, which “collectively contribute to a significant reduction in the data load and an increase in the efficiency of memory usage”. Apple has also reportedly been developing a proprietary LLM named Ajax since early 2023, aimed at integrating AI into its services such as Siri, Apple Music, and core apps.

Qualcomm, on its part, claims that using its new Snapdragon 8 Gen 3 processor (likely to power the Samsung Galaxy S24 series), LLM models can run up to 20 tokens/sec — one of the fastest in the smartphone industry — and generate an image at a fraction of a second with Stable Diffusion (a popular text-to-image generative AI model capable of creating photorealistic images given any text input within tens of seconds). Brands including ASUS, Honor, Nubia, OnePlus, Oppo, realme, Sony, Xiaomi and ZTE have committed to adopting this chip into their forthcoming premium phones but they will have to work with Qualcomm to integrate AI into their devices.

That said, big tech companies are already working on including some GenAI features in smartphones. During Google’s I/O 2023 in May, the company introduced new ways to personalize the Android phone, powered by its advances in generative AI technology. For instance, Magic Compose — new Messages by Google feature powered by generative AI — can offer suggested responses based on the context of the messages, and rewrite in different styles. Magic Editor in Google Photos (Pixel 8 and Pixel 8 Pro) is a new experimental editing experience that uses generative AI to help reposition and resize subjects or use presets to make the background pop with just a few taps. Audio Magic Erase can reduce distracting sounds in your video including howling winds or noisy crowds.

Further, Call Screen uses AI to reduce spam calls. It will silently answer calls from unknown numbers with a more natural-sounding voice to engage the caller. It’s also smart enough to separate the calls you want from the calls you don’t. And soon, Call Screen (similar to Samsung’s ‘Smart Call’ function which lets you know who’s calling even when the number isn’t on your contact list) will suggest contextual replies without having to answer the phone.

Running Stable Diffusion on a smartphone is just the beginning, according to Qualcomm, which aims to use a single AI stack that works across different end devices and different models. This implies that the optimizations for Stable Diffusion to run efficiently on phones can also be used for other platforms like laptops, extended reality (XR) headsets, and virtually any other device powered by Qualcomm Technologies. 

Qualcomm notes that running all the AI processing in the cloud will be too costly, making efficient edge AI processing very important. According to Counterpoint, Qualcomm is likely to capture over 80% of the GenAI smartphone market for the next two years while MediaTek is likely to catch up with its Dimensity 9300-based devices.

Originally appeared on: TheSpuzz