What are the top machine-learning frameworks for microcontrollers

Machine learning (ML) is becoming increasingly important for microcontrollers because it enables smart and autonomous decision-making in embedded systems. The many Internet of Things (IoT) applications — often called “smart devices” — only become intelligent thanks to ML.

Microcontrollers are commonly used in edge computing devices where data is processed locally rather than being sent to a centralized server. ML on microcontrollers allows for real-time processing of data at the source, reducing latency, and bandwidth requirements.

Before concrete machine-learning frameworks emerged in the microcontroller space, ML models were used in tiny machines by manually optimizing algorithms and models for the specific constraints of the microcontroller devices. Since then, ML has become far more customized, particularly for microcontrollers. Customization is based on specific ML learning models for specific niche applications in the embedded space.

TensorFlow Lite was the first and most significant framework for using ML on microcontrollers. TensorFlow was released in 2015, and TensorFlow Lite followed soon. It’s specifically aimed at mobile and embedded devices.

“TensorFlow Lite for Microcontrollers” was designed to run ML models on microcontrollers and other devices with only kilobytes of memory. It provided tools and pre-trained models that could be easily deployed. This opened the doors for a broader range of developers and applications in Tiny Machine Learning (TinyML, the practice of deploying ML models on extremely resource-constrained or embedded devices).

Today, many other frameworks enable deploying machine learning models onto microcontrollers. For an embedded engineer, it’s essential to be familiar with at least one of these frameworks, as ML is the future of embedded devices. After web servers and clouds, most other artificial intelligence (AI) applications will emerge from embedded devices. AI is also likely to experience a shift from cloud to edge computing.

In this article, we’ll discuss different frameworks that can be used to deploy ML models on microcontrollers.

Why use ML on microcontrollers?
Machine learning is important for embedded devices, particularly the IoT. Everyday electronics can become “smarter” by directly integrating ML models into microcontrollers. This means they require no connection to an external processor or cloud service for functions like speech recognition, gesture detection, predictive maintenance, and others.

Running ML models directly on microcontrollers allows for real-time processing with minimal latency. This is essential for applications, including safety-critical systems and user-interactive devices that call for quick reactions.

As microcontrollers are designed to be highly power-efficient, which is vital for battery-operated devices, running ML models on these devices can lead to significant energy savings compared to sending data to and from a cloud server for processing.

ML on microcontrollers also solves two major challenges in the path of smart devices – security and privacy:

Reducing the dependency of devices on a cloud server or service
Removing the need to transmit data over the Internet enhances security and privacy.

As the data is locally processed on a secure device, there’s less chance of a data breach. Local data processing further eliminates the requirement for constant connectivity and the bandwidth that goes along with it. This is especially helpful in places where Internet access is spotty or nonexistent.

Embedding machine learning in microcontrollers is more cost-effective than setting up a cloud-based infrastructure for data processing. This also opens up possibilities for new applications and innovations, particularly in areas where traditional ML deployment isn’t feasible due to size, power, or connectivity limitations.

The top ML frameworks
TensorFlow Lite for Microcontrollers is no longer the only framework for deploying machine learning on microcontrollers. Several exist as TinyML is one of the fastest-changing fields. Some of the most significant of them are mentioned below.

1. TensorFlow Lite for Microcontrollers
2. Edge Impulse
3. MicroML
4. Cube.AI
5. TinyMLPerf
6. TinyNN

TensorFlow Lite for Microcontrollers is the earliest framework for deploying ML models on microcontrollers, which can be found in everything from wearables to IoT devices. It is an extension of TensorFlow Lite, Google’s lightweight solution for mobile and embedded devices. Its runtime can fit in as little as 16KB for some models on microcontrollers, making it ideal for battery-powered or energy-harvesting devices.

TensorFlow Lite can run on bare-metal microcontrollers without needing a full operating system, providing a set of pre-trained models optimized for performance on resource-constrained devices. It supports several microcontroller architectures and is easily integrated with existing hardware setups.

However, implementing ML on such constrained devices is not without challenges. Major issues are memory limitations, power constraints, and the need for model optimization. TensorFlow Lite addresses these through model quantization (reducing the precision of the numbers in the model), pruning (removing parts of the model that are less critical), and efficient use of hardware accelerators where available. Some notable use cases include wearable devices, smart home devices, predictive maintenance in IIoT, crop monitoring, and real-time health monitoring.

Edge Impulse is a complete platform that lets developers collect data, train a model in the cloud, and then deploy it to a microcontroller. It has integrated TensorFlow Lite for Microcontrollers and offers an easy-to-use interface, making it suitable for those new to the field. It’s designed for edge ML, providing tools and resources to collect, process, and analyze real-world data, then develop, train, and deploy ML models directly onto edge devices. These devices range from small microcontrollers in sensors to more powerful single-board computers.

This platform allows developers to collect data from various sources, including sensors and existing datasets. They can then easily design, train, and validate ML models on this data through a dashboard. The platform also has automatic optimization techniques to ensure models are lightweight yet effective for edge deployment. Once a model is trained, developers can deploy it on a target edge device from the platform.

Edge Impulse supports an extensive list of hardware and architectures. Developers can test their models in real-time and monitor their performance once deployed, enabling continuous improvement and optimization. More sophisticated AI algorithms, broader hardware support, and deeper integration with cloud services for hybrid edge-cloud solutions are expected soon.

MicroML is a lightweight machine-learning framework that runs directly on microcontrollers without an operating system. It allows on-device processing, which is critical for applications requiring real-time decision-making, such as autonomous vehicles or medical devices.

MicroML is at the head of a computing paradigm shift, enabling ML in even in limited settings. By directly implementing intelligent decision-making on microcontrollers, MicroML is leading to new machine-learning niches. With advancements in model compression techniques and the development of more powerful microcontrollers, the range and efficacy of MicroML applications will only expand. Additionally, as the IoT continues to grow, so will the demand for intelligent edge computing solutions like MicroML. Though this is a niche solution compared to the others, it’s suitable for several applications.

Cube.AI, developed by STMicroelectronics, is a part of the STM32Cube software platform, a comprehensive software suite for STM32 microcontrollers and microprocessors. It allows developers to convert pre-trained neural networks into optimized C-code for STM32 microcontrollers.

Cube.AI takes neural network models, typically developed and trained on powerful computing resources, and converts them into an optimized form for STM32 microcontrollers. It supports models trained in popular ML frameworks like TensorFlow and Keras, providing flexibility for developers. The tool optimizes the models to run on microcontrollers’ limited resources (memory and processing power) without compromising performance. Cube.AI could support more neural network types and better optimization algorithms and will offer broader compatibility with emerging ML frameworks in the future.

TinyNN is a small and efficient neural network framework for microcontrollers, designed to be lightweight and easily portable. TinyNN is intended to be minimalist, requiring minimal memory and processing power, making it ideal for low-cost, low-power microcontrollers. It focuses on streamlined neural network models that are sufficiently powerful for a range of tasks yet simple enough to run on microcontrollers. It emphasizes efficiency in memory usage and computational requirements, which is crucial for battery-operated or power-sensitive applications.

TinyNN’s capabilities are expected to increase as developments in neural network design, model compression methods, and energy-efficient computing open new applications for microcontrollers.

TinyMLPerf is not a framework but an important benchmarking tool. It’s part of the MLPerf suite for benchmarking TinyML solutions, measuring and comparing the performance of ML models on microcontroller systems. This is crucial for determining which frameworks or tools to use.

While MLPerf focuses on a broad range of ML platforms and applications, TinyMLPerf specifically targets low-power, resource-constrained microcontrollers commonly used in IoT and embedded systems. It offers a set of standardized tests, making it easier to compare different ML solutions in terms of efficiency, speed, and size. These benchmarks can run on actual hardware, reflecting real-world performance.

The benchmarks typically work by running a series of predefined ML tasks and measuring key parameters like execution time, power consumption, and model size. The results show how well a given setup performs under typical TinyML workloads.

TinyMLPerf covers tasks relevant to TinyML, such as image recognition, audio processing, and sensor data analysis. It allows developers and researchers to objectively compare the performance of different ML models and hardware platforms in a TinyML context. It also helps identify the strengths and weaknesses of various approaches, guiding future improvements.

Conclusion
TensorFlow Lite for Microcontrollers was the first framework for deploying machine learning on microcontrollers. With the changing AI landscape, other notable frameworks have emerged. For example, Edge Impulse is an all-in-one platform where developers can integrate TensorFlow Lite’s ML solutions with various hardware platforms and evaluate their performance in real-time.

Additionally, MicroML has come up as a solution to implement ML models directly over microcontrollers. Cube.AI is optimizing many ML models in Tensor and Keras for microcontrollers. Also, TinyNN has introduced neural networks to microcontroller platforms. And TinyMLPerf benchmarks the real-life performance of ML models and tools on various microcontroller systems.