ESP32 is a powerful 32-bit controller that clocks at speeds 160~240 MHz. Designed for IoT, the controller has on-chip Wi-Fi (40 MHz), Bluetooth 4.0, Bluetooth Low Energy, CAN, and an Ethernet interface. With 34 GPIO, 18 analog pins, 16 software PWM channels, 4 SPI ports, 2 I2C ports, 2 I2S ports, and 2 UART ports, ESP32 offers endless possibilities for embedded design. Being such a capable chip, ESP32 can be deployed to power a variety of IoT network nodes.
Different IoT devices have different hardware requirements. Managing data right at the heart of a “Thing” is one crucial aspect of embedded hardware. Once you would have selected ESP32 for your IoT design, you do not need to worry about computational resources. ESP32 is more than sufficient to run a networked embedded application. Memory and RAM are the areas where microcontrollers often have their natural limits, but ESP32’s RAM (512 KB) is sufficient for any embedded or IoT task. ESP32 also has 4~16 MB flash, of which 512 bytes is used for storing data, while the rest is used to store user code.
However, the available 512 bytes (ROM) for data can be insufficient for many IoT projects, particularly those that involve using audio or video files or ones that require longer onboard data logging. For such projects, we need to explore other storage media that can be interfaced with ESP32 to extend memory. Fortunately, ESP32 can be uploaded with different file systems that remove the restriction to use flat files only. In this article, we will explore different storage devices that can be interfaced with ESP32 and investigate different possible file systems on ESP32 to enable adequate data storage and management.
ESP32 flash memory
ESP32 comes with on-chip or onboard flash memory of 4MB, 8 MB, or 16 MB. Most of this memory space is reserved for storing the bootloader, radio driver, and application code. As noted earlier, only 512 bytes of flash memory is available for storing user data. The flash memory is partitioned into three slots for non-OTA configuration—NVS library, PHY_init, and factory. The NVS library partition is the only space where user application parameters can be saved. The factory partition stores the bootloader, while the PHY partition stores the radio driver for Bluetooth and Wi-Fi, responsible for implementing the physical networking layer. With OTA configuration (Over-The-Air updates), there are three additional partitions, of which one “OTAdata” is used to store the OTA app, and two partitions are used to store OTA updates.
The flash memory on ESP32 can only store scalar user data initiated through variables or data structures in the code. These scalar values are stored in the NVS library partition as application parameters. The flash memory is a few MB in size, which is only sufficient for cost-sensitive applications having a small memory footprint. Some parts of the flash memory can be partitioned to store user data such as text, images, and other files only after mounting a filesystem (SPIFFS or FAT) onto it. Otherwise, flash memory can only store user-generated binary code and application parameters. The SPI flash on ESP32 may use 2-line or 4-line operation mode. The data rate can be set 40MHz or 80MHz.
Extending memory with SD cards
Although the flash memory can be mounted and partitioned to store data like images and other files, it is often insufficient to store an appropriate volume of data. For example, audio and video files could be several megabytes in size. These file types can never be stored on flash; the best alternative is using SD cards.
SD cards 4GB up to 32GB tend to work fine with ESP32, where the maximum size for a file can be up to 4GB. This is plenty of space for storing audio, video, and database files. ESP32 can stream audio/video, capture images, and maintain large data logs by interfacing an SD card of sufficient size. There are two interfaces on ESP32 to access SD cards—SDIO and SPI. If the SDIO interface is used to attach an SD card reader with ESP32, it occupies six fixed GPIOs on the board. The SPI interface can be multiplexed with any GPIO and requires three or four pins. It is possible to connect more than one SD card with ESP32 by configuring additional pins as chip-select. The SDIO interface has a default frequency of 20 MHz that can be configured to 40 MHz for 1/4/8 line modes. The frequency for the SPI interface is configurable through user code. In ESP32-S2, only the SPI interface is available for interfacing SD cards. The SD card must have a FAT filesystem to work with ESP32.
Extending memory with eMMC cards
eMMC cards are similar to SD cards and use the same driver as the SD cards. It is important to note that eMMC memory cards are not supported in ESP32-S2. With other models of ESP32, eMMC cards can be interfaced to store audio, video, and database files. These memory cards can be interfaced with ESP32 only using the SDIO interface. The default data rate is 20 MHz which can be set to 40MHz with an 8-line mode and 4-line DDR mode. The eMMC cards are often interfaced with ESP32 chips in wearables like smartwatches. In such applications, the eMMC card stores system applications containing graphics and/or database files.
Extending memory with EEPROM
In low-power applications like the IIoT, images, audio, or video files do not need to be stored. The controller needs some extra space to store configuration parameters, maintain logs, or restore data before re-transmission. In such applications, SD cards or eMMC cards can be overkill draining the battery as well as unnecessarily multiplying the device cost. For these situations, extra space of 1MB or 16MB can be added to the device by interfacing EEPROM with ESP32. The ESP32 supports AT24C01/02 EEPROM chips from Atmel and can be interfaced with ESP32 via an I2C interface. ESP32 can also read/write EEPROM at a data rate of 100 to ~400KHz and access data through direct addressing.
Supported filesystem in ESP32
A typical ESP32 configuration is straightforward—until you have to deal with flat files. However, when it is required to store audio, video, database, or other complex files on the ESP32 board, complications arise. Microcontrollers are designed to store flat data and rarely need to operate on complex file types. Fortunately, ESP32 can be uploaded with different file systems to store and handle complex file types. The following file systems are supported in ESP32.
- NVS library
- Virtual file system
NVS library filesystem
One of the default partitions in ESP32 flash is the NVS library. The partition can be used to store application parameters. The application data stored in NVS is not lost when ESP32 shutsdown, restarts, or encounters power failure. The partition needs to be activated using the NVS library in the user code, which then mounts the NVS library file system on the respective partition. The file system allows storing key-value pairs where keys can be ASCII strings and values can be integers, strings, or binary large objects (BLOBs) such as character arrays. The NVS partition is still not suitable for storing large strings. Depending upon the default or custom partition table applied, the partition is limited in size to KB or MB.
Serial Peripheral Interface Flash File System (SPIFFS) is the default file system supported by ESP32. SPIFFS is a lightweight microcontroller file system with SPI NOR flash of less than 128 MB. The file system requires less RAM compared to the FAT file system. It supports wear leveling, data consistency, and POSIX support. The file system can be mounted to ESP32 or unified to a virtual file system on ESP32. The SPIFFS file system is suitable for partitioning flash memory for storing images, audio, video, and other files.
The SPIFFS file system does not come with directory support because it is not platform-independent. An organized directory structure is often required in many situations, such as when ESP32 is hosting a small web server with a couple of HTML, CSS, and image files. The other situation can be when external memory devices like SD cards or eMMC cards are interfaced with ESP32 and have files stored not directly under the root but in directories. In such cases, ESP32 can be mounted with the FAT-Fs file system. The FAT-Fs acts as a platform-independent layer and provides a unified interface to access all available memory devices like external flash chips, SD cards, and eMMC.
The file system supports a directory structure and enables data encryption. The FAT-Fs supports memory devices gigabytes in size, while SPIFFS is only suitable for flash memories up to 128 MB. The wear leveling is optional in FAT-Fs. The FAT-Fs file system on ESP32 must be used to store audio, video, images, and other file types when the files need to be stored on external memory devices of large capacity or require a directory structure for storage.
The ESP32 can also be uploaded with Virtual File System. It unifies SPIFFS and FAT-Fs file system on ESP32 and provides a file-like interface to external drivers. The virtual file system is useful when both SPIFFS and FAT file systems are required on ESP32 for certain reasons.
The management of user and network data can be challenging with microcontrollers. ESP32 is no exception. ESP32 is a powerful Wi-Fi development platform for building IoT applications of all sizes. Fortunately, the memory on ESP32 can be extended with external flash/EEPROM, SD cards, and eMMC. The external flash chips or EEPROM are suitable when a small increment in storage capacity is required. If the application requires storing large files like audio, video, or database files, SD cards or eMMC cards should be relied upon.
With the appropriate selection of onboard or on-device storage, the next important step in data management on ESP32 is selecting a proper filesystem. If an application only requires storing important key-value parameters that must not be lost while shutdown or power failure, the NVS library filesystem is sufficient. For example, IP-level parameters for connecting ESP32 to an existing network can be stored in the NVS partition as key-value pairs over the NVS library filesystem. If the application requires storing audio, video, or data logs of small size, SPIFFS is most suitable because it will not consume RAM. For applications that require storing large files, organizing files into a directory structure, or requiring data to be encrypted, the FAT filesystem on ESP32 is the option. For some rare circumstances, both SPIFFS and the FAT filesystem may be required on ESP32, after which ESP32 can be uploaded with the virtual file system to extract the benefits of both FAT and SPIFFS filesystem features. Although ESP32 supports storing audio and video files, the maximum data rate of 40MHz can still be a restriction.
You may also like:
Filed Under: Applications, IoT, Tutorials