Engineers Garage

  • Electronic Projects & Tutorials
    • Electronic Projects
      • Arduino Projects
      • AVR
      • Raspberry pi
      • ESP8266
      • BeagleBone
      • 8051 Microcontroller
      • ARM
      • PIC Microcontroller
      • STM32
    • Tutorials
      • Audio Electronics
      • Battery Management
      • Brainwave
      • Electric Vehicles
      • EMI/EMC/RFI
      • Hardware Filters
      • IoT tutorials
      • Power Tutorials
      • Python
      • Sensors
      • USB
      • VHDL
    • Circuit Design
    • Project Videos
    • Components
  • Articles
    • Tech Articles
    • Insight
    • Invention Stories
    • How to
    • What Is
  • News
    • Electronic Product News
    • Business News
    • Company/Start-up News
    • DIY Reviews
    • Guest Post
  • Forums
    • EDABoard.com
    • Electro-Tech-Online
    • EG Forum Archive
  • DigiKey Store
    • Cables, Wires
    • Connectors, Interconnect
    • Discrete
    • Electromechanical
    • Embedded Computers
    • Enclosures, Hardware, Office
    • Integrated Circuits (ICs)
    • Isolators
    • LED/Optoelectronics
    • Passive
    • Power, Circuit Protection
    • Programmers
    • RF, Wireless
    • Semiconductors
    • Sensors, Transducers
    • Test Products
    • Tools
  • Learn
    • eBooks/Tech Tips
    • Design Guides
    • Learning Center
    • Tech Toolboxes
    • Webinars & Digital Events
  • Resources
    • Digital Issues
    • EE Training Days
    • LEAP Awards
    • Podcasts
    • Webinars / Digital Events
    • White Papers
    • Engineering Diversity & Inclusion
    • DesignFast
  • Guest Post Guidelines
  • Advertise
  • Subscribe

The First Spark/Hadoop ARM Cluster

By Samidha Verma September 5, 2016

Hadoop-ARM-cluster

Figure 1: Spark Hadoop ARM Cluster
 

Heng Yan posted this experimental project where he has prototyped an ARM-based cluster, designed to process Big Data. It’s a 22-node Cubieboard A10 with 100 Mbps Ethernet.

The designing of an ARM chip does not allow processing of Big Data but it is gradually becoming powerful enough to do so. Several attempts have been made to run Apache Hadoop on top of an ARM cluster and many more experiments are taking place successfully. But still the question arises that whether it is feasible to do Big Data on a low-cost ARM cluster?

The doubt arises because of the disk with slow I/O and networking of ARM SoCs, Hadoop’s MapReduce that will really not be able to process a real Big Data computation, which average 15GB per file.

cluster-running-spark-and-hadoop

Figure 2: Cluster running of Spark and Hadoop
 

The cluster running of Spark and Hadoop may solve the answer. As Hadoop’s Map Reduce is not a good choice to process on this kind of cluster, so only HDFS is used and an alternative was tried to found to stumble upon Apache Spark. It is important to know that Spark is an in-memory framework which optionally spills intermediate results out to a disk when a computing node is running out-of-memory and luckily it runs fine on the cluster.

The cluster has total 20GB of RAM and only 10GB is available for data processing. In case, larger amount of memory when tried to allocated, some nodes will die during the computation.

But the cluster here is good enough to crush a single, 34GB, Wikipedia article file from the year 2012. This was possible after a tweaked word count program was ran in Spark’s shell and waited for few minutes and finally the cluster answered the word count. 

The designing consisted of 20 Spark worker nodes, and 2 of them running Hadoop Data Nodes that allowed us to understand that the data locality of Spark/Hadoop cluster. The Hadoop’s Name Node and the Spark’s master node were ran on the same machine, whereas another machine was the driver.

This could be said as success as an ARM system-on-chip board has demonstrated an enough power to form a cluster and process non-trivial size of data. Still, there are missing puzzle-pieces to be found and the need to choose the right software package. Attempts are being made to develop a new one, bigger by CPU cores with small size.

The demonstration of this project is available on the following website-


Filed Under: Reviews

 

Next Article

← Previous Article
Next Article →

Questions related to this article?
👉Ask and discuss on Electro-Tech-Online.com and EDAboard.com forums.



Tell Us What You Think!! Cancel reply

You must be logged in to post a comment.

EE TECH TOOLBOX

“ee
Tech Toolbox: 5G Technology
This Tech Toolbox covers the basics of 5G technology plus a story about how engineers designed and built a prototype DSL router mostly from old cellphone parts. Download this first 5G/wired/wireless communications Tech Toolbox to learn more!

EE Learning Center

EE Learning Center
“engineers
EXPAND YOUR KNOWLEDGE AND STAY CONNECTED
Get the latest info on technologies, tools and strategies for EE professionals.

HAVE A QUESTION?

Have a technical question about an article or other engineering questions? Check out our engineering forums EDABoard.com and Electro-Tech-Online.com where you can get those questions asked and answered by your peers!


RSS EDABOARD.com Discussions

  • High failure rate of LLC converter
  • Broad band impedance matching network for loop antenna using transformer for wireless power transfer
  • Earth leakage circuit with TL071
  • IGBTs without negative gate drive
  • VHF radio

RSS Electro-Tech-Online.com Discussions

  • Pic18f25q10 osccon1 settings swordfish basic
  • turbo jet fan - feedback appreciated.
  • More fun with ws2812 this time XC8 and CLC
  • I Wanna build a robot
  • Pickit 5

Featured – LoRa/LoRaWan Series

  • What is the LoRaWAN network and how does it work?
  • Understanding LoRa architecture: nodes, gateways, and servers
  • Revolutionizing RF: LoRa applications and advantages
  • How to build a LoRa gateway using Raspberry Pi
  • How LoRa enables long-range communication
  • How communication works between two LoRa end-node devices

Recent Articles

  • How IoT network topologies work
  • The top five AI startups to watch in 2025
  • STMicroelectronics unveils SoC based on secure MCU
  • Nexperia’s 48 V ESD diodes support higher data rates with ultra-low capacitance design
  • Taoglas releases Patriot antenna with 18 integrated elements covering 600 to 6000 MHz

EE ENGINEERING TRAINING DAYS

engineering

Submit a Guest Post

submit a guest post
Engineers Garage
  • Analog IC TIps
  • Connector Tips
  • Battery Power Tips
  • DesignFast
  • EDABoard Forums
  • EE World Online
  • Electro-Tech-Online Forums
  • EV Engineering
  • Microcontroller Tips
  • Power Electronic Tips
  • Sensor Tips
  • Test and Measurement Tips
  • 5G Technology World
  • Subscribe to our newsletter
  • About Us
  • Contact Us
  • Advertise

Copyright © 2025 WTWH Media LLC. All Rights Reserved. The material on this site may not be reproduced, distributed, transmitted, cached or otherwise used, except with the prior written permission of WTWH Media
Privacy Policy

Search Engineers Garage

  • Electronic Projects & Tutorials
    • Electronic Projects
      • Arduino Projects
      • AVR
      • Raspberry pi
      • ESP8266
      • BeagleBone
      • 8051 Microcontroller
      • ARM
      • PIC Microcontroller
      • STM32
    • Tutorials
      • Audio Electronics
      • Battery Management
      • Brainwave
      • Electric Vehicles
      • EMI/EMC/RFI
      • Hardware Filters
      • IoT tutorials
      • Power Tutorials
      • Python
      • Sensors
      • USB
      • VHDL
    • Circuit Design
    • Project Videos
    • Components
  • Articles
    • Tech Articles
    • Insight
    • Invention Stories
    • How to
    • What Is
  • News
    • Electronic Product News
    • Business News
    • Company/Start-up News
    • DIY Reviews
    • Guest Post
  • Forums
    • EDABoard.com
    • Electro-Tech-Online
    • EG Forum Archive
  • DigiKey Store
    • Cables, Wires
    • Connectors, Interconnect
    • Discrete
    • Electromechanical
    • Embedded Computers
    • Enclosures, Hardware, Office
    • Integrated Circuits (ICs)
    • Isolators
    • LED/Optoelectronics
    • Passive
    • Power, Circuit Protection
    • Programmers
    • RF, Wireless
    • Semiconductors
    • Sensors, Transducers
    • Test Products
    • Tools
  • Learn
    • eBooks/Tech Tips
    • Design Guides
    • Learning Center
    • Tech Toolboxes
    • Webinars & Digital Events
  • Resources
    • Digital Issues
    • EE Training Days
    • LEAP Awards
    • Podcasts
    • Webinars / Digital Events
    • White Papers
    • Engineering Diversity & Inclusion
    • DesignFast
  • Guest Post Guidelines
  • Advertise
  • Subscribe