Engineers Garage

  • Projects and Tutorials
    • Electronic Projects
      • 8051
      • Arduino
      • ARM
      • AVR
      • PIC
      • Raspberry pi
      • STM32
    • Tutorials
    • Circuit Design
    • Project Videos
    • Components
  • Articles
    • Tech Articles
    • Insight
    • Invention Stories
    • How to
    • What Is
  • News
    • Electronic Products News
    • DIY Reviews
    • Guest Post
  • Forums
    • EDABoard.com
    • Electro-Tech-Online
    • EG Forum Archive
  • Digi-Key Store
    • Cables, Wires
    • Connectors, Interconnect
    • Discrete
    • Electromechanical
    • Embedded Computers
    • Enclosures, Hardware, Office
    • Integrated Circuits (ICs)
    • Isolators
    • LED/Optoelectronics
    • Passive
    • Power, Circuit Protection
    • Programmers
    • RF, Wireless
    • Semiconductors
    • Sensors, Transducers
    • Test Products
    • Tools
  • EE Resources
    • DesignFast
    • LEAP Awards
    • Oscilloscope Product Finder
    • White Papers
    • Webinars
  • EE Learning Center
    • Design Guides
      • WiFi & the IOT Design Guide
      • Microcontrollers Design Guide
      • State of the Art Inductors Design Guide
  • Women in Engineering

The First Spark/Hadoop ARM Cluster

By Samidha Verma

Hadoop-ARM-cluster

Figure 1: Spark Hadoop ARM Cluster
 

Heng Yan posted this experimental project where he has prototyped an ARM-based cluster, designed to process Big Data. It’s a 22-node Cubieboard A10 with 100 Mbps Ethernet.

The designing of an ARM chip does not allow processing of Big Data but it is gradually becoming powerful enough to do so. Several attempts have been made to run Apache Hadoop on top of an ARM cluster and many more experiments are taking place successfully. But still the question arises that whether it is feasible to do Big Data on a low-cost ARM cluster?

The doubt arises because of the disk with slow I/O and networking of ARM SoCs, Hadoop’s MapReduce that will really not be able to process a real Big Data computation, which average 15GB per file.

cluster-running-spark-and-hadoop

Figure 2: Cluster running of Spark and Hadoop
 

The cluster running of Spark and Hadoop may solve the answer. As Hadoop’s Map Reduce is not a good choice to process on this kind of cluster, so only HDFS is used and an alternative was tried to found to stumble upon Apache Spark. It is important to know that Spark is an in-memory framework which optionally spills intermediate results out to a disk when a computing node is running out-of-memory and luckily it runs fine on the cluster.

The cluster has total 20GB of RAM and only 10GB is available for data processing. In case, larger amount of memory when tried to allocated, some nodes will die during the computation.

But the cluster here is good enough to crush a single, 34GB, Wikipedia article file from the year 2012. This was possible after a tweaked word count program was ran in Spark’s shell and waited for few minutes and finally the cluster answered the word count. 

The designing consisted of 20 Spark worker nodes, and 2 of them running Hadoop Data Nodes that allowed us to understand that the data locality of Spark/Hadoop cluster. The Hadoop’s Name Node and the Spark’s master node were ran on the same machine, whereas another machine was the driver.

This could be said as success as an ARM system-on-chip board has demonstrated an enough power to form a cluster and process non-trivial size of data. Still, there are missing puzzle-pieces to be found and the need to choose the right software package. Attempts are being made to develop a new one, bigger by CPU cores with small size.

The demonstration of this project is available on the following website-


Filed Under: Reviews

 

Questions related to this article?
👉Ask and discuss on EDAboard.com and Electro-Tech-Online.com forums.



Tell Us What You Think!! Cancel reply

You must be logged in to post a comment.

HAVE A QUESTION?

Have a technical question about an article or other engineering questions? Check out our engineering forums EDABoard.com and Electro-Tech-Online.com where you can get those questions asked and answered by your peers!


Featured Tutorials

  • Introduction to Brain Waves & its Types (Part 1/13)
  • Understanding NeuroSky EEG Chip in Detail (Part 2/13)
  • Performing Experiments with Brainwaves (Part 3/13)
  • Amplification of EEG Signal and Interfacing with Arduino (Part 4/13)
  • Controlling Led brightness using Meditation and attention level (Part 5/13)
  • Control Motor’s Speed using Meditation and Attention Level of Brain (Part 6/13)

Stay Up To Date

Newsletter Signup

Sign up and receive our weekly newsletter for latest Tech articles, Electronics Projects, Tutorial series and other insightful tech content.

EE Training Center Classrooms

EE Classrooms

Recent Articles

  • EdgeLock A5000 Secure Authenticator
  • How to interface a DS18B20 temperature sensor with MicroPython’s Onewire driver
  • Introduction to Brain Waves & its Types (Part 1/13)
  • An Embedded Developer’s Perspective on IOT (Internet of Things)
  • Understanding NeuroSky EEG Chip in Detail (Part 2/13)

Most Popular

5G 555 timer circuit 8051 ai Arduino atmega16 automotive avr bluetooth dc motor display Electronic Part Electronic Parts Fujitsu ic infineontechnologies integratedcircuit Intel IoT ir lcd led maximintegratedproducts microchip microchiptechnology Microchip Technology microcontroller microcontrollers mosfet motor powermanagement Raspberry Pi remote renesaselectronics renesaselectronicscorporation Research samsung semiconductor sensor software STMicroelectronics switch Technology vishayintertechnology wireless

RSS EDABOARD.com Discussions

  • Need help to choose accelerometer
  • Pic 16f877A Hex file
  • 7 segment display connections
  • finding attenuation coefficient in CST
  • Effect of variable gain amplifier and LNA on the input RF signal's phase

RSS Electro-Tech-Online.com Discussions

  • How to designing a battery charging indicator circuit for 18650 battery pack
  • NOR gate oscillator in LTspice not working
  • ICM7555 IC duty cycle limit at high frequency?
  • I've DESTROYED 3 piezo buttons :((((
  • led doorbell switch
Engineers Garage
  • Analog IC TIps
  • Connector Tips
  • DesignFast
  • EDABoard Forums
  • EE World Online
  • Electro-Tech-Online Forums
  • Microcontroller Tips
  • Power Electronic Tips
  • Sensor Tips
  • Test and Measurement Tips
  • 5G Technology World
  • About Us
  • Contact Us
  • Advertise

Copyright © 2022 WTWH Media LLC. All Rights Reserved. The material on this site may not be reproduced, distributed, transmitted, cached or otherwise used, except with the prior written permission of WTWH Media
Privacy Policy | Advertising | About Us

Search Engineers Garage

  • Projects and Tutorials
    • Electronic Projects
      • 8051
      • Arduino
      • ARM
      • AVR
      • PIC
      • Raspberry pi
      • STM32
    • Tutorials
    • Circuit Design
    • Project Videos
    • Components
  • Articles
    • Tech Articles
    • Insight
    • Invention Stories
    • How to
    • What Is
  • News
    • Electronic Products News
    • DIY Reviews
    • Guest Post
  • Forums
    • EDABoard.com
    • Electro-Tech-Online
    • EG Forum Archive
  • Digi-Key Store
    • Cables, Wires
    • Connectors, Interconnect
    • Discrete
    • Electromechanical
    • Embedded Computers
    • Enclosures, Hardware, Office
    • Integrated Circuits (ICs)
    • Isolators
    • LED/Optoelectronics
    • Passive
    • Power, Circuit Protection
    • Programmers
    • RF, Wireless
    • Semiconductors
    • Sensors, Transducers
    • Test Products
    • Tools
  • EE Resources
    • DesignFast
    • LEAP Awards
    • Oscilloscope Product Finder
    • White Papers
    • Webinars
  • EE Learning Center
    • Design Guides
      • WiFi & the IOT Design Guide
      • Microcontrollers Design Guide
      • State of the Art Inductors Design Guide
  • Women in Engineering