Engineers Garage

  • Electronic Projects & Tutorials
    • Electronic Projects
      • Arduino Projects
      • AVR
      • Raspberry pi
      • ESP8266
      • BeagleBone
      • 8051 Microcontroller
      • ARM
      • PIC Microcontroller
      • STM32
    • Tutorials
      • Audio Electronics
      • Battery Management
      • Brainwave
      • Electric Vehicles
      • EMI/EMC/RFI
      • Hardware Filters
      • IoT tutorials
      • Power Tutorials
      • Python
      • Sensors
      • USB
      • VHDL
    • Circuit Design
    • Project Videos
    • Components
  • Articles
    • Tech Articles
    • Insight
    • Invention Stories
    • How to
    • What Is
  • News
    • Electronic Product News
    • Business News
    • Company/Start-up News
    • DIY Reviews
    • Guest Post
  • Forums
    • EDABoard.com
    • Electro-Tech-Online
    • EG Forum Archive
  • DigiKey Store
    • Cables, Wires
    • Connectors, Interconnect
    • Discrete
    • Electromechanical
    • Embedded Computers
    • Enclosures, Hardware, Office
    • Integrated Circuits (ICs)
    • Isolators
    • LED/Optoelectronics
    • Passive
    • Power, Circuit Protection
    • Programmers
    • RF, Wireless
    • Semiconductors
    • Sensors, Transducers
    • Test Products
    • Tools
  • Learn
    • eBooks/Tech Tips
    • Design Guides
    • Learning Center
    • Tech Toolboxes
    • Webinars & Digital Events
  • Resources
    • Digital Issues
    • EE Training Days
    • LEAP Awards
    • Podcasts
    • Webinars / Digital Events
    • White Papers
    • Engineering Diversity & Inclusion
    • DesignFast
  • Guest Post Guidelines
  • Advertise
  • Subscribe

Training deep neural networks with less energy

By Shree Mod May 22, 2020

Representative image showing neural network in deep learning.

Early Bird is an energy-efficient method developed to train Deep Neural Network (DNNs), artificial intelligence that supports self-driving vehicles, facial recognition, and similar types of applications. It was developed by researchers from Texas A&M University and Rice University.

A study by lead authors Haoran You and Chaojian Li of Rice’s Efficient and Intelligent Computing (EIC) Lab showed Early Bird could use 10.7 times less energy to train a DNN to the same level of accuracy or better than typical training. EIC Lab director Yingyan Lin led the research along with Rice’s Richard Baraniuk and Texas A&M’s Zhangyang Wang.

“A major driving force in recent AI breakthroughs is the introduction of bigger, more expensive DNNs,” Lin said. “But training these DNNs demands considerable energy. For more innovations to be unveiled, it is imperative to find ‘greener’ training methods that both address environmental concerns and reduce financial barriers of AI research.”

The reason for Early Bird
Training cutting-edge DNNs is costly. A 2019 study by the Allen Institute for AI in Seattle found the number of computations needed to train a top-flight deep neural network increased 300,000 times between 2012-2018. Another 2019 study by researchers at the University of Massachusetts Amherst found the carbon footprint for training a single, elite DNN was roughly equivalent to the lifetime carbon dioxide emissions of five U.S. automobiles.

DNNs are constituted by billions of artificial neurons that learn to perform specific tasks. They are capable of taking decisions very much like human beings. Without any explicit programming, deep networks of artificial neurons can learn to make human-like decisions — and even outperform human experts — by “studying” a large number of previous examples. One example is AlphaGo, a deep network trained to play the board game Go, beat a professional human player in 2015 after studying tens of thousands of previously played games.

“The state-of-art way to perform DNN training is called progressive prune and train,” said Lin. “First, you train a dense, giant network, then remove parts that don’t look important — like pruning a tree. Then you retrain the pruned network to restore performance because performance degrades after pruning. And in practice you need to prune and retrain many times to get good performance.”

Rice University’s Early Bird method for training deep neural networks finds key connectivity patterns early in training, reducing the computations and carbon footprint for the increasingly popular form of artificial intelligence known as deep learning. (Graphic courtesy of Y. Lin/Rice University)

How it works
Training strengthens the bondings between most useful neurons and filters out those that can be pruned away. So, it’s possible to “prune” a network as specialized tasks are performed only by a fraction of artificial neurons. Pruning plays a key role in reducing the model size and computational costs, thereby increasing the affordability of DNN training.

“Our idea in this work is to identify the final, fully functional pruned network, which we call the ‘early-bird ticket’ in the beginning stage of this costly first step.” Lin and her team searched for early-bird tickets by studying key network connectivity patterns in the initial stage of training. The team found that Early Bird could emerge as one-tenth or less of the way through the initial phase of training.

This project is a crucial step in making AI techniques more environmentally-friendly and inclusive. If everything goes well, it will open doors to AI innovation with a single laptop or limited computational resources.

“Our method can automatically identify early-bird tickets within the first 10% or less of the training of the dense, giant networks. This means you can train a DNN to achieve the same or even better accuracy for a given tasks in about 10% or less of the time needed for traditional training, which can lead to more than one order savings in both computation and energy,” added Lin.


Filed Under: News
Tagged With: riceuniversity, texasa&muniversity
 

Next Article

← Previous Article
Next Article →

Questions related to this article?
👉Ask and discuss on Electro-Tech-Online.com and EDAboard.com forums.



Tell Us What You Think!! Cancel reply

You must be logged in to post a comment.

EE TECH TOOLBOX

“ee
Tech Toolbox: 5G Technology
This Tech Toolbox covers the basics of 5G technology plus a story about how engineers designed and built a prototype DSL router mostly from old cellphone parts. Download this first 5G/wired/wireless communications Tech Toolbox to learn more!

EE Learning Center

EE Learning Center
“engineers
EXPAND YOUR KNOWLEDGE AND STAY CONNECTED
Get the latest info on technologies, tools and strategies for EE professionals.

HAVE A QUESTION?

Have a technical question about an article or other engineering questions? Check out our engineering forums EDABoard.com and Electro-Tech-Online.com where you can get those questions asked and answered by your peers!


RSS EDABOARD.com Discussions

  • System for measuring the amount of Joules delivered from a capacitor discharge.
  • Two sections broadband impedance matching
  • The GaN revolution must now happen?
  • How to find the resonance frequency and impedance of a planar spiral coil in HFSS?
  • PhD for Electronics Designers

RSS Electro-Tech-Online.com Discussions

  • The Analog Gods Hate Me
  • Impact of Tariffs on PCB Fab
  • More fun with ws2812 this time XC8 and CLC
  • I Wanna build a robot
  • Wierd makita battery

Featured – LoRa/LoRaWan Series

  • What is the LoRaWAN network and how does it work?
  • Understanding LoRa architecture: nodes, gateways, and servers
  • Revolutionizing RF: LoRa applications and advantages
  • How to build a LoRa gateway using Raspberry Pi
  • How LoRa enables long-range communication
  • How communication works between two LoRa end-node devices

Recent Articles

  • The top five AI startups to watch in 2025
  • STMicroelectronics unveils SoC based on secure MCU
  • Nexperia’s 48 V ESD diodes support higher data rates with ultra-low capacitance design
  • Taoglas releases Patriot antenna with 18 integrated elements covering 600 to 6000 MHz
  • Amphenol RF introduces SMPM to SMPM assemblies on RG-178 cable

EE ENGINEERING TRAINING DAYS

engineering

Submit a Guest Post

submit a guest post
Engineers Garage
  • Analog IC TIps
  • Connector Tips
  • Battery Power Tips
  • DesignFast
  • EDABoard Forums
  • EE World Online
  • Electro-Tech-Online Forums
  • EV Engineering
  • Microcontroller Tips
  • Power Electronic Tips
  • Sensor Tips
  • Test and Measurement Tips
  • 5G Technology World
  • Subscribe to our newsletter
  • About Us
  • Contact Us
  • Advertise

Copyright © 2025 WTWH Media LLC. All Rights Reserved. The material on this site may not be reproduced, distributed, transmitted, cached or otherwise used, except with the prior written permission of WTWH Media
Privacy Policy

Search Engineers Garage

  • Electronic Projects & Tutorials
    • Electronic Projects
      • Arduino Projects
      • AVR
      • Raspberry pi
      • ESP8266
      • BeagleBone
      • 8051 Microcontroller
      • ARM
      • PIC Microcontroller
      • STM32
    • Tutorials
      • Audio Electronics
      • Battery Management
      • Brainwave
      • Electric Vehicles
      • EMI/EMC/RFI
      • Hardware Filters
      • IoT tutorials
      • Power Tutorials
      • Python
      • Sensors
      • USB
      • VHDL
    • Circuit Design
    • Project Videos
    • Components
  • Articles
    • Tech Articles
    • Insight
    • Invention Stories
    • How to
    • What Is
  • News
    • Electronic Product News
    • Business News
    • Company/Start-up News
    • DIY Reviews
    • Guest Post
  • Forums
    • EDABoard.com
    • Electro-Tech-Online
    • EG Forum Archive
  • DigiKey Store
    • Cables, Wires
    • Connectors, Interconnect
    • Discrete
    • Electromechanical
    • Embedded Computers
    • Enclosures, Hardware, Office
    • Integrated Circuits (ICs)
    • Isolators
    • LED/Optoelectronics
    • Passive
    • Power, Circuit Protection
    • Programmers
    • RF, Wireless
    • Semiconductors
    • Sensors, Transducers
    • Test Products
    • Tools
  • Learn
    • eBooks/Tech Tips
    • Design Guides
    • Learning Center
    • Tech Toolboxes
    • Webinars & Digital Events
  • Resources
    • Digital Issues
    • EE Training Days
    • LEAP Awards
    • Podcasts
    • Webinars / Digital Events
    • White Papers
    • Engineering Diversity & Inclusion
    • DesignFast
  • Guest Post Guidelines
  • Advertise
  • Subscribe