Neural Network Controlled Self-Driving (RC) Car

Inspired by the video of the ALVINN autonomous car which drives itself along normal roads at CMU in the mid-90s, this article will focus on building a self-driving remote control car with neural network. The ALVINN system functions on the idea of framing every couple of seconds and then passing it to neural network, trained by watching a human drive in similar environments. The trained neural network then can steer to stay on the road ahead.

This article will work on system which could operate in two modes. First mode is for the system that records video frames and control output from a human driver and use the recording later to train neural network. The second mode is drive mode, which captures live video frames, pass them to a trained neural through radio control network which makes predictions about how to drive.

Designing

The focus is majorly on the system that should be able to record video from the car, pass frames to a neural network and control the car’s steering or motors. Mounting an Android phone on the car would help to gather video frames and make neural network predictions locally on the device. Then hacked car is to be controlled from the onboard phone using an Arduino based Android ADK board, while data is recorded and transferred to a computer for training.

So the system consists of Android phone mounted on the car. It has an app running that connects to a server running on a laptop computer via Wi-Fi and streams video frames across the connection. A computer that runs a Java app called “Driver” which acts as both a TCP server that streams image frames from the phone and a user interface allowing a human driver to control the car with the mouse. The video frames are saved to disk and neural network is trained using these labelled frames in a separate environment on the computer. And an Arduino Uno, connected to the computer via USB and hacked to connect to key presses on the car’s radio controller PCB.

The Neural Network

As on the ML course, the Neural Network is trained using an octave program and here 25345 units in the input layer is used and there are four units in the output layer that corresponds to each of the instructions we can send the car – go forwards, backwards, left or right. The network backpropagation is used which produces weights corresponding to the contribution each input layer unit makes for the activation of the hidden layer units. Further, the contribution hidden layer unit makes is used for the activation of each of the output layer units.

In order to make predictions in auto mode, the author also implemented the same network topology in Java that contains the interesting code and is a generic neural net implementation, used for any three layer network. The Driver app uses this Java implementation which is set up by the octave script at the end of the training process.

Radio Control with Arduino design for commands to be sent over the USB / Serial interface that could be sent to the car via its original controller.

GET the cheapest RC car, just make sure the push button is on/off type rather than a continuous control. The hacking is done by modifying the radio control unit and the Arduino board will press them for us based on the value received over the serial port.

Figure out how the RC controller works

Try to understand the RC controller apart and figure out how it works. Follow the track of each switch to the nearest solder joint on the original board then find the pads for each switch. Identify the joints that matter, attach patch wires to each point with a soldering iron. The controller which pads for up/down left/right were being connected to when the relevant switch was pressed. Then remove the PCB from the original controller housing altogether and power it with 3.3V from the Arduino board. A simple circuit on a breadboard is built to switch the connections from software running on the Arduino board.

Arduino sketch

Now all you need is a sketch to run on the Arduino board which reads a byte from the serial interface and decodes it to determine which button is to be pushed on the remote control. The car used here is very fast and difficult to drive round a small circuit, therefore the pulse chosen in the forward direction is 250 ms followed by a 500 ms pause.

You can experiment with different values or remove this altogether if you try with a slower car. You can too build your own car through source code of the Android app, Java Driver app and octave training scripts.