Streaming and graphing real time data with the Raspberry Pi

So you have a fancy little sensor/contraption and its working great, the system is portable maybe even in a waterproof case or is mobile with no room for a display. However, how do you know the quality of your data or if the system is working without carrying a display or connecting up an SSH session? A black and white terminal impresses no one and is a terrible way to view trends in data.

There is an answer and it is fantastically flexible. You turn your Raspberry PI into a wireless access point and web server. Then with a little HTML, JavaScript, JSON and the chart.js library, you have a clean and dynamic interface for your project.


The particular project in this example uses the following hardware:


When assembled and strapped to a buoy anchored in the swan river it is capable of logging and analysing the motion and frequency of waves. This data is collected as part of the research for Woodside’s River Lab run in partnership with UWA’s Oceans Institute. Here you can see the real-time data stream from the buoy in a lab environment.


The Sensor

The BNO055 sensor performs all of the internal calculation and compensation to output clean acceleration and heading information, however as a demonstration to students and visitors the data is hard to visualise when it must be copied and charted from a CSV file. To improve the accessibility of the data and increase its value this tutorial will utilise several scripting & markup languages and several key libraries:

  • Python
  • HTML

  • CSS
  • JavaScript


Configure the Raspberry Pi

To configure the Raspberry Pi and the website there are several steps. Keep in mind any existing configuration or modifications to a vanilla Raspbian Jessie install may have unforeseen ramifications to this tutorial. Thankfully for my fingers many of these steps are already well documented, the links below will take you to the tutorials I used for some of these steps.

  1. Install Apache Web Server
  2. Configure Raspberry Pi as an Access Point
  3. Modifying your python script
  4. Configuring your script to run on boot
  5. Writing a webpage with chart.js

Having configured your Raspberry Pi as a hot spot web server as per tutorials 1 & 2 above, you should be able to load your web page on a laptop or mobile phone connected to the access point according to the IP address you configured in the second tutorial. If you followed the tutorial verbatim this is 172.24.1.1 although in my application I configured my /etc/network/interfaces AND /etc/dnsmasq.conf files with the address 192.168.0.1 as that is common across other hardware we develop.


Modifying your python script

To improve the portability of our data and leverage existing data handling tools built into python and JavaScript we need to output our sensor data as a JSON file. It is a simple and human readable file format for structuring data reminiscent of a dictionary/list within many languages. Python has some built-in functions that make it easy to output a dictionary as a JSON file. In this example, we want to create one file that details the current sensor reading as well as the calibration state of the sensor.

Most importantly, our JSON data is written to the directory for our web server /var/www/html/whatever.JSON  This means our web server can share the file when it is requested by a client. Any JSON file can be placed and requested natively via http://192.168.0.1/whateverTheFileNameIs.JSON this is great for viewing configuration information and system states without the additional effort of writing and formatting a full web page.

JSON can be viewed directly in the browser with no html required. Great for checking system states

Running the script on boot allows for truly headless operation. Although some form of visual feedback that the code is running, such as flashing an LED, will save any head scratching should the Pi fail to boot or script fail. To run the script on boot we will be modifying the /etc/rc.local file again with another line.

 (sleep:10;python /whatever/the/file/path/is/script.py)& 

As all operations within the rc.local file are already executed with privileges you will not need to include sudo in the command. Full copies of the source code used in this example can be found here.

A word of caution. As you are writing & reading the same file concurrently your code must be tolerant of not being able to access the file on occasion.

Now we have everything configured on the raspberry pi and our data is streaming we can start to build our chart.js data interface.


Andrew Van Dam

An electronics expert based in Australia dedicated to creating great tools for the maker community.


A Brief History of the Raspberry Pi

Who would have thought a US$35 computer would change our lives so much. The Raspberry Pi foundation have changed the way we learn, teach and innovate by bringing a hardware platform which is cheap enough for the ‘average joe’ yet powerful enough to drive some of our most ambitious ideas. I have no doubt in my mind that I have left some info out here. Please feel free to leave a comment below to educate me a little bit! happy for input. I’ll keep updating this timeline as new releases come out. Thanks.



Some additional credits for the pics and info: Adafruit.com, Raspberry Pi Foundation, elinux.org

Who would have thought a US$35 computer would change our lives so much. The Raspberry Pi foundation have changed the way we learn, teach and innovate by bringing a hardware platform which is cheap enough for the ‘average joe’ yet powerful enough to drive some of our most ambitious ideas. I have no doubt in my mind that I have left some info out here. Please feel free to leave a comment below to educate me a little bit! happy for input. I’ll keep updating this timeline as new releases come out. Thanks.


The Alpha Boards

Announced in August 2011, the ‘Alpha’ boards were among the first revisions. The Board itself was of a large format for debugging, was populated with headers for GPIO, JTAG, DSI, CSI and switches for LED and I/O testing.


Raspberry Pi First Production Board

Announced in 29th February 2012, The first of the “credit-card” sized computers was released. The initial batch of 10,000 (All Model-B’s) which were manufactured in China.


Raspberry Pi Model B

The RPi full production board – Model B Rev 2.0 was announced in April 2012, first orders were not sent out until June 2012. Manufacturing and orders were made via RS and Farnell. This meant that higher volumes of units were produced and made available. September 2012 saw the announcement of Model B rev 2.0 also being manufactured by Sony.


Raspberry Pi Model A

As of February 2013, the Model A was made available from distributors in Europe. The board had 256MB of RAM rather than the originally planned 128MB. The RPI foundation produced a small Beta batch of boards before handing over manufacturing to RS and Farnell. The Model A boards use an identical Rev 2.0 PCB but come with a different selection of components fitted.


Raspberry Pi Model B+

Announced on the 14th July 2014 The Model B+ was the first of the new layout. The B+ featured a new and faster processor, new VideoCore, 512MB RAM, Micro SD, 4 x USB slots, HDMI and many other upgrades.


Raspberry Pi Model A+

A few months after the release of the B+, the RPi foundation released a lower-cost variant labeled ‘A+’, At a new low price point of US$20 it was a smaller form and only a few components had been left out. But from an IoT perspective, it was still a very powerful unit for embedded hardware projects.


Raspberry Pi 2 Model B

Announced n 2nd February 2015. The RPi 2 Model B+ retained the same US$35 which was key to the continued success of the platform. The RPi 2 featured another upgrade to the CPU and an upgrade to 1GB of RAM. Other changes were also made.


Raspberry Pi Zero

The Raspberry Pi Zero was released in November 2015. The idea behind the Zero was a cheaper/smaller form factor that could be used to embed inside smaller projects. On 28th February 2017, the foundation then released the RPi Zer ‘W’ which had built-in wireless and Bluetooth.


Raspberry Pi 3 Model B

Another upgrade to the CPU (1.2Ghz 64-bit quad-core ARM Cortex-A53) was announced with a new release of the RPi 3 Model B on the 29th February 2016. Again the Raspberry Pi foundation achieved the same US$35 price point.


Raspberry Pi 3 Model B+

Announced in “Pi Day”, 14th March 2018. The Raspberry Pi 3 Model B+ is an incremental improvement over the previous RPi 3 for the same US$35 price.


Some additional credits for the pics and info: Adafruit.com, Raspberry Pi Foundation, elinux.org


Raspberry Pi HAT design files

I have embarked on a journey to create a Raspberry Pi HAT for a little project of mine and I wanted to share a couple of things that I think may help you speed up your development time in the future. As of 11/5/16 I have tested out the DXF importing it into KiCad and using as the edge cut profile. The blank PCB’s test HATS we had made up fit nicely on the RPi2. As I push further on this journey I’ll continue to post any design files that I feel could help you with future iterations.

I can confirm that this fits onto the rPi3 also.


RPi HAT Files



Edit files on your Raspberry Pi the easy way

For us Linux “late-adopters” it can be a daunting task of carrying out a routine file transfer to our Raspberry Pi from our Windows PC. However, the learning curve is not so great. There are plenty of easy tools for achieving this feat. If you have not been introduced to Notepad++ then we will give you a quick introduction. We will also educate you on a small plugin inside of Notepad++ that allows you to run a lightweight FTP client. This client is really only suitable for your scripting files or editing configuration files but you will learn the basics of how FTP works and then have the skills to upgrade to a more robust FTP client for moving other files onto the Raspberry Pi.


Prerequisites

  • Install Notepad++ on your windows PC. https://notepad-plus-plus.org/
  • Download Putty. Link Below
  • Ensure your Raspberry Pi is plugged into your network via Ethernet (cat5)
  • Power up your Pi
  • Ensure your LAN has DHCP enabled...most routers will have this set as default.


Finding your RPi IP address

If you already have your raspberry pi on the network and you know its IP address then that is half the battle. Alternatively, if you have just finished installing Raspbian and you are not sure what to do from here then do not panic!! Check this article to find your IP address. http://dirtyoptics.com/find-raspberry-pi-ip-address/


Method 1: Notepad++ (My Favourite!)

Navigate your way to https://notepad-plus-plus.org/ and download the latest release for Notepad++. Install as per any other windows application and run it up. You will notice it looks very similar to the generic windows text editor however it does allow for some syntax highlighting. A handy little editor also if you are just starting to dive into Python and other coding languages.

Once Notepad++ is open, navigate to: Plugins / NppFTP / Show NppFTP Window. (If you do not see NppFTP you may need to go to the plugin manager and install it)

Once you have the NppFTP window open you will need to create an SFTP profile for your RPI. Please note that when SSH is enabled on your Pi it also opens up port 22 for you to utilize SFTP over the SSH connection. (Well that’s my understanding anyhow!).

  • Insert Pi IP
  • Port 22
  • Select SFTP
  • Username: pi (If left as default)
  • Password: raspberry (If left as default)

After you have set it up, click close/save. Find and click the connect button in the NppFTP window and connect to the profile you just set up. After a few seconds, you should be able to view a “windows like” file tree of your Pi. You can also double-click on any text file and edit directly in the Notepad ++ editor. When you click “save” it will automatically upload that file back to the Pi. Particularly useful when editing Python code and you want to run directly on the Pi. This can be dangerous at times if updating important configuration docs. Ensure you back them up first.


Method 2: Using SSH/Terminal and 'Nano' Editor

If you are feeling adventurous, and want to use SSH to edit configuration files then strap yourself in. For the ‘un-initiated’, Linux commands differ heavily from the standard Win DOS commands. You will notice some similarities but for the most part, it’s a bit of a learning curve. We are not going to cover absolutely everything here but the basics for editing files whilst you are in a terminal session. (Accessing you Pi directly)

The first thing you will want to ensure is that your RPI is powered up, Plugged into your network and you know the IP address. Assuming you have installed Raspbian onto your Micro SD and inserted the SD card prior to powering up we can then begin to access the Pi via SSH. (SSH = Secure SHell). This is pretty much a standard way of accessing your Raspberry Pi if you do not have a monitor available. SSH is enabled by default as part of the Raspbian build.

You will now be required to download and open a small terminal program called ‘Putty’ (There are other, but this is the most popular) http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html run the putty.exe from your PC and input the following information.

  • Hostname/IP Addres - Insert your RPI IP
  • Port 22
  • Select SSH

Now that you have logged into your Raspberry Pi via SSH its time to start Nano. Nano is a Linux command line text editor. It is pretty simple to get running and use. It can be run in two different ways.

sudo nano

This will create a blank text entry. The correct syntax to follow is:

sudo nano /path/to/filename

If you use a path that is not valid or it cannot find the file you want to edit, then a blank entry will be created.

If you wanted to edit the Raspberry Pi config file, then the command would look like:

Now that you have accessed the config file with nano, you can go through and make your changes. Use the arrow keys to navigate through the text file, and the usual backspace/enter to move things around.

It would be wise to make a backup file of this configuration first.

Once complete, hit CTRL-X to exit, then Y to save. It will overwrite the old file with your new one.



What Next?

  • Download and try winSCP to transfer images and larger files.
  • Setup an FTP server on the Raspberry Pi.
  • Utilise a standalone FTP client to connect to the Raspberry Pi. (Filezilla/cuteFTp etc etc)


32 x 32 LED Matrix setup on RPI2

After building a few LED matrices from LED strip lighting and soldering a ton of connections, I decided to give these pre-fabricated LED matrix modules a shot. After a bit of research it turns out that using only a Raspberry Pi and a fully functional Library from the Legend Henner Zeller, you can accomplish just about anything. The cheap Chinese panels that we sourced are fitted with a HUB75 connector which is easily interfaced with a breadboard and some jumper wires or using the opensource Active-3 board, again designed by Henner. His library can be found here: Rpi-RGB-LED-Matrix Library – https://github.com/hzeller/rpi-rgb-led-matrix


Bill of Materials (BOM)

  • LED Matrix: http://www.aliexpress.com/store/1239156 (P5 32x32 modules with Hub75 are a good starting point.)
  • A Raspberry Pi 2 or 3
  • A breadboard and T-Cobbler RPI 40 pin Breakout (Just to make life easier! You can grab these from Adafruit.)
  • Alternatively build an Active-3 adapter for easier chaining of Matrix Panels. (See hzeller github page for more details.)
  • Some Jumper Wires
  • A sense of adventure.....


Code

Assuming you are semi-proficient with Linux and have installed Raspbian than you can follow along here. Otherwise you first need to setup your RPI and access the terminal/SSH interface. The guide here will get you up and with the “NOOBS” installation for your pi.

Install Henner Zeller’s LED Matrix library onto your Pi:

sudo wget https://github.com/hzeller/rpi-rgb-led-matrix/archive/master.zip

Unzip the Archive:

sudo unzip master.zip

Once unzip completes you should then be able to view the directory and it’s contents:

cd rpi-rgb-led-matrix-master/

You then need to compile the library by running the command:

sudo make

Once compiled, you can run the following command, this will give you an output of all the available switches:

sudo ./led-matrix

This command will output the following for your reference:

$ sudo ./led-matrix
Expected required option -D <demo>
usage: ./led-matrix <options> -D <demo-nr> [optional parameter]
Options:
        -r <rows>     : Panel rows. '16' for 16x32 (1:8 multiplexing),
                        '32' for 32x32 (1:16), '8' for 1:4 multiplexing; Default: 32
        -P <parallel> : For Plus-models or RPi2: parallel chains. 1..3. Default: 1
        -c <chained>  : Daisy-chained boards. Default: 1.
        -L            : 'Large' display, composed out of 4 times 32x32
        -p <pwm-bits> : Bits used for PWM. Something between 1..11
        -l            : Don't do luminance correction (CIE1931)
        -D <demo-nr>  : Always needs to be set
        -d            : run as daemon. Use this when starting in
                        /etc/init.d, but also when running without
                        terminal (e.g. cron).
        -t <seconds>  : Run for these number of seconds, then exit.
                        (if neither -d nor -t are supplied, waits for <RETURN>)
        -b <brightnes>: Sets brightness percent. Default: 100.
        -R <rotation> : Sets the rotation of matrix. Allowed: 0, 90, 180, 270. Default: 0.
Demos, choosen with -D
        0  - some rotating square
        1  - forward scrolling an image (-m <scroll-ms>)
        2  - backward scrolling an image (-m <scroll-ms>)
        3  - test image: a square
        4  - Pulsing color
        5  - Grayscale Block
        6  - Abelian sandpile model (-m <time-step-ms>)
        7  - Conway's game of life (-m <time-step-ms>)
        8  - Langton's ant (-m <time-step-ms>)
        9  - Volume bars (-m <time-step-ms>)
        10 - Evolution of color (-m <time-step-ms>)
        11 - Brightness pulse generator
Example:
        ./led-matrix -t 10 -D 1 runtext.ppm
Scrolls the runtext for 10 seconds

Now its time to get some output onto the panel. If you are running a singular 32×32 panel, you should be able to run the example without issue.

sudo ./led-matrix -t 10 -D 1 runtext.ppm

If your panel is connected correctly and powered up you should see a scrolling image pass through the panel. With the example above it will only last 10 seconds. You can now begin to experiment with the above switches to see what output you can achieve.


Advanced

  • Output RPI GUI to Matrix Panels (Coming Soon!)
  • Output Twitter msg to Matrix Panels (Coming Soon!)
  • Use Pixelpusher Protocol on Matrix (Coming Soon!)


Media