Fork me on GitHub

Python library for the PCF8475 on the Raspberry Pi

Posted by
Matt Sieker
in RaspberryPi on Sat 13 December 2014

While there was progress on my environment monitor project tonight, I'm going to write that up later... er, today, considering it's after 4AM currently. I'm going to do a quick writeup instead on a side project that is related.

Part of my project is interfacing with a pair of PCF8475 I2C IO expander chips to control a relay board, and read buttons. To this end, I've created a small Python class to interface with these chips, available on Github. It supports basic reading and writing to the chips, along with being able to trigger a callback when the /INT line is triggered. Usage is quite simple:

Eventually I'm going to add the ability to toggle a specific pin, while maintaining the state of the other pins.

There are comments.

Raspberry Pi Environment Monitor - New Sensors and a Web API

Posted by
Matt Sieker
in RaspberryPi on Thu 11 December 2014

Now that my shipments from China, Digikey, Adafruit and other parts of the world have arrived, I should probably write an update on this project.

First off, I have some MCP9808 temperature sensors. These have greatly improved resolution over the DHT11 I was using before, as shown by this chart:

A very colorful chart with wiggly lines.

The left bit of data is when the DHT was used for both temperature and humidity. The flat line is when the Pi was powered off for a bit, and then it resumes with the DHT still used for humidity, but the MCP used for temperature. It is now very clear when the furnace turns on and off. You could make it out in the previous data, but with much greater resolution.

To interface with the new sensor, which is I2C based, I once again turned to Adafruit with their Python Library for the MCP9808. Integration basically amounted to just putting a call to it when I read my sensor:

Since the DHT sensor is the slower, I let that drive the rate of sampling. Then I disregard the temperature information from the DHT, and just read the MCP sensor, and log that. The same 10 second rolling average is there to smooth the data.

Now that I have this data, I need some way to get to it. I started work on implementing a somewhat RESTful API using Twisted. There are endpoints for sensor and weather data currently, that are set up like this:

I can pass how much data I on the query string, compute the number of minutes from that, and pass that on to my Redis wrapper. For easier consumption, I add a string version of the date and time to the outgoing data. To make this data easier to consume, I made the helper function renderCsvOrJson:

If application/json is not explicitly in the Accept header, I return CSV data with instructions that it is an attachment, so the browser downloads a CSV file. This lets me, for now, easily make my charts by just visiting the endpoint and waiting for the CSV to generate and download.

On the topic of generating data sets, once I implemented my API, I noticed that it took excessive amounts of time to generate the data. This was traced down to the N+1 round trips to Redis to get the data:

Knowing that Redis had a Lua scripting interface, I turned to this. But then I discovered that Raspbian has a rather old version of Redis... from before they added Lua. This lead to the fun process of updating my Pi to run the Testing branch, which took ages to install the packages. Once this was done, I wrote my first Lua, ever. And reading the documentation, it seemed that msgpack was the best option to pass data between Redis and Python.

With this I can pull a weeks work of data in the time it took to pull 8 hours of data previously.

And now, and email I just got leads me to mention something else I've worked on: A screenshot with some stuff.

I decided to play with KiCad last week, and try my hand at designing a PCB. The board itself is a rather simple IO expansion, with the pair of PCF8574s adding pins for buttons and the relay board, with a header for the new 320x240 LCD display that came from China. I decided to design this board for the Raspberry Pi B+ so I had access to more IO, and better placement of the Vcc pins for both +3.3v and +5v. I've ordered some extra-tall stacking headers for this board so I can plug my breakout cable into this board. This will clear up a bit of the rats nest on my breadboard, and the large amount of space taken by the LCD. The board itself is 2mm larger than the Pi in both width and height, to give me a slightly easier time routing, and to actually give me room for the buttons (Which, I just noticed the ground and power fills are not even across the bottom of the board. Oh well, that's what v1.1 is for, along with moving the resistors to the middle of the board vertically). Considering I never did PCB design before, and I was able to make a board ready for fabrication in two days, the learning curve for Kicad wasn't that bad though, it does have its quirks.

For fabrication, I remember reading about Dirty PCBs awhile back on Hack-A-Day, and decided to use them. It is somewhat odd that they offer cheap 5x5 boards, which are smaller than both the Arduino shield, and the Pi. It's still the cheapest option though. My board is here if for some reason you want an amateur designed Raspberry Pi IO expansion board.

There are comments.

Raspberry Pi Environment Monitor - Push Updates and Sensors

Posted by
Matt Sieker
in RaspberryPi on Tue 02 December 2014

With things being put on the display, the ability to update that display with live data is needed. Most of the infrastructure is already in place, it just needs to be finished up and tied together. And also, my first shipment of random tat arrived today, including my DHT11 sensors and the relay board.

The Redis wrapper gained a new member GetDict that loads a hash from Redis, and had Subscribe implemented. Subscribe takes a callback method that gets passed into an instance of RedisCallbackSubscriber, which inherits from txredis' RedisSubscriber.

The display manager gained a callback method that gets passed into Subscribe:

Simple matter of looking at the channel, and pulling the hash out of Redis, and putting it in the right place. And now I have a very simple current conditions display on my little LCD.

During sensor implementation, since the script needs to run as root to have access to GPIO, pygame behaved badly. I'm not quite sure the root cause, it seems to be some interaction between pygame and twisted, but it would get wedged to the point that SIGKILL does nothing. For now, in the interest of making things work, I've moved pygame initialization out into my main script, and added a LoopingCall to pump pygame messages. This at least lets me handle keyboard keys now, and the program will actually quit. I don't like it, since I can no longer run the display manager on its own.

My current sensors are DHT11's. I got a pack of 4 cheap on Amazon. Then I'm using the DHT-sensor-library put out by adafruit.

Most of it is just a bunch of setup code. The only interesting bits involve deferToThread to keep the blocking sensor poller from blocking twisted, and also returning a deferred from LoopingRead to ensure twisted will not re-schedule the method until the sensor read is done. The other interesting bit is using a deque as a circular buffer to calculate the rolling average of the sensor readings over the past n readings. Then, once every 10 seconds, the rolling average is calculated, and shipped off to Redis.

With the sensor polling added now, Python sits at about 10% CPU usage, with SPI writing to the display taking about 6%. These numbers still seem acceptable.

However, one thing I have noticed with these DHT11s is that it doesn't seem like the temperature changes all that much. Although, I'm not sure if that's an actual thing, or just lack of resolution on this sensor. The temperature does change when I breathe on the sensor, so I know it's working. I'll probably spring for a more expensive DH22, or perhaps use the DH11 for humidity, and get a DS18B20 or something. Either way, there's a lot of other work to do before I hook this thing up to my thermostat.

There are comments.

Raspberry Pi Environment Monitor - Displaying simple stuff

Posted by
Matt Sieker
in RaspberryPi on Sun 30 November 2014

Now that I have data, I need to be able to see that data. Time to play with PyGame and the frame buffer.

After much reading of the PyGame docs, I noticed the oddity that text is always drawn on its own surface. Not a major thing, but different from most things I've dealt with in the past. This also involved a long wait to install the Droid font family. Droid Sans Mono has been my favorite monospace font for awhile, and looks good at small font sizes, which is what I'm dealing with here on this 220x176 screen.

The frame buffer setup code is basically ripped from the Pointing Pygame to the Framebuffer sample on Adafruit, with a bit of added code to specify the frame buffer, set up my drawing surface and hide the mouse cursor.

Currently I have the display refreshing about twice a second, mainly to ensure I don't wait too long drawing the clock displayed, otherwise that looks kind of strange. With that refresh rate, the SPI driver uses about 6% of the CPU on the Pi, with Python using another 5%. Screen drawing should be my highest continuous task, as most other tasks I have planned for this to do range from one every few seconds to once every few minutes. Though, these numbers will probably go up a bit once I get my new display that's a whopping 320x240.

A bad photo of some text on a screen

This display looks much better in real life. I should get an overhead lamp on my work table.

I need to work out a good way to make it easier to set up what goes on the screen where than hard-coding all of it. But that can wait until I have the display with the size I'll be using.

The display does not yet update the weather as the weather updater pushes the updated conditions out, since I do not have the subscribe parts of the Redis wrapper implemented. That will probably be next on my list of things to do. It does, however, use a new method on the Redis wrapper to do the initial load of the data, GetLatestTimeSeriesMember:

This call will remain even after I have the push updates implemented, since the latest update might not be available when the display first draws.

There are comments.

Raspberry Pi Environment Monitor - Forecast.io Integration

Posted by
Matt Sieker
in RaspberryPi on Sun 30 November 2014

With Redis support in place, it's time to create the first thing to write into the store: The weather forecasts.

As stated earlier, I decided on using forecast.io due to their easy (and free for the most part) API. Most of the code for the forecast downloader involves talking to Redis or dealing with Twisted.

Originally I was going to exclude the daily forecast bits from the API call, but then I noticed by reading the docs that some interesting bits of information are in there, like sunrise and sunset times, along with daily highs and lows. As I'm not too interested in displaying an extended forecast on this thing, I'm just grabbing basic data I'm interested in from the first daily data element returned, which is the current day.

Once I have the data I'm interested in, I push it into Redis, add it to the time series sorted set, and push out to subscribers that there's been an update.

Now that I have data, I need to actually do something with this data. To the display!

There are comments.

Raspberry Pi Environment Monitor - Twisted Redis

Posted by
Matt Sieker
in RaspberryPi on Sat 29 November 2014

In the first part, I settled on using the Twisted framwork in Python, along with using Redis as a data store. There's a Redis library for Python, but I was unsure how to tie that into Twisted. But, it turns out that, once again, someone has done the hard work with the txRedis library.

My first goal is writing a wrapper around txRedis to expose the functionality I need. In the process of learning txRedis, I learned about the inlineCallbacks functionality in twisted that either wasn't there the last time I did something with Twisted (at least 3 years ago), or I ignored then. It seemed odd at first, until I realized it was essentially the async/await pattern in C#.

For my wrapper, I want to do the following things:

  • Store a dictionary, returning a unique key
  • Add a key to a sorted set, with the score being the current time.

    This will allow me to query a list of objects within a store during a time frame (say, the last 10 minutes). Which leads to the next function.

  • Query a store for items during the last n minutes.

  • Publish a new item for consumers to act on
  • Listen for new published items

This, with the exception of the subscribing, is implemented thusly:

The settings object passed into the __init__ is a dictionary created by merging together a pair of yaml files. One for non-private data to act as a template, the other for private data (API keys, locations, and so on)

With this, the basic outline of how I would like my redis interface to look like is complete. Next up is the first consumer of this class: The weather logger.

There are comments.

Raspberry Pi Environment Monitor - Intro

Posted by
Matt Sieker
in RaspberryPi on Fri 28 November 2014

After seeing this Raspberry Pi Thermostat Controller on Hackaday, I started scheming. I've had issues with my thermostat being a POS, and since it's an apartment, I really don't want to replace the thermostat with a new one. I also had an Adafrut 2.2" TFT laying around, looking for a use.

The first step was getting the display hooked up. Luckily, someone else has done the hard work with the fbtft kernel module. A quick breadboard later (and realizing that the SDA pin on my Pi breakout board was the I2C clock pin, and not the SPI clock), I had this:

Some rats nest on a breadboard

Which my phone camera really does not like taking a picture of that display in the low light of my office. After some fiddling with the rotation of the display to get something natural for how I plan on looking at it. Next up was deciding what I wanted it to do:

  • Grab the local weather conditions.

    I wanted something to get the current temperature, humidity, pressure and other statuses, with an easy to use API, update the results periodically, and shove them into a data store. These should also be displayed on the LCD, perhaps with a graph (and some buttons to cycle through graphs)

  • Read sensor data

    I have sensors on order for temperature and relative humidity, along with a pressure sensor. I want to be able to read these, and shove them into some sort of data store. The temperature sensor will also feed into the thermostat control. I'd also like to expose this data over a web interface. This should also be shown on the LCD.

  • Control my thermostat

    Take the temperature data and use that to control relays hooked up to my thermostat, like in the hackaday linked project. Also have the status and controls for this over the web.

  • Support for remote sensors

    I also have some ESP8266 boards on order, and I would like one of these in my bedroom so I don't freeze my ass off when it gets cold, since my bedroom is the furthest room from the thermostat. I would like these to post sensor data to the main unit over some sort of RESTful interface.

Since everything but the first relies on parts I don't currently have, I'll concentrate on grabbing the local weather data, logging that to a store, and drawing it to the LCD. I can also wire up a basic web interface once that is done.

So, on the software side of things, I've decided on the following:

  • Python
  • PyGame for drawing to the frame buffer
  • Twisted for handing HTTP requests (both incoming and outgoing), along with message passing and timed tasks
  • Redis for a data store

The various bits will talk amongst themselves using Twisted's producers and consumers.

For the weather reporting, I've decided on forecast.io. It has a rather simple JSON interface, and allows 1000 API calls free per day, which considering I'm planning on updating the outside weather once every 5 minutes, that works out to less than 300 requests per day, so I'll be well and good using their API.

Next up, getting the weather data and doing stuff with it.

There are comments.

Movies on Stellaris - Part 1

Posted by
Matt Sieker
in Stellaris on Sun 17 February 2013

So, I got a Stellaris Launchpad the other day, and was wondering a project to do with it. After a bit of thought, and looking at the various parts I had sitting around, which included a TFT LCD Display with an SD Card, and thinking back to a project I'd seen before, I decided for my first project, I'd see how fast I can pull data from the SD card and write it out to the LCD (both over SPI).

The video file? Since I didn't want to write any complex decoding at this point on the board, I decided to have a basic format that's RLE encoded. And what video would encode quite well with RLE? I'm glad you asked:

Simple black and white. I'm not going to worry about the audio right now.

The first step: How on earth to read the video, and then write it back out encoded? The answer? OpenCV and Python. A half dozen lines to read the video frame by frame, smash down the color depth (not really needed right now, but it's there in case I want to process the image further at some point), and then shove the image into a byte array.

For testing, I wrote a small C# program that reads in the files one by one, creates an image out of them, and shoves them into a picturebox on a form. Also simple.

A sanity check: Can I take the video frames, encode them via a brain dead method, and read them back out?

Image

Yep. Although this is about the most brain dead method possible: Each pixel is two bytes, one indicating a run length of 1, the other storing the value of the pixel.

Now, let's try using RLE encoding to get something out of it. The RLE variant I use works as follows:

Use itertools.group to get byte runs in the data

Each group back from itertools:
   Is the length greater than the minimum run?
      Flush the small run buffer
      Write out run length + 128
      Write out byte value for run
    else
      if small runs buffer + current run >=127?
         Write out run length
         Write out small runs buffer
      Add run to small runs buffer

Very rough psuedocode. Essentially, if a byte value > 127, the byte value - 128 is the length of the run, with the next byte being the data byte. If it's < 127, the next n bytes are treated as literal. There's some added logic if the run length is greater than 127 where it figure out how many 127-length runs to write out.

Overall efficiency isn't the best of course, since this is a rather simple method of compression. The largest frame at 23011 bytes is this one:

Image

Lots of grays and shading. Closely followed by this one at 23009 bytes:

Image

Once again, rather expected worst case. I've attempted some dithering in the pipeline to help. Overall size for 6568 frames is about 40MB, reduced to a resolution that will fit on the LCD display I have. The video runs at 30fps, I plan on reducing that to 15fps at first, which would reduce the size to roughly 20MB. So, my target transfer rate off the SD card over the SPI bus is about 93KBps. Some preliminary research puts this in the "very possible" range.

I plan on joining together the loose files in a rather simple format of (file length)(file data)(file length)(file data) , etc, with the final length being 0. This gives roughly 6.5kb of overhead, since I can cram the file lengths into 16 bit ints.

Next up, poking at the chip itself.

There are comments.

Using Visual Studio for a node.js project

Posted by
Matt Sieker
in Coding on Sun 22 July 2012

I recently ran across the need to be able to work on some node.js code from within Visual Studio (The rest of the project is .Net, with node being used for a few unique features). Previously I had done this work over a SSH session to Emacs, but I decided I wanted something more integrated.

My first task was to make a project file that would work in Visual Studio, with the build tasks being whatever I wanted to do to my Javascript, and nothing else. To do that, I first chose to create a blank MVC project, then delete all of it's contents and references. Then I opened up the project file and removed the following line:

<Import Project="$(MSBuildToolsPath)\Microsoft.CSharp.targets" />

And then added the following:

<Target Name="Build">
</Target>
<Target Name="Clean">
</Target>
<Target Name="Rebuild" DependsOnTargets="Clean;Build">
</Target>

Then the project was reloaded in Visual Studio to make sure all was well. I then added my Javascript files to the project, ran a build, and make sure Visual Studio didn't complain about anything.

Next up was the ability to run JSHint against the code in my project, while excluding pre-minified files. I already had node.js on my path, so I installed jshint as a global package with

npm install jshint -g

After that, I grabbed the jshint reporter that worked with Visual Studio from the node-jshint-windows project and dropped it into my build libraries folder. Then I modified the Build task as follows:

<Target Name="Build">
    <ItemGroup>
        <ToLint Include="@(Content)" Exclude="**\*.min.js" />
    </ItemGroup>
    <Exec Command="jshint --reporter &quot;$(SolutionDir)libraries\build\vs_reporter.js&quot; %(ToLint.Identity)" ContinueOnError="false" />
</Target>

This takes any files within the Content ItemGroup created by Visual Studio, excludes pre-minified files, and then runs jshint against them. The errors will then show up in the Error Output window just like any other sort of build errors. If desired, ContinueOnError can be changed to true, if you would rather JSHint errors be reported as warnings.

There are comments.

Using the Sparkfun LCD shield with the MSP430 - Part 2

Posted by
Matt Sieker
in MSP430 on Sun 15 July 2012

First, I now have the code for this on Github: MSP430NokiaLCD. Several things have changed in this code from the the earlier posting:

  1. SPI bit-banging loops are unrolled. I wish there was some way I could get rid of the if in there...
  2. Some drawing functions are now in place: LcdDrawPixel, LcdDrawLine, LcdDrawRect. Most of these functions come from James Lynch's documentation mentioned in the previous post, modified to work with my bit-banging routines.
  3. After finding the Universal Color LCD graphics library on the 43oh forums, I initially tried swapping out my library with this one, and seeing if I could get
    it to work with my LCD. No dice. However, looking through it, I modified my library from 12-bit color to 8 bit palletized color. The loss of color depth probably isn't noticeable on these screens that much. And there's less data to move across the wire.

I've got a bouncing square going, and I think I have it animating about as fast as I can without too much tearing or blurring. I think it's about as best as I'm going to get out of this LCD:

Here's cycling through the palette:

Compared to the previous 12 bit code:

It's a decent improvement.

Next, I'll probably work on text rendering. I seem to recall a minimal font where each letter was made up of a handful of pixels, but now I can't remember what it's called. I might just use Droid Sans Mono for it.

There are comments.