Learning from running out of memory all the time
HARD. WARES.
Attention: As of January 2024, We have moved to counting-stuff.com. Subscribe there, not here on Substack, if you want to receive weekly posts.
Over the past couple of weeks, I’ve been on a tiny electronics binge messing with little projects around the house. My electronics skills are fairly basic, on the level of soldering wires to redirect power around and occasionally putting components specified in kits onto boards, so they’re a bit simplistic.
One project was wiring up an under-desk box that took audio from 2 computers in RCA, lets me switch freely between them, while also splititng the signal to output in over-complicated ways to a volume-controlled set of speakers and headphone amps simultaneously. That allowed me to remove a horrific rat’s nest of splitters, switch boxes, and ridiculous stacks of cable adapters from my desk.
Slightly more advanced was learning to solder tiny 3mm sized surface-mount LEDs to upgrade the ~$15 Lite Brite toy that we had gotten for the kid. The toy itself only had 6 LEDs in it that led to some overly dark spots, so I did some research how LEDs are wired and added 2 more into pre-made slots in the casing. It’s still not evenly lit enough for my tastes but a further upgrade would involve putting an LED strip light inside it.
Then I learned about “buck converter” circuits that you can buy for around $1, that’ll take one voltage and switch it to another, letting me convert a battery-powered thermometer in my window into a USB powered one.
Finally, I gave in to temptation and did the thing I’ve avoided for years — working with software controlling hardware. I bought a 64x64 LED matrix board and an Adafruit Matrix Portal board and am in the process of getting it to show the weather and forecasts for later in the day so that I know what to wear and pack if I go outside.
Manipulating hardware takes a lot of learning
Despite being quite handy with computers and DIY projects, I’ve been avoiding software/hardware projects for years now because it touches upon a surprising amount of knowledge.
The biggest barrier to me was how you have to actually come to a limited understanding of things that you’d never have to think about as a pure software programmer. For example, sending or reading data through a physical wire actually means sending pulses of voltage out a specific I/O pin on the circuit board. It also means you have to understand what voltage is being sent (is it 3 volts? 5? Or something else entirely). That voltage may be compatible with the device on the other end of the wire, or it needs further electronics to change the voltage.
You then have to learn pesky details related to what you want to do. Motors and AC voltage? LEDs? Various sensors that send data in certain ways? How the heck do you “read” a value off of a black chip package anyways? So you have to think about the protocols the signals are being sent. While there are semi-standard protocols and libraries to handle most common things, you still have to understand them well enough to at least be able to read spec sheets and buy compatible parts as you design and plan your project. You even have to understand how to connect to the device itself via serial console to even manipulate things.
Finally, after you dive into all that fundamental knowledge you need in order to do blink an LED light, you finally have to learn a new programming environment. Either something C++-flavored in the base Arduino programming language, or Python environment running with hardware constraints so tight it’s like you’re working in a different universe.
The primary reason why I bought the Matrix Portal board that I linked to above is because it simplified a TON of that learning for me. Adafruit’s got a lot of example code and tutorials for hooking up to the board and running CircuitPython on it. (CircuitPython is Adafruit’s flavor of MicroPython made for handling their boards.) The board is also designed to run the weird Hub75 protocol that’s typically used to run these cheap LED matrix boards (and conveniently plugs right into the port). It can power a single panel from a sufficiently beefy USB-C power source.
Those nifty features simplified the learning curve for this project down to “buy the things for about $80, hook them up, start coding”. Otherwise, I’d still be reading specs trying to figure out what play swell with what.
I keep running out of memory!
So as a data scientist that has been very comfortable with using Python for random tasks the past 20 years now, you’d think that manipulating a LED board by modifying example code would be a breeze. And for the most part, it was pretty easy to do, and I had a functioning demo within a couple of hours of tinkering.
Then I started hitting into issues.
The biggest issue is most definitely memory. The board has a chip with 192kB of SRAM memory — that’s all the RAM we have for code execution. It’s not even close to 1 megabyte! In my normal work, program memory is cheap because they’re effectively rounding errors compared to the data sets that I have to manipulate. If I’m running out of memory doing an operation, it’s always the dataset’s fault (or an accidental duplicate copy of said dataset).
But on this embedded device, every tiny detail matters. Want to load a 64x64 pixel splash screen? That’s going to cost some bytes. There’s 4096 individual pixels that need to be addressed and controlled on a 64x64 panel, that take distinct R, G, and B values to set the color — more bytes. The demo code specifically loads up the individual glyphs from the font you’re using to display so that you don’t waste memory on extended ASCII characters you likely won’t be needing.
In fact, memory got so tight that when I wanted to convert the unixtime value that the OpenWeather API gave so that I could extract the time for a given forecast, the 19kB datetime library wound up taking my program over the limit and it crashed with a memory error. In the end, I had to google up a native Python function that did the conversion and paste it in to my code to do the job
What’s worse is that 192kB of RAM is actually pretty darn large for boards that are roughly with Arduino code. The CircuitPython interpreter needed to parse my code obviously needs much more room to run on. If you look around, many other boards based on various other microcontrollers come with RAM as small as 2kB, while 32kB boards are pretty common for official Arduino boards. You really have to pay attention to what your needs are and select your development platform based on features and price.
Memory isn’t the only other constraint
While having my code flat out crash due to memory is a big issue, there’s other thorns to deal with.
The first is handling errors. There’s literally no space on the device to log issues with my code. The code has to be able to swallow and recover from any exceptions it might throw when plugged in, or it just stops working entirely and I won’t know why because there’s no stack trace to examine. Right now, I work around this by keeping the board plugged into my PC with the serial console window open — that allows my terminal to log any errors in the scroll buffer. I won’t have this luxury once the board is “done” and its hanging on a wall somewhere. Right now I have to leave it on for a number of hours, maybe half a day, before a novel error condition crashes the thing and I have to deal with it
Another consideration is design. I don’t have a design background, and honestly my aesthetic sense for data display is quite weak. But it’s up to me to come up with some sort of usable layout for the information that I might want. You could argue it’s a good thing that it’s forcing me to think about these issues, but it also means the end result is very far from elegant.
Forcing myself to work with these constraints probably good for me
Just like how the constraints of poetry can actually make you more creative as you learn to work around them, I think that forcing myself to get better about working in very resource-constrained environments is a net positive to my life. It’s already pushed me into less-used sections of the Python standard library — how often have you imported the gc garbage collector library into your data science code?
As I take on increasingly complex projects, I might be forced to ditch Python with it’s memory hogging interpreter and adopt the C-like language used for working with Arduino boards just to squeeze more functionality out of a limited amount of RAM. We obviously rarely use C/C++ to do data science, but it’s never bad to learn about allocating memory and handling pointers since the concept forms much of the basis for Python object references.
But eventually, there would come a point where I’d want to do more complex computation than can ever be done on a tiny microcontroller. The only way to do that would be to not do the computation on the microcontroller at all. Instead I’d have to stand up a API server somewhere that will do my custom processing for me and return the necessary result back to my little device. Just think of all the data engineering practice you’re going to get when you need to coordinate two data systems that must communicate to share data. Maybe you run a simple HTTP based service that returns a JSON blob? Or will you be more internet-of-things-ish and use MQTT messaging to send messages back and forth? The possibilities are endless once you consider factors like where would you host your code and how would everything communicate.
It’s a very good way to get some hands on practice with a very scaled down version of what we do in our day-to-day work of taking data measurements from the world, transmitting them somewhere for processing and display.
As a bonus also get to wrestle with the very messy aspects about interfacing with the real world. Any data collection you get is going to be affected by all the physical constraints of a real world sensor. You’d think a temperature sensor is easy enough to implement since there’s tons of example builds and code out there… until you see how the value jumps around during the day, or is reading way too high because it’s too close to a heat source, or it’s affected by the sun.
Then there’s the aspects where it’s not your sensor handling that’s a problem, just the communication system it’s on being unreliable. How do you handle dropped data and connections? What happens when data comes in late? More importantly, what changes can you make to try to minimize such incidents? Normally, most of these decisions fall down to the engineering team and we can usually rely on them to help come up with solutions.
Now we’re our own engineering team! I promise it’s fun giving yourself tickets! It’s sorta hilarious when you’re making code changes on a sensor or a server and wondering if you should somehow enforce a data contract with yourself.
Lower friction entry points
If somehow I convinced you to attempt a tiny hardware project for fun, I highly recommend a couple of relatively entry points.
First, the maker community around small microcontrollers is really prolific, so go out and search for beginner projects until you find one that looks interesting. Maybe it’ll be an LED board like mine, maybe it’s a sensor or a doorbell or a clock, maybe it’s cosplay items with lights. There’s definitely going to be something of interest and doesn’t cost very much to try out.
The thing to keep in mind for a project is that you NEED it to walk you through the whole thing from start to finish. That includes what specific hardware and exact parts to buy, how wire them all up, and exact code examples to get this to work. While most people I know are usually adverse to “vendor lock-in” for our software choices, you actually want to run towards established vendors for a first hardware project. Go with big brand names, Raspberry Pi, Arduino, Adafruit, etc. that have tutorials specifically made for their hardware and a big community of people putting functional examples up. The worst that can ever happen is that the hardware bits you buy are “locked in” to some ecosystem and you can’t reuse it too much. Most of the knowledge you pick up will be transferrable stuff.
You want, to the best of your ability to get to that “this will definitely work” point without spending any money if at all possible. Because you are going to mess up something, put a diode in backwards, reverse polarity on the power, accidentally short out a circuit so nothing seems to work until you remove it. Your first project is goal is to learn how to overcome your own mistakes. Unlike software mistakes, every hardware mistake potentially costs money. You’ll accidentally not notice a tiny detail and buy the wrong part (I have a whole box of wrong parts from aborted projects), or accidentally fry something something with your soldering iron.
Once you have a working basic implementation of your project, THEN you can experiment and embellish upon it and make it yours. By then, you should be comfortable enough to experiment with changing the code or adding a new feature.
Give it a shot, it can be a lot of fun and even make something in your home a tiny bit more fun or convenient. And share it with everyone at our Discord server if you do make something!
If you’re looking to (re)connect with Data Twitter
Please reference these crowdsourced spreadsheets and feel free to contribute to them.
A list of data hangouts - Mostly Slack and Discord servers where data folk hang out
A crowdsourced list of Mastodon accounts of Data Twitter folk - it’s a big list of accounts that people have contributed to of data folk who are now on Mastodon that you can import and auto-follow to reboot your timeline
Standing offer: If you created something and would like me to review or share it w/ the data community — my mailbox and Twitter DMs are open.
New thing: I’m also considering occasionally hosting guests posts written by other people. If you’re interested in writing something a data-related post to either show off work, share an experience, or need help coming up with a topic, please contact me.
About this newsletter
I’m Randy Au, Quantitative UX researcher, former data analyst, and general-purpose data and tech nerd. Counting Stuff is a weekly newsletter about the less-than-sexy aspects of data science, UX research and tech. With some excursions into other fun topics.
All photos/drawings used are taken/created by Randy unless otherwise credited.
randyau.com — Curated archive of evergreen posts.
Approaching Significance Discord —where data folk hang out and can talk a bit about data, and a bit about everything else. Randy moderates the discord.
Support the newsletter:
This newsletter is free and will continue to stay that way every Tuesday, share it with your friends without guilt! But if you like the content and want to send some love, here’s some options:
Share posts with other people
Consider a paid Substack subscription or a small one-time Ko-fi donation
Tweet me with comments and questions
Get merch! If shirts and stickers are more your style — There’s a survivorship bias shirt!