Attention: As of January 2024, We have moved to counting-stuff.com. Subscribe there, not here on Substack, if you want to receive weekly posts.
This week we’re taking a short break from data talk to do some hardware talk. It’s been a while since I’ve wandered off into a hobby topic and is a nice way to refresh my brain. Data will return next week!
The past week or so, I had put together a macropad for myself. Aside from the time needed to pick out the ~$45 kit (#NotSponsored) and order parts like keycaps (blank flat-ish DSA profile) and switches (Gateron red). All told, for $100 and about 2-3 hours of soldering time, I had a working mini numpad replacement. It’s the first time I put together any kind of custom keyboard from parts, and it was an interesting lesson in finally coming to an understanding of how keyboards work.
For anyone who’s been curious about custom keyboards, I’d recommend just diving in because they’re largely foolproof beginner projects. It took a lot of time figuring out all the pieces for this in the abstract when I could have learned much faster with a physical object to test and build to.
How keyboards actually work =O
Maybe you’re like me and have wondered how a keyboard works before and come across this concept of a “keyboard matrix”. I’ve read that specific wiki page multiple times over the years and it never clicked for me how it worked until I put one together.
The gist is that instead of having 1 circuit for every key, which would require a microchip controller to have over 100 pairs of input/output pins, the keyboard is wired up in a grid-like formation using a set of row wires and column wires. The controller continuously iterates through the row lines and send voltage through that row pin. Any keypress would result in closing the circuit and voltage going back to a column. The keyboard controller would know the row/column coordinate of which key was being pressed and know to send the correct code to the computer. Scan fast enough and you’ll get a perfect illusion of completely independent keys.
The major subtlety involved is how, in the case of multiple keys being pressed simultaneously, electricity can flow from one column back through pressed keys into other columns and cause the detection of ghost keypresses. For a simple example, look at the grid above and imagine you pressed a trio of keys so that you activate 2 rows and 2 columns (thus one shared key between them). It’ll form a circuit where power from either two rows will power BOTH columns because the switch lets power flow through them into other columns. Similar confusion can also have nearby keystrokes electrically mask a key from being sensed.
The solution to both is to make sure every key can send electricity to the sensing column while ensuring keys can’t pull in electricity FROM the column. This is done with the addition of a very cheap diode. That way nothing else on the same column can send signals backwards through other switches to cause all the issues of sharing a column with other keys. This is also why, when you’re building a keyboard, you’re soldering a ridiculous number of little diodes everywhere on top of the gazillion keys.
Luckily, building a keyboard requires zero understanding of any of this. You just put parts according to build guides and PCB markings. Once you get the hang of soldering neat joints quickly, you can zoom through all the parts in once quick go.
Now the software
This was the part that was largely a mystery to me before I had a working example. Most discussions about building keyboards goes into all the hardware because soldering seems the scariest part for beginners. All the tutorials typically end with “and then just install the open source QMK firmware on it and program it how you want”. There are other firmware available depending on what features you need and what microcontrollers you’re using. You can even write your own custom one if you were using something obscure.
The keyboard I bought came with a very common “Pro Micro” controller that supports QMK as well as the “VIA” on-the-fly remapping tool. Via was causing issues for me so I went back to hardcoding my key layouts using the compilation process. That way involves cloning the (ridiculously bloated) QMK github repo, and then following instructions to build a specific keyboard/layout.
For some reason, the repo maintainers decided to store every keyboard manufacturer’s community contribution of setups and keymaps in the same repository as the base code. This means you’re downloading tens of thousands of small text files at once. To make things worse, building requires some kind of scanning through the file structure, and the bloat made a simple build take over 6 minutes on my machine. Simply deleting all the keymaps/manufacturers I didn’t care about cut the build time to 2 minutes. If this doesn’t teach us a bit about how to organize our git repos, nothing will.
What QMK and other firmware provide is a simplified way to program disparate microcontrollers and key wiring setups (which depend on the specific PCB you’re using), and then abstract that away so that users can focus more on assigning key codes and other functionality.
As you can see below, it has a bunch of C code to set various flags like which microcontroller pins are defined as rows, and which are defined as columns, as well as where the diode is. Then it creates this LAYOUT array that creates a visual representation of the keyboard for easier key mapping later.
// Much of the base configuration for my keyboard
/* Key matrix pins */
#define MATRIX_ROW_PINS { F5, B2, B3, B1, F7, F6 }
#define MATRIX_COL_PINS { B5, D7, C6, D4, B6 }
/* COL2ROW, ROW2COL */
#define DIODE_DIRECTION COL2ROW
/* This is a shortcut to help you visually see your layout.
* The first section contains all of the arguments representing the physical
* layout of the board and position of the keys.
* The second converts the arguments into a two-dimensional array which
* represents the switch matrix.
*/
#define LAYOUT( \
k00, k01, k02, k03, \
k10, k11, k12, k13, \
k20, k21, k22, k23, \
k30, k31, k32, k33, k34, \
k40, k41, k42, k43, k44, \
k50, k51, k52, k53, k54, \
\
BACK00, BACK01, BACK02 \
\
) { \
{ BACK00, k00, k01, k02, k03 }, \
{ BACK01, k10, k11, k12, k13 }, \
{ BACK02, k20, k21, k22, k23 }, \
{ k30, k31, k32, k33, k34 }, \
{ k40, k41, k42, k43, k44 }, \
{ k50, k51, k52, k53, k54 } \
}
...
Then, in defining the keymap, you can create this concept of “layers” (think of the Fn or Shift key you might have on your keyboard and how it lets the same key do multiple things). You can easily have 16 layers of functionality, stacked on top of each other with some being toggled on, and others being momentary based on another held key. You can get super creative with them if you’re so inclined.
My goal was to make a device that would let me flip through pages easily in bed (the wireless keyboard I have ridiculously makes pgup/pgdn a Fn key combination that requires both hands or stretching 7 3/4”), so I made a second layer that lets me control the mouse with keys. You can see the code assigning keys to the layout below. The specific key codes come from this documentation. Controlling the mouse this way is very much like using an old school game controller like back in the old days.
// Configuration of the actual keymaps using QMK's built-in codes
// creates a base "numpad-like" layer, then a "mouse control" layer
const uint16_t PROGMEM keymaps[][MATRIX_ROWS][MATRIX_COLS] = {
/* Base */
[_BASE] = LAYOUT(
KC_ESC, TG(_FN1), TG(_FN2), TG(_FN3),
KC_NUM, KC_PSLS, KC_PAST, KC_PMNS,
KC_P7, KC_P8, KC_P9, KC_PPLS,
KC_MUTE, KC_P4, KC_P5, KC_P6, KC_PPLS,
KC_LSFT, KC_P1, KC_P2, KC_P3, KC_PENT,
KC_BSPC, KC_P0, KC_P0, KC_PDOT, KC_PENT,
KC_F5, KC_F6, KC_F7
),
[_FN1] = LAYOUT(
KC_ESC, TG(_FN1), TG(_FN2), TG(_FN3),
KC_MS_ACCEL0, KC_MS_ACCEL1, KC_MS_ACCEL2, KC_MS_WH_UP,
_______, _______, _______, KC_MS_WH_DOWN,
KC_MUTE, KC_LEFT_ALT, KC_TAB, _______, _______,
KC_LSFT, KC_MS_BTN1, KC_MS_UP, KC_MS_BTN2, KC_PGUP,
KC_LCTL, KC_MS_LEFT, KC_MS_DOWN, KC_MS_RIGHT, KC_PGDN,
_______, _______, _______
),
...
At the end, once you’re done editing your files and defined all the functionality, you run the command to compile it “time qmk compile -kb mechwild/murphpad -km custom”
in my case. That’ll generate a simple binary, which you then use qmk’s toolbox app to send it to the keyboard. This particular controller is easy to flash, plug it in, quickly tap the reset button twice, and it’ll enter flash mode which the qmk toolbox will autodetect and let you flash. It’ll auto-reboot once its done and you can get to using the keyboard immediately.
Incidentally, if you find all this firmware flashing annoying, there’s a QMK firmware known as Vial (an open source implementation of Via) that you can compile in that lets you remap your keyboard on the fly. It requires mapping the keys into a config so that the tool knows what the layout is, similar to a straight QMK setup. But most manufacturers have already gone through the trouble for you.
The only downside I’ve experienced is that Via stores your current keymap in the EEPROM, so manually re-flashing the firmware doesn’t actually “take” unless you also reset the EEPROM. Mine had some issue mapping the rotary volume knob so I gave up on Via pretty early on.
If you want to code your own firmware
The process seems pretty straightforward if you know basic hardware programming (I don’t so I can just barely follow along). Reddit’s r/MechanicalKeyboards links to a detailed explanation.
From r/MechanicalKeyboards's wiki on firmware
1- Configure the pins for the matrix
2- Set the output pins (typically columns) and the input pins (typically rows)
3- Loop across the matrix, turning on one output pin and reading each input pin
4- If a pin reads high, it is pressed down
5- Using the row/column position of the key, decide what to do (send keypress, change layers, whatever)
6- If a key was pressed, add it to the buffer
7- When the whole matrix has been read, send the buffer over USB
For those those who don’t want to assemble stuff
I took on this project because I wanted to mess with some hardware, but I’m sure many people don’t want to go through the trouble. Luckily, there are a number of cheap macropads/numpads of various button sizes that are programmable via QMK available for purchase. They often come with nice features like an actual case (sometimes just stacked acrylic, but other times machined aluminum). The price is often cheaper than going with a similar sized kit, at the cost of the lack of customizability. Economies of scale are pretty scary.
It’s freeing to just do whatever
I’ve been considering getting one of these little mini keyboards for a couple of years now. They’re a pretty expensive (~$80 for the parts used, $50-100 for commercial offerings with various key counts and features), and I just wasn’t sure if I had a valid use case up until now. I already use a tiny Nektar USB MIDI keyboard to fire off key combinations for highly repetitive tasks, so having a much more expensive thing to do the same job felt silly. It was only after being frustrated at the wireless keyboard on the other computer that tipped me over.
I also overthought, to a ridiculous extent, how many keys I needed. Would 9 suffice? How about 16? I’d have to imagine ahead of time my potential use cases to figure out what keys I’d need, and how many. Things got very difficult because I’m very poor at predicting my future decisions. I wound up with 26 plus a knob that can also act as a button. It’s way overkill for what I actually needed, as evidenced by how many keys in the layouts are blank. Maybe I’ll use them in the future, or not.
In the end, it was just easier to stop worrying and grab the most promising-looking kit that had more keys than sense and forge ahead. Having lots of extra buttons means I can lay everything out easily and use the thing from memory — this corner is the mouse control, that corner is for scrolling, that layer is numpad, and this layer is for the RGB LED controls.
It was really nice to just experiment and out layouts until I found something that worked. It’s not maximally efficient, but it doesn’t have to be. I think I’ve been trying to over-engineer things at work a bit too much lately, which comes from working on building work processes, and that seems to have bled into other aspects of life.
What’s next (including data writing)
Well, this post today was about making tools and just… having fun getting utility in the moment without worrying about pesky issues of technical debt, reuse, architecture, and maintenance. But it is sparking some thoughts about tool-making for data. This post is already WAY too long, so I’ll pen it for next week. The idea in my head is that while data science obsesses over tools, especially vendor offerings nowadays, there appears to be markedly less random hacking going on compare to 10, 15 years ago. It’s a definite sign of maturity in the space, but also a bit sad.
Meanwhile, in the hobby space, since I had bought 120 keyboard switches for $38 and only used 26 of them. I’m going to have to find a use for extra low voltage contact switches in future electronics projects. Or maybe just build another keyboard… maybe completely from scratch (including from designing the PCB) this time?!?
If you’re looking to (re)connect with Data Twitter
Please reference these crowdsourced spreadsheets and feel free to contribute to them.
A list of data hangouts - Mostly Slack and Discord servers where data folk hang out
A crowdsourced list of Mastodon accounts of Data Twitter folk - it’s a big list of accounts that people have contributed to of data folk who are now on Mastodon that you can import and auto-follow to reboot your timeline
There’s a bunch of data people on Bluesky right now, but it’s a weird invite-only meme-fest so IF that starts becoming a thing in the future I’ll start mentioning it here.
Standing offer: If you created something and would like me to review or share it w/ the data community — my mailbox and Twitter DMs are open.
Guest posts: If you’re interested in writing something a data-related post to either show off work, share an experience, or need help coming up with a topic, please contact me. You don’t need any special credentials or credibility to do so.
About this newsletter
I’m Randy Au, Quantitative UX researcher, former data analyst, and general-purpose data and tech nerd. Counting Stuff is a weekly newsletter about the less-than-sexy aspects of data science, UX research and tech. With some excursions into other fun topics.
All photos/drawings used are taken/created by Randy unless otherwise credited.
randyau.com — Curated archive of evergreen posts.
Approaching Significance Discord —where data folk hang out and can talk a bit about data, and a bit about everything else. Randy moderates the discord.
Support the newsletter:
This newsletter is free and will continue to stay that way every Tuesday, share it with your friends without guilt! But if you like the content and want to send some love, here’s some options:
Share posts with other people
Consider a paid Substack subscription or a small one-time Ko-fi donation
Tweet me with comments and questions
Get merch! If shirts and stickers are more your style — There’s a survivorship bias shirt!
What an adventure. Next, on to rolling your own device drivers?