IRV

Let’s get this out of the way first: instant-runoff voting (IRV, RCV) is not the solution. There are many issues, but I’ll boil it down to two: complexity, and the center squeeze.

Complexity

IRV is significantly more complex than FPTP, and this has several negative effects.

• Voters find it harder to vote, discouraging participation
• Election results/processes are harder to understand, increasing distrust & animosity
  • Multiple rounds increase confusion and delay
• Voting machines must be replaced or fully reprogrammed

If you’re a social choice theorist or other form of nerd, you don’t care about any of this, which is why groups such as Debian vote using possibly the most accurate and complex voting system, the Schulze method.

But here in the real world these concerns matter a lot. Please try explaining how IRV works to your oldest relative. Seriously, I tried and it didn’t go over great. Or imagine explaining it to your least favourite local politician. Being able to understand the voting system of your country really matters!

Changing voting machines feels like a small thing but it’s probably not. At least it’s a cost that would be worth eliminating if possible.

The center squeeze

Worse than being complex, IRV has a glaring technical issue.

The center squeeze is a specific kind of vote splitting. From Wikipedia:

In a center squeeze, a majority-preferred and socially-optimal candidate is eliminated in favor of a more extreme alternative. Extreme or polarizing candidates who focus on appealing to a small base of core supporters can thus “squeeze” broadly-popular candidates trapped between them

Understanding that the usage of “extreme” doesn’t mean extremist and “center” doesn’t mean centrist, it’s easy to recall examples of this. Here in Canada there is often vote splitting between the Liberal Party (center-left) and the NDP (left) that results in the right-wing Conservative Party winning seats. Even if the majority of voters would prefer the Liberals win over the Conservatives, they may lose due to votes lost to the NDP. This discourages voters from voting for the NDP in the first place (strategic voting), resulting in two-party domination.

You are familiar with this for FPTP, but it unfortunately occurs with IRV also. Imagine some results like this:

It is hopefully clear that “Middling” should win, as it is the party that appeals to the most people. But with IRV, Middling is eliminated immediately for having the fewest first-round votes. In the end “Worst” wins. This once agains encourages strategic voting where Best fans put Middling first to prevent Worst from winning, see here for details.

Empirically, this has happened before, for example in Alaska. There are few countries that use IRV nationally: Australia is one and they end up having a two party system. Ireland’s president is elected using IRV and this is pretty much a two party deal as well.

IRV discourages candidates with broad appeal, and tends towards two-party states.

How you vote already

Before talking about the fix for this, let’s look at a type of voting I do much more often than election voting.

Probably you’ve done the same! It’s just the easiest way to pick event dates, the movie to watch, where to go for dinner, etc. You can select everything you want, not just your favourite, and the results are easy to determine.

This method of voting happens to be superior to IRV, and you already know how to use it. It’s called approval voting.

Approval Voting

In approval voting, you select all the options you approve of, instead of choosing one as with plurality. The option/candidate with the most votes wins. It’s that simple. This is better than FPTP, avoids the issues with IRV, and doesn’t introduce novel issues.

Let’s revisit our criticisms of IRV from before and see if they apply:

• Voters find it harder to vote: not really, and voting for only one candidate like before is still allowed
• Election results are harder to understand: nope, votes are just tallied
  • There is only one round
• Voting machines must be replaced/reprogrammed: nope, only minor code differences should be necessary
• Center squeeze: doesn’t happen with approval voting in practice! Votes are not split among candidates since voters can vote for more than one candidate.

From a more scientific perspective, approval voting has been tested in voting simulations and it does quite well (click for sources):

For that last graph, the author Jameson Quinn wrote:

[Approval voting’s] VSE [voter satisfaction efficiency] is around 89-95% for most levels of voter strategy. That’s not the best of the methods I tested, but it certainly is the best “bang for the buck”; a simple reform, with basically no downsides, which improves outcomes hugely.

That just about sums up how I feel about it. There is a massive electoral improvement just sitting around, waiting to be adopted. And once you consider the complexity that theoretically better methods entail, approval voting stands out as a clear winner to me.

Usage

You can take a look at Wikipedia for details, but approval voting in practice is mainly limited to two cities, both in the USA: St. Louis and Fargo. If you’re in the US and want to improve this, the Center for Election Science is a good place to start.

Unfortunately I’m not aware of any Canadian-specific resources. The best I can find is that Ontario MPP Ted Hsu wrote about it one time, and there was a passionate submission to Trudeau’s 2016 Committee on Electoral Reform by a computer science student. Clearly there is work to be done.

If you’d like to create approval voting polls online, one site I’ve found is strawpoll.vote, which supports approval voting under the name “multiple choice”.

You can find more resources regarding voting systems in my digital garden.

Conclusion

IRV is better than FPTP, but why stop there? Approval voting is simply better than both. I will not be endorsing or promoting IRV in the future, and will instead try to raise awareness of approval voting.

Writing

The place that first comes to mind where I write things is my journal, Daylio. In 2023 I wrote 13.4k+ words there, or 1100+ words per month. That’s more than my last blog post. Mostly these are just thoughts at the top of my mind that day, or interpersonal things I’ve been trying to work through. Writing things out can really clear my mind, and helps me feel like I’m not running through life forgetting those abstract social things, like “X really surprised me today when…”

Key features of Daylio are logging your mood and activities. I’ve found this to be helpful personally, to stay more aware of my mood and how I spend my time. Daylio can use this data to draw correlations or generate reports for you, but personally these are not that useful since I don’t make myself write an entry every day. Having that level of data collection is appealing, but only journaling when I feel like it’s important helps it not feel like it’s a chore.

Journaling through my phone instead of paper seems little funny, but in practice it’s very convenient, and quickly typing my thoughts on my phone is already something I’m used to from messaging friends.

I also use Obsidian. I previously had a large collection of “cool stuff”-type bookmarks that I never looked at, and now most of these have been transferred into a much more accessible “personal wiki”, most of which is public.

I have been contributing content to these pages (beyond the original bookmark links) for the past several months, mostly adding bits here and there, but occasionally doing deep dives like on Local Currencies or Software Licensing.

Finally, I set up a self-hosted instance of Monica in 2023, mostly just to get my contacts in order. I will write things in here sometimes, like when I’m getting a lot of information about someone and would like to remember/review all of it, such as when dating.

Goal setting

Moving on to smaller word counts, there’s goal setting. Discrete tasks that I’m trying to track or remind myself of all go in the Loop Habit Tracker app. It has no annoying gamification, but the simple act of tracking my progress (or lack thereof) has definitely helped me stay aware of whatever I’m trying to do. The phrase that’s come to mind is “tracking shapes behaviour”.

Longer term and more abstract goals go in a private Obsidian note. For 2023 I wrote out a laundry list of goals across many categories, but nearing the end of the year I realized annual goals is too long a timeline, at least for my life currently. Starting in winter 2023-24, I’ve set goals for the season (meteorological). For example in winter (Dec 1) I set five goals, and on the first of each month (Jan, Feb, Mar) I reflect how well progress is going with these goals. On March 1st I will also set the next few goals for the spring.

It hasn’t even been one full season yet, so I might shift this system, but so far I like it a lot. Reflecting monthly is helpful, but more importantly having seasonal goals feels way more useful than any other time frame. My life and goals are heavily affected by the seasons, and it’s useful to be able to shift my focus like that.

Synergy⁉️

The systems described thus far can combine in helpful ways. For example I like to look over my Daylio entries and moods for the month when doing my goal reflection, to remind myself of how the last month went. If one of my goals is “do activity X at least Y times per month”, I can look at my habit tracking app to see whether I completed it, then write some of my own feelings on the topic.

Productivity

At the absolute minimum of writing there’s productivity tools. I use Todoist for day-to-day tasks and a digital calendar to keep track of events. These feel like fundamental, bedrock parts of daily life that would be hard to live without, to the point that I rarely think about the fact that I use them.

Conclusion

If there’s one thing I can take away from all these systems, it’s that it’s truly very helpful to write things down. Grappling with your thoughts can feel like trying to do mental math, and the solution is the same in both cases: write it out!

It feels obvious now, but this issue wasn’t to me at the time. The problem is you can never change any defaults. Here’s a real issue from Amfora: Alice installs v1 of my code, that uses search engine A. The search engine is of course configurable, and so by default search_engine = A gets put in her config file. A little while later, search engine A goes down forever, and so the wise developer (me) changes the default search engine to fancy new search engine B. But when Alice installs v2 of the application, it reads search_engine = A in her config file and will never use the new default! Alice would have to read the release notes and manually make the change herself, which is a pretty big ask.

This gets worse for interrelated config keys, like for theming. Amfora has multiple config keys to set the colour of various UI elements, and occasionally new ones are added. If all these keys were set in the config file by default, then the default theme could never be changed. For example, imagine a v3 that adds a new theme key like button5_colour, and also changes the default theme. For users with the application preinstalled, they would see the old theme like before from their config file, except the new “button 5” would have a colour from the new theme, looking weird or even invisible. In Amfora’s case this was thankfully avoided as theme keys were not kept in the default config.

The solution

Many applications on my laptop and server appear to solve this problem in a half-baked manner, where all the configuration keys are indeed written to the config file, but as comments. Technically this avoids the problem described above, but I’m not a fan since this can mislead the user. When the application is updated and the defaults change, these comments will then be out of date, possibly leading to some frustating debugging sessions. (“I swear IPv4 was already disabled!”)

In my opinion the solution to this is simple: the user has to write the config options themselves. At most the application could create the file with just a few comments, instructions on how to find documentation (manual, web page). This way the only options in the config file are the ones that the user manually written in. Only those will remain in place when application defaults are changed.

This is simple for one direction: proving some data existed after a certain point in time. That can be easily achieved by including unguessable current information, such as news headlines and stock prices. A classic pop culture example is kidnappers proving a hostage is still alive by having them pose with the day’s newspaper.

But trusted timestamping is concerned with the opposite direction: proving data existed before a certain time. This is not so easily done, but it is possible, and the use cases are quite important in my opinion.

Purpose

I see three main reasons for doing trusted timestamping.

1. Verifying old cryptographic signatures
2. Data integrity
3. Proof of technology ✨

In addition to these there is the obvious one: authenticity. Trusted timestamping can prove that a document that claims to have been made on a certain date actually was.

Verifying signatures

This is the mainstream industry use case, as far as I can tell. For example Apple requires timestamping as part of its code-signing process. The reasoning is simple: sometimes private keys get leaked or revoked. After that happens, how can we validate any previous signatures? They might have been made by a third party using the leaked key. But if the signature was timestamped, we can know that the signature was created before the key was made invalid. This is a real problem that trusted timestamping solves.

Data integrity

A basic method of ensuring data integrity is storing hashes. But what if those hashes are tampered with? Using signatures instead allows us to verify that no one but a person/org we trust has modified the data. But what if we don’t want to trust that person, or their ability to guard their keys? Using signatures adds an extra moving part and security risk that in some cases may not even be needed. If we can agree that the data was valid and unmodified at a certain point in time, trusted timestamping will provide data integrity forever. An example use case is backups, or storing org-internal archival data like legal agreements.

Proof of technology

This is my personal, special, modern use case. As AI-generated content gets better and better, being able to prove that a piece of media was generated before certain AI software even existed will become extremely important. Trusted timestamping has the ability to end all these new uncomfortable arguments that such-and-such picture was created whole-cloth by some incrutable algorithm, and instead bring us back into old school discussions about whether it looks “PhotoShopped”.

Of course, it can be argued we are too far gone at this point, AI image generation is too good. But AI video remains unsolved for now, and the images are not truly perfect yet. As the saying goes, the best time to plant a tree timestamp data was 20 years ago, the second best time is now.

Software

There currently exist two viable trusted timestamping methods I am aware of. There is OpenTimestamps, and there is the Time-Stamp Protocol, or RFC 3161. The former timestamps using Bitcoin (but efficiently, not one timestamp per block) and the latter uses the signature of a trusted service (Time Stamp Authority).

Here is a quick comparison:

You can learn more about OpenTimestamps on the website. Beyond the actual RFC linked above, you can learn more about RFC 3161 by playing around with OpenSSL, see man openssl-ts.

I’m not going to dive any deeper into these tools than the above table, but my general recommendation is to use RFC 3161 with a good third-party unless not needing to trust anyone is actually a requirement for your project. And you could always do both!

Usage

OpenTimestamps has a CLI tool and libraries in several programming languages, see here. It also has a GUI on the website.

RFC 3161 is supported by OpenSSL as a CLI tool with the openssl ts command. To make RFC 3161 timestamping on the CLI easier (as it’s not just one OpenSSL command), I’ve created a Bash script each for timestamping and verifying. You can get them here.

The only GUI option for RFC 3161 I’m aware of is on the freeTSA.org website, although personally I have no reason to trust freeTSA. Better than nothing though, for anyone who doesn’t use the commandline.

For trusted third-parties there are various options, including reputable companies such as DigiCert.

Conclusion

It’s time for trusted timestamping! The cost of this is basically zero (storing a few kilobytes of proof at most), but the advantage can be very large. It’s one of those things that you’ll wish you had done at the start.

Many groups can act on this (in increasing order of importance):

• General public: timestamp your important documents and media to prove their authenticity. You can do this today on your phone with ProofMode, and on the desktop using CLI or GUI tools as discussed above.
• Archivists: timestamp everything you archive, for integrity and authenticity. For redundancy consider using both methods mentioned above and many RFC 3161 timestamp authorities.
• Developers: if you work on software where data or signature integrity is at all important, add trusted timestamping to your application. And if you use Git signing, you can easily start timestamping all those signatures, see here.

If you create software or projects related to trusted timestamping, feel free to let me know about them!

In the early days of Gemini, I had a lot of fun reading posts as they emerged from various gemlogs, responses to those posts, and writing my own posts. It felt new and fresh and fun, an exciting secret place full of potential. Eventually, as the quarantine stage of the pandemic winded down and my life shifted, things fell into a more natural pace for me, casually checking in with gemlog updates and reading people’s posts every so often. Over time, the novelty of the protocol wore off, which is perfectly normal. When I wasn’t actually developing code, what I was left with was simply the experience of Gemini (and my browser) as it actually was, without added feelings around meta topics like protocol specifications.

As much as I want to like and use Gemini, in practice I simply don’t. Over the months (and now years!), it just doesn’t keep me coming back for more. I first mentioned this feeling all the way back in May 2021, but I’ve wrestled with writing this blog post because I couldn’t pin down the reasons. Why don’t I use it? Am I too warped by modern convenience to be able to enjoy a slow Internet?

The reason

After a lot of thinking, I’ve realized there is one main reason I don’t keep coming back to Gemini: it offers no advantage over how I already use the Web.

In practice, the Web already has all the Gemini content I’m interested in from various people, and then of course everything else. Having everything in one place (whether my web browser or feed reader) makes for a much nicer experience.

Gemini is a reaction to bloated modern websites, but in fact I don’t actually visit that many gross websites like that. When I do, my ad blocker and paywall bypasser usually make them decent again. Otherwise, I spend the majority of my non-work Internet time on lightweight sites like my feed reader and Hacker News, and some time on sites that Gemini can’t emulate: YouTube, Reddit, Discord. The reality is that Gemini just wouldn’t actually improve this experience for me.

Gemini also has downsides. I enjoy being able to express a bit of personality on my website through CSS, which is just not possible with Gemini. And I’m come to realize I really do appreciate the unique design of different people’s websites as well. If someone’s design is unbearable, using my feed reader or Firefox’s reader view gives me the choice not to experience it, but with Gemini I don’t have the choice at all.

Finally, a more personal issue is that reading long form text in monospace really sucks, and of course that’s all Amfora can do, operating in the terminal. This makes developing and testing my browser easy, but makes actually using it for reading gemlog posts not ideal — for me anyway. Long reading on my computer is already annoying for me, and this doesn’t help.

The future of my Gemini software

So, what does this mean for my software repos that deal with Gemini? Mostly I will be making official what has been obvious for some time: active development is not happening. Here is the rundown:

• Amfora: maintenance mode. When possible, I’ll make/merge bug fixes, and maybe slowly merge feature PRs by others.
• md2gemini: archived. It’s always been a pile of hacks, and now I’m not that invested in fixing it.
• go-gemini: maintenance mode. Bug and spec fixes only. The code should be pretty simple to continue maintaining in case anyone is using it.
• gemget: maintenance mode. Being able to simply download Gemini content will still occasionally be useful to me.
• gemlikes: archived. It’s always been a toy, and now will no longer see any further development.
• go-gemini-socks5: maintenance mode. I somehow don’t expect this single file will need many changes.

The future of my capsule

gemini://makeworld.space will remain online for the foreseeable future, but I will no longer be syndicating my blog posts or site updates there. I will leave existing gemlog posts at least, but other pages that would require updating may be removed.

Bye, Gemini

Goodbye Gemini! I hope you continue to exist for those who want it, and to remind people that Web alternatives are possible.

I am reviewing the spec from commit 1a37b2f, which is from June 10, 2022.

Design

Various protocol design comments below.

YAML

Please don’t use YAML for the peer list. YAML is not great. A simple JSON list would be better. A separate Markdown file can provide human-readable metadata if needed.

Key rotation

The key rotation scheme is definitely interesting. I like the idea of limiting the damage a leaked key can do, similar to TLS cert renewal. However I think the user experience of having to generate a new key every two years and tell all your friends about it is quite bad, and will make people wonder why they have to do this. I get the idea of culling old followers, but the reality is that this is a one-way protocol will no follow notifications, so the effect of this culling won’t be felt.

One solution is to have the majority of people not actually manage their own keys, but that doesn’t feel great either. I think it would probably be better to just let keys stick around with no expiry. This seems to have worked out okay for Scuttlebutt.

Server-to-server communication

First off, it’s not explicitly mentioned that one server sending a board to another triggers that server to send the board to others, etc. It seems obvious this is the case in retrospect, but I think it would be good to mention to make the gossip nature of this part of the protocol clear.

More importantly, I think there is a large issue with this push-only server communication. Imagine the following scenario:

1. Bob publishes a new version of his board.
2. Server A receives and stores this version.
3. Alice can view Bob’s latest post using Server A, and all is well.
4. Server A goes offline, perhaps for a day or two.
5. Bob publishes a new version of his board. Server A never receives this update.
6. Server A comes back online.
7. Alice uses Server A, and thinks she is getting Bob’s latest post, but she is not.

Obviously it’s impossible to always have perfect consistency, and in the scenario described things would eventually work out, but I still think this is a problem. It is easily solved by speccing a server-to-server pull mechanism as well.

This could be something like:

On server start up and at every interval X, do a GET request for each key this server is aware of to two random servers in the realm.

X should be something long like several days.

This would require all servers to allow GET /<key> requests for anyone, which goes against a later part of the spec:

Servers are not obligated to provide this endpoint for all requesters; they might provide it only to users of a particular client, or only to users who have paid for access, or according to any other scheme.

I get the idea behind this, and so I’m not sure how to reconcile it with what I’ve said above. I lean towards requiring all servers to be open to everyone. This has the advantage of making all clients server-agnostic as well, if they want to be.

Finally, I’m no gossip protocol expert, but it seems like it would be easy for servers to be missed when only 5 are getting selected at random. It would be nice for someone who knows the math for these things to be able to explain the probability of servers being left behind.

Difficulty factor

The difficulty factor stuff is definitely cool. But I’m not sure if increasing difficulty only once most boards are used up makes sense. Without a single point-of-truth like a blockchain, this can introduce data races, as others have pointed out. Additionally, due to TTLs board will be disappearing all the time, so there would need to be literally millions of active users for the difficulty to become meaningful.

What if this difficulty factor was baked in from the start? Preventing spam from the inception of the protocol/realm seems better than only near the end. For example the spec could define a difficulty based on a numerical threshold, or keys needing to have X preceding zeros, etc. Whatever works in a way that doesn’t weaken Ed25519 (!!). That number would be picked to have a meaningful difficulty for modern computers, taking maybe 10 minutes or more.

Client -> Server interaction

I understand this is a draft, but I just thought I’d mention that the spec doesn’t explain how clients are supposed to talk to servers, beyond the APIs available.

For example, when a client user updates their board, what servers does that client send the new board to? Just its favourite one? A random selection from the realm list? Etc.

One section of the spec alludes to clients maintaining lists of trusted servers, so I expect details on all this are coming. Looking forward to hearing them!

If the signature is not valid, the client must drop the response and remove the server from its list of trustworthy peers.

“The agony and ecstasy of public key cryptography”

I love this section. Cryptography is beautiful magic, and I’m glad to see it used and appreciated here.

Nitpicks

Here are some small comments or questions I have about the spec. They don’t change my feelings overall.

• Why is the limit 2217 bytes? I’m unable to find any meaning behind this number online. It seems to me that this should probably be larger, or at least well explained and justified.
• Limiting to HTTP/1.1 seems unecessary, allowing HTTP/2 to be used transparently by servers that support it would do no harm
• How do clients enforce the board aspect ratio? I don’t know enough about HTML/CSS to understand how boards wouldn’t just be able to override this with !important or something.
• The client is supposed to “open links in new windows or tabs” — how does that work when the board HTML might specifically say to open in the same tab?
• It would be nice to have the correct Content Security Policy in the spec, as they look tricky to get right.
• The protocol mentions TLS but doesn’t specifically disallow non-TLS servers. Would be good to see that spelled out.
• Putting the exact CORS requirements needed in the spec would be helpful, as CORS is tricky to get right.
• It’s not mentioned, but it seems like servers should reject PUT requests if the last-modified meta tag is older or equal to the timestamp of the server’s version of the board, same is If-Unmodified-Since
• The Prefer header in the spec looks like a big fancy official header like Vary or Accept and I was surprised to look it up and find out it’s not. I could see it becoming one in the future though, so it might be better to use something like Spring-Prefer.
• 2217 bytes * 10 million is indeed 22.17 GB, but it is also 20.65 GiB, and the latter is more meaningful for computers.

Odds and ends

Here are few thoughts that popped into my head while reading, I couldn’t fit them into other sections.

• Nothing mentions how often servers should update their realm list
• DNS could be used to get around having to share public keys with people
  • Set a TXT record on _spring83.mydomain.com with your public key
  • Then clients will resolve that, so you can just tell your friends to follow you by entering mydomain.com

Conclusion

I’m quite excited about this protocol, and will be following its development closely. As usual, I’d like to dive in over my head and start developing a client and server, but I will hold off this time, until the spec is a bit more finalized, and likely until Robin releases his software as well. I’ve got my hands full working on a secret DNS project for now anyway 😉. But I will definitely be participating when the time comes. Having a little space of HTML to share, with strict limits — it sounds like an awesome way to experiment and get creative. I look forward to it!

When I first saw the Badger 2040 on Hacker News, I knew I had to get it. It seemed like a great little package to hack on, with an e-ink screen, buttons, and an LED, all for a good price. And finally I’d get my hands on an RP2040, which has interested me since it first came out.

Development

The development experience on the Badger 2040 is really fantastic, and great for beginners. As someone who’s only experienced embedded development from the side of firmware loading and C++, this was a welcome change. Here’s how I add a feature to some code on the Badger 2040:

1. Plug it in
2. Open Thonny
3. Open the main.py file (or any other file on the device) from within Thonny
4. Change the code
5. Press the Run button in Thonny and watch the Badger reset instantly and load my new code
6. Unplug the Badger once I’m satisfied with my changes

It’s super simple and fast, and makes for really efficient development cycles. Combine this with MicroPython, and overall writing code for the Badger is really fun and fast, which is exactly what I want for a hobby project. 😁

See the getting started guide for more about coding on the Badger 2040.

Wordle

Although the Badger is designed to be used as a badge, I’m not going to a conference any time soon. So I decided to make a project that subverted things a little bit. The obvious choice was Wordle, which seems to be the project du jour. As someone who plays Wordle on the computer, I was excited to create a portable Wordle-playing device. Sure phones exist, but playing on e-ink is way cooler!

Here’s a demo video. Note I’m using my Badger on its side, the marketed orientation is landscape.

This input method obviously has some limitations, but it’s not as tedious as I thought it would be. It takes longer to enter, but in my experience not so much longer that it’s annoying to use.

You can play as many games as you like. With no RTC or Wi-Fi support, there was no way to keep track of the what day it was anyway, so I figured I might as well turn it into a feature.

I was impressed with the update speed. There are a variety of options, and the fastest one (UPDATE_TURBO) is really quite fast, and works well for scrolling through letters. There is definitely some ghosting (although it looks worse in the video than in person), but that goes away after a slightly longer update when moving to the next cell.

Words

For word data, I extracted the two wordlists from the original Wordle javascript: the list of winning words and the list of all five-letter English words. With some grep, sed, and Python, I converted these into two text files, where words were stored concatenated. Since they’re all 5 bytes long, there’s no need for any separator.

Strangely, the winning words were missing from the total wordlist, so I added them back in and resorted the list, before saving to a text file. In total the two files were 76,405 bytes (~75 KiB), but the Badger has lots of flash space so I didn’t need to apply any compression.

To load the files on to the Badger, all I had to do was upload them to the filesystem through Thonny. Having a native filesystem makes loading data like this a breeze. And I can read data from the files just like I would with regular Python, using open() and .read(). This is a big difference from what I’m used to with embedded development, where I’d have to put the data in a variable in some header file somewhere, or use some sort of preprocessor that would add in the data.

I also ended up having to implement a binary search, which I used to check if the word submitted for that row was actually a valid word. Another fun part of embedded developement: getting back to basics!

Improvements

There are a few improvements I’d like to make to the project. One abstract improvement would be to improve the UX/UI — making it easier to tell which cell you’re currently working on, for example, and/or highlighting the current row.

I’d also like to switch to using partial updates instead of fullscreen updates where possible. I don’t think these are any faster, but it would make for a less jarring effect than the whole screen flashing in front of you. Unfortunately for some reason the Y and height of the update region both have to be a multiple of 8, so it’s not as simple as just updating the area that changed.

Badger library

Python documentation for the Badger 2040 is available here.

The dev seems to be on top of things and responded very quickly to my issues, so that was great. Overall the API is good, I was impressed to see things like decimal text scaling and arbitrary text rotation. But it would be nice to have the ability to provide your own custom font, like Adafruit display libs allow. The fonts provided are nice, but none of them are monospace, for example. And they all seem to be converted from vector fonts. Using a bitmap font, where pixels are hand-chosen for a specific size, would provide much cleaner glyphs, at the expense of decimal scaling of course.

Conclusion

I had a lot of fun working on this simple self-contained project, and was happy to fully complete a personal project. Thanks to Pimoroni for creating this product! I hope to see people doing cool things with the Badger 2040 in the future as more people get their hands on them, I think they have a lot of potential.

Code is available on GitHub here.

Thanks for reading!

Eventually, I got it working, so here is that guide. Hope this helps! This guide might also work on macOS, but I have not tested it.

Thanks to Noah Broyles for the script that helped me figure this all out.

First off, I was only able to use Python 3.8. I tried with 3.9, but I was not able to install it under WINE. If you are able to, feel free to use that, and change references to 38 in this post to 39. I was also using PyInstaller 4.5.1, but other versions should work fine.

The first step is to install WINE for your Linux system. WINE is a compatibility layer that allows the running of Windows software on non-Windows systems. You can install WINE easily enough from your distro’s package manager. I am using wine-6.16, but most likely your distro’s version will work fine.

Next, download the Python 3.8 Windows installer. You can download that directly from the official Python Software Foundation here. With it downloaded, you can run wine python-3.8.9.exe to begin the installation. WINE may prompt you to install something like .NET or Mono, you should click Yes and let it install.

You should not use the default installation, follow these steps:

1. Check “Add Python 3.8 to PATH”
2. Click “Customize installation
3. Click “Next”
4. Click “Install for all users”
5. Set the install location as C:\\Python38
6. Click “Install”
7. Close the window.

Using the C:\\Python38 path helps keep things standard for this blog post.

Now Python 3.8 should be successfully installed in WINE. You can check this by running wine C:/Python38/python.exe --version, which will run Windows Python through WINE. Amazing!

Next we’ll need to upgrade pip, to make sure things will install fine. Run wine C:/Python38/python.exe -m pip install --upgrade pip. WINE might spit out a lot of warnings about libgnutls or something or other, but as long as Pip doesn’t report any errors it’s fine.

Now if your project has any dependencies, you should export them into the standard Python requirements.txt file. You can then install them into your WINE Python with this command:

wine C:/Python38/python.exe -m pip install -r requirements.txt

If you’re using a superior dependency manager like Poetry, you should be able to install and use that in WINE, although I haven’t tested that.

In any case, you’ll need PyInstaller to be installed, so run this command if PyInstaller wasn’t in requirements.txt already:

wine C:/Python38/python.exe -m pip install pyinstaller

Now you can run PyInstaller and an EXE will be built! Just run

wine C:/Python38/Scripts/pyinstaller.exe ...

and replace ... with whatever arguments you used to build a Linux executable.

Enjoy!

That’s why I’ve launched my own NFT marketplace, one that’s better than all the rest!! No fuss, no waiting, no crypto, no trouble at all. Check it out!

Enough Tea 🍵

This post contains some images that need to be viewed at 100% size to be seen correctly. If you normally browse at a higher zoom level than 100%, please zoom out when you get to any of the images.

Unfortunately some of these images are out of date. See here for up to date images, with slightly improved results.

Just yesterday, I released my dithering library, written in Go. It’s the product of many hours of research, experimentation, and refactoring. I’m excited that it’s finally out, and to see what people create with it. Personally I’d like to create a CLI dithering tool at some point that uses it.

Creating this library required research into the different algorithms for dithering, and how to implement them. Unfortunately, not all of that knowledge is easily found. It’s spread across blog posts, Wikipedia pages without citations, old books, and code comments. Recently, Surma wrote an article called Ditherpunk — The article I wish I had about monochrome image dithering. It is an invaluable resource that combines lots of knowledge about dithering into one place. The main thing missing from the post, however, is going beyond “monochrome”. Surma shows us how to dither with a 1-bit palette of black and white, but what about more shades of gray? What about colour images? With this blog post I’d like to explore those techniques, taking them out of my code and into English, so you can easily apply them elsewhere.

First off, please read Surma’s post. It explains what dithering is, how it works, and many different algorithms. I don’t feel the need to explain these again, but merely add on what I’ve learned.

Before we start

An HN commenter informed me that you cannot accurately represent linear RGB with just 8-bits (0-255), you need at least 12. Because of this, I have updated the blog post to use 16-bit color (0-65535), and will be updating the library shortly. Make sure you do this in your code too!

Overview

Dithering operations consist of at least two steps, applied to each pixel. Sometimes there are further steps, like “modify these nearby pixels”, but these are the basic ones.

1. Modify the pixel’s colour value.
2. Find the colour in the palette that is “closest” to that modified value and use that on the output image.

Now, we know from Surma’s post that step 1 must be done with linear RGB values. Otherwise, different values will be affected differently — for example adding 5 to each colour won’t affect all colours the same way.

But what about step 2? How do we find the closest colour in the palette? In 1-bit dithering we don’t have to worry about this, because anything above 0.5 is white, and anything below is black. But when your palette colours can be anyhere in a 3D space, it is something that needs to be figured out.

Perhaps surprisingly, we’re not looking for a way that matches human perception. In fact, we are using Euclidean distance with linear RGB values, which doesn’t match human perception at all! Why is this? Thomas Mansencal (@KelSolaar) explains it best:

You can factor out the observer [the human] because what you are interested in here is basically energy conservation. The idea being that for a given pool of radiant power emitters, if you remove a certain number of them, by how much must the radiant power of the remaining ones be increased to be the same as that of the full pool. It is really a ratio and doing those operations in a linear space is totally appropriate!

This helped it click for me. Dithering can be thought of as trying to reconstruct the “radiant power” of the original pixel colours, while restricted to a certain set of “emitters”, aka the palette colours. It is only with linearized RGB values that we can properly measure the radiant power.

Update December 2022: After agonizing over several issues with my dithering code, I eventually discovered an improvement over just Euclidean distance.

Weight each channel distance according to how humans percieve its luminance. This is better than just “radiant power” from a physics perspective, because human luminance does not treat red, green, and blue equally. Green is weighted the highest for example, because humans are the most sensitive to green.

The weight values can come from Rec.709 / sRGB, so now distance is calculated using the weightings 0.2126 R + 0.7152 G + 0.0722 B, where R, G, and B are the elements in the Euclidean distance calculation. (Squared difference in each channel between the colour in the palette and the modified pixel) All values are still linearized!

The exact code changes I made for this improvement can be seen here.

Random Noise (grayscale)

Random noise is a good one to start with, to show the differences between 1-bit dithering and larger palettes.

Surma does this by basically just adding a random number from -0.5 to 0.5, and then thresholding it.

grayscaleImage.mapSelf(brightness =>
  brightness + (Math.random() - 0.5) > 0.5 
    ? 1.0 
    : 0.0
);

In my library, there are two separate random noise functions. One is for grayscale, and one is for RGB. The grayscale one looks like this:

roundClamp(float32(gray) + 65535.0*(rand.Float32()*(max-min)+min))

The math with min and max is just to put the random value in the desired range. If we clean that up it’s more understandable:

roundClamp(float32(gray) + 65535.0*rand())

rand() just represents a function that does whatever range we want, -0.5 to 0.5 in this case.

So, obviously this is quite similar to what Surma does. The heavy lifing of the dithering in this case is done by the other code, the part that finds the best palette colour.

The main thing that’s worth noting here is how the rounding works. roundClamp rounds the value to an integer, and then clamps it to the range 0-65535. But how is the rounding done?

Another HN commenter shared this link which discusses different rounding methods, and the problems with the way rounding is often done. The best solution is to use a rounding function that does this:

Given a number exactly halfway between two values, round to the even value (zero is considered even here).

round( 1.7 ) –> 2 round( 2.7 ) –> 3
round( 1.5 ) –> 2 round( 2.5 ) –> 2
round( 1.3 ) –> 1 round( 2.3 ) –> 2

This is not really about dithering, but this is a pretty important point to get things right mathematically. Make sure you do it! Otherwise your outputs will be biased.

Random Noise (RGB)

Back to random noise, but for colour this time.

r = roundClamp(float32(r) + 65535.0*(rand.Float32()*(maxR-minR)+minR))
g = roundClamp(float32(g) + 65535.0*(rand.Float32()*(maxG-minG)+minG))
b = roundClamp(float32(b) + 65535.0*(rand.Float32()*(maxB-minB)+minB))

And simplified:

r = roundClamp(float32(r) + 65535.0*randR())
g = roundClamp(float32(g) + 65535.0*randG())
b = roundClamp(float32(b) + 65535.0*randB())

Pretty simple, it just adds randomness in each channel. This can be done with grayscale images too, but it won’t work very well, because grayscale colours only actually have one dimension. So it will just not add as much randomness as you would expect.

Also note that usually you’ll want the random ranges to be the same for R, G, and B.

Here’s an example:

Ordered Dithering

Modifying threshold matrices

Most of the resources online that talk about ordered dithering talk about a “threshold matrix”. “Thresholding” is how these matrices are applied for 1-bit dithering. You divide the matrix by whatever number is specified, scale it to the colour value range, and compare it to each pixel in the image. If it’s less than the matrix value, make it black, otherwise white. Obviously this doesn’t work with any other kind of palette. So what’s the solution?

Wikipedia offers an answer. Unfortunately there’s no citation, but I’ve confirmed independently that it works. Here it is with some of my own math added as well.

$cell$ is a single cell of the matrix.

$strength$ is a percentage, usually a float from $0$ to $1.0$. It’s the amount the matrix will be applied to the image. The closer to zero it is, the smaller the range of input colors that will be dithered. Colors outside that range will be quantized. Usually you’ll want a strength of $1.0$, to apply the matrix and dither fully, but sometimes reducing it can be useful, to reduce noise in the output image. It is inversely proportional to contrast – that is, when you reduce the $strength$, it is visually similar to increasing the contrast.

$strength$ can also be negative, from $0$ to $-1.0$. This is useful in certain cases where the matrix usually makes things bright, like what Surma describes with Bayer matrices.

Note that $65535$ is multiplied by $strength$ because the colours in my code are in the range $[0, 65535]$. If yours are different you can change that number.

$max$ is the value the entire matrix is divided by. It represents the maximum value of the matrix, and normalizes it by dividing. Usually this is the product of the dimensions of the matrix. It can also be the largest value in the matrix plus one.

The result of applying this operation to each cell of the matrix is a new, precalculated matrix, which can be added to a pixel’s colour value for dithering. Adding 0.5 does not need to happen in this case. In my library, I call the function that does this convThresholdToAddition, because that’s essentially the purpose of this — converting a threshold matrix into one that can be used for addition.

Note: This is designed for matrices that range from $0$ to $max - 1$. If you’re using a matrix you found that starts at $1$, you’ll usually just want to subtract one from each value when you program it, then apply the operation I described above.

Using a modified matrix

Now that you’ve modified the matrix so it can be used for addition, it needs to be applied to the image. This is pretty simple. Use modulus so the matrix values are tiled across the image, and add the same value in the R, G, and B channels. If you’re using a grayscale image you can just apply it in the one channel, or still use RGB. Since the same value is being added it doesn’t make much of a difference. Like always, make sure to clamp the values to the proper colour range.

Doing all of this definitely works with colour images, but it’s not the greatest. Here’s an example, where the palette is red, green, and black.

Here it is where the palette is red, green, black, and yellow.

As you can see, it doesn’t really emulate any of the yellow in the first example, while Floyd-Steinberg can. Once yellow is added to the palette it looks pretty good though.

Error diffusion dithering

Error diffusion dithering is done pretty much the same way as Surma described. The main difference is that the quantization error is measured for R, G, and B, instead of a single gray value. The final bit of (pseudo) code looks like this:

roundClamp(r + er * matrix[y][x])
roundClamp(g + eg * matrix[y][x])
roundClamp(b + eb * matrix[y][x])

The variables starting with e represent the current quantization error.

Serpentine

This is unrelated to color dithering, but it’s something my library does that Surma didn’t mention, and something I couldn’t really find information on online — serpentine dithering.

This is when the horizontal direction of error diffusion dithering (usually left-to-right) reverses for every other line, going right-to-left. This is simple to implement, and improves results by removing line artifacts.

The one thing that was hard to find any info on was how you need to reflect the error diffusion matrix horizontally every time you go backwards as well. Otherwise you modify pixels to your right that have already been finalized.

Here’s an example of the difference serpentine dithering makes, using the simple 2D error diffusion matrix, as it has the worst artifacts.

In my experience, it makes sense to always do serpentine dithering for the best results.

Strength

In the ordered dithering section I talked about how a $strength$ variable could be used to control how much dithering was applied. This can also be done for error diffusion, and it appears some graphical suites like ImageMagick even have this built-in, as is described here.

Here is the modified pseudocode for making use of a strength value:

roundClamp(r + er * strength * matrix[y][x])
roundClamp(g + eg * strength * matrix[y][x])
roundClamp(b + eb * strength * matrix[y][x])

And here’s an example of different strength values with Floyd-Steinberg dithering, with a palette of red, green, yellow, and black.

As you can see, the strength variable is pretty much inversely proportional with contrast. A value like $0.8$ might be good to reduce some noise in the image, but go too low and the colours will just be inaccurate. Sticking to $1.0$ is a reasonable default.

However this does not apply to Bayer dithering of color images. As Surma mentions, Bayer matrices are fundamentally biased to making images brighter. For color images, the bias occurs in all three RGB channels, and can distort image colors overall, making the final dithered image look inaccurate. Several sites recommend a value like $0.64$ (256/4), and in my experience it’s a good default for color image dithering, with Bayer as well as ordered dithering in general.

Other things

This post is pretty long now, but I do have a few other things to share.

One is about clustered-dot dithering. It looks like this:

Currently my library just does clustered-dot dithering with some preprogrammed matrices I’ve found around the Internet and in a couple books. I was unable to find any proper instructions on how to generate a clustered-dot dither matrix, so that they can be created on the fly, and in any size. The books I read hinted that it was possible, but I couldn’t find anything. If you know how to do this, please contact me! I will update my library and this post if I figure anything out.

The other thing is that I used Joel Yuliluoma’s per-cell algorithm for creating Bayer matrices, as written here (the second code block). Since I had to rewrite it in Go, it’s cleaner to read and understand than the original C. If you’re interested in implementing this yourself, you might prefer reading it. My implementation is here. It’s been tested to make sure it works the exact same as Yuliluoma’s.

Thanks

Thanks for reading! I hope this is helpful to anyone else who is interested in dithering. And if you know Go, check out my library! Make something cool with it. 😄

Looks like this is getting some traction on Hacker News! Feel free to comment there.