At the end of Late May Journal: building things I said I didn’t need, I said:

I’m not quite sure what I’ll do next. I am starting to hanker a little after doing some more work on toykv, finishing up the range-scan functionality I never got working well. Somehow, the browser tab holding “Programming Rust” is still open after not having written any Rust for over a year now. Perhaps it’s time to blow the dust off and get back to it.

Like talking about tools and MCP servers inspired me to write Rapport’s MCP support, just thinking about toykv again stirred up enthusiasm for it.

And so, during June, most of my home coding project hours were spent rewriting much of toykv:

• I updated the on-disk format to use a block-based storage layer.
• I finished up the scan function that allows reading a range of keys. Previously you could only read a single key. The new block-based format involved writing a bunch of iterators that simplified completing this work.

Overall, this made toykv more … real. More like a real database would be structured on disk (though still far from being production quality).

I also found myself having another try at using Zed, as there are now efforts to use the increased focus on Vim functionality to support Helix style interaction.

I spent my “projects time” in the latter half of May working on my AI apps, Rapport, codeexplorer and a bit on my other ai-toys.

First off, after saying that I wasn’t sure whether it was worth adding tool support to Rapport, I ended up going all the way and adding support for connecting to local MCP servers.

Second, I decided that codeexplorer deserved to graduate its own repository. It felt like it had outgrown being part of ai-toys.

Finally, I wrote a streamlit UI around the image generation capabilities of GPT-4o. No more “you must wait 15 hours to make more pictures of gigantic anime cats” error messages in ChatGPT for me!

In All People Are Created Educable, a Vital Oft-Forgotten Tenet of Modern Democracy, we are reminded of the importance of education in the functioning of democracy:

Many shocking, new ideas shaped the American Experiment and related 18th century democratic ventures; as an historian of the period, I often notice that one of the most fundamental of them, and most shocking to a world which had so long assumed the opposite, often goes unmentioned — indeed sometimes denied — in today’s discussions of democracy: the belief that all people are educable.

Again, most of my spare time was dedicated to AI learning and experimenting:

• I continued reading and coding from Build a Large Language Model (from scratch).
• I updated Rapport, my home-brewed chatbot app, a few times. I added a couple of major features and several quality-of-life improvements. I really like it now.
• I did some further work on my codeexplorer ai-toy. Now you can ask a model to edit code, and I added support for more providers than just Anthropic. However, experimentally, Claude is still one of the best models for code editing.

Thus concludes chapter 4 of Build a Large Language Model (from Scratch). Coding along with the book’s examples, I now have an untrained version of GPT-2, first released in 2019. When fed with “Hello, I am” as a prompt, the untrained model outputs gibberish. This post’s title is taken from that gibberish.

Next comes Chapter 5, which will cover the training that will take us from gibberish to intelligible text. But for this post, I wanted to take the time to capture my thoughts at this point in the book.

Rather than explaining concepts that others have covered better, I’ll share my stream of consciousness about how fascinating and weird it is that this stuff works at all.