What I love about low level engineering

Ramblings from a database dev

Another year another new year resolution! Ever since freshman year of college, I’ve always wanted to make something with my own hands - games, personal websites (when that was all the buzz), another webapp, you name it. As any weekend project goes, after a day or two they all fizzled out. Fast forward 8 years, I still haven’t successfully finished any of my weekend projects. What’s ultimately missing I think is external motivation that keeps me accountable, so I made a promise to myself to write a blog about what I’m interested in learning, or show off whatever I’ve built to the world at least once every other month (and talking about this publicly being the external motivator). So if you’re reading this in the future and if this is the only post you’re seeing, you know I’ve spectacularly failed. If it isn’t, you’re in for a treat.

I’ve never really written something like this before, so as my first blog post I’m going to tell you about me. What I do, what I like, why I’m doing the things I do right now, and what I hope to build in the future. So here’s a short TLDR:

Who am I? Software engineer at SingleStore, working in the query optimizer team. I’ve built my own OS from scratch once, and now I get to work on database internals! I love working on low level systems, and I’ll dig deeper into why later in this blog.

What do I want to write about? There are two main themes. Sometimes at work, I see all these concepts that I simply don’t have time to explore during my regular 9-5. Custom allocators, lock-free data structures, compilers, … I take these for granted as an established infrastructure in the codebase, but I don’t really understand why we use them and what tradeoffs exist (well, I know what they are at a surface level, but I can’t explain every detail of it). So I want to implementing these from scratch and play around with it, and also see how other systems implement these concepts. Let’s take concurrency for example. I could implement different versions of the same data structure: a lock-free version, mutex version (but with different implementation of and granularity of mutex) and benchmark them… that sounds fun!

Another theme is fun weekend projects that has nothing to do with systems, but I want to build these with AI tools. I’ve been an AI skeptic for a while, mostly because I want to understand every line of code I write, and I haven’t been impressed with the models’ code generation abilities in the past. But I’ve been hearing the tech and the integration has been getting better by the day, and I want to see if I can use these tools to enhance my learning while boosting my productivity - doesn’t hurt to try!

Still here? These sound interesting? Great! Let me tell you a longer version of who I am and how I got to this point.

Discovering Computer Science

About a decade ago when I was still a high school student, I was convinced that I would be working as a biomedical engineer. Credits to my academic godfather who was a retired surgeon teaching biology, I learned how to learn under his rigorous standards, and naturally grew curious about biological systems and chemistry. During this time I also dabbled in competitive mathematics, but frankly I didn’t enjoy the process. I wanted to make something, and proving a complex theory through logical steps was fun but wasn’t satisfying my soul. I was more interested in something tangible, something concrete, something I can see affecting the world.

I was fortunate to get accepted into Carnegie Mellon University’s (CMU) Engineering school, with the intention of doing a ChemE + BME double major. Little did I know, ChemE actually had very little to do with chemistry. What I did enjoy was the first computer science course I took at CMU called Principles of Imperative Computation. It’s your standard data structures and algorithms class. For me, learning about allocation, pointers, and linked list was an eye opening moment. I fell in love with the idea that you can take a primitive concept, organize them in a clever way, and build something powerful. I loved that class so much to the point where I ended up working as a teaching assistant for that course for 5 semesters throughout my time at CMU.

I was looking through my camera roll from this time of my life and found so many CS memes saved like this. I really was drinking that CS kool aid.

I ended up switching my major to Electrical and Computer Engineering, and started taking as many CS classes as I could. I liked learning what really goes under the hood at the memory level. I liked learning about what operating system is, how malloc works, how the distributed systems coordinate, how embedded systems works through their real-time challenges. The more classes I took, the more I understood all the “magic” behind the technology we use every day (and the more it got difficult explaining to my parents what I’m learning).

Building my first OS

This was during COVID era when all classes were through Zoom. Combined with my desire to take as many systems classes as possible and pure boredom the lockdown world presented, I decided to take 15-410, Operating System Design and Implementation. Many people who take this class will say the same thing, but this really was a transformative class on how I view systems in the perspective of having a good design.

With my group partner (who also happens to be my coworker today), we spent around 40 hours a week to build an operating system for x86 from scratch, testing on Simics and a lot of designing and reading through handout and Intel documentation. Classes I took before this was pretty straight forward; there is an algorithm or some system with clear instructions on how to build it, and we just had to execute. This course was vague on purpose. Project handouts had a specification (the syscalls to implement) and some background information, but it never went into how we should build it.

A grueling semester later, I scraped by with a passing grade and an operating system that is able to do all the basic things - bootstrapping, handling multiple processes & threads, virtual memory, interfacing with devices, etc. Looking back, I struggled a lot throughout the semester. I was not trained for an advanced type of reasoning yet, and I got easily frustrated when the answers weren’t obvious.

Once I felt the walls closing in and cornering me when I couldn’t just “make it work”, it bruised my ego.

What stuck with me, however, the feedback on how we designed it, and the consequence of such design.

Our code was literally reviewed by hand which I find pretty cool

This is one of the feedback I got on the context switching code. One design decision we’ve made is to store thread metadata in dynamically allocated data structure. This made sense to us back then since we had no idea how to do this otherwise. The critical flaw is that there is no way to handle allocation failure in the middle of context switching, which led to some goofy situations like above. This sat with me for a while. How do I even design this out, and how do I have a foresight to avoid this in the first place?

At the end of the semester, I was more comfortable working with this kind of code, but this also made me realize there is so many more considerations and learning to do to build production grade systems software. I’m grateful for this transformative semester. I learned the importance of designing something early and designing it well for software that demands such craftsmanship.

First full time gig at an AI startup in Korea

After my junior year, it was time to do my military service as a Korean citizen. Thankfully the Korean government allows people to work at a startup for 2 years as replacement to the service, so I took that up and worked at VoyagerX building Vrew. At the time we had around 10 developers working on the product, so we all wore many different hats and got the full stack experience. I had a blast working here. We all worked as product engineers, sometimes even participating in user interviews and working closely with designers and PMs working through UX flow, and building out the frontend and backend to bring the feature to life.

One project I keep dear to my heart is the Elasticsearch project. I built an ETL pipeline and an Elasticsearch + Kibana setup for server health monitoring and user action logs. The fun part here is that the database was hosted on-premise, so I also got to experience all the pains that comes with it. Fun anecdote: one month before my last day at the company the only disk completely decide to give out and went into emergency read-only mode. We had no redundancy built in. Me and my manager scrambled to buy new disks that day and worked through the weekend to bring it back up online with disks in RAID array (finally).

Another project I had fun with: Templates. I had strong opinions that we should benchmark Canva and build a video editing template platform

This experience shaped me in a different way that sparked my interested in systems again. Working at a higher abstraction layer, I saw what real life workload looked like. Building a full stack application, owning a feature end-to-end, and iterating on customer feedback really itched the builder side of my identity.

But I couldn’t shake off the feeling that something was missing. I now had some breadth in the field, but I missed that deep technical experience. I wanted to drill down. I wanted to understand a complex system.

I realized I couldn’t stand to not understand something at a very deep level when I work with it on a daily basis.

I often ran into performance issues working with AWS RDS and our on-premise Elasticsearch. I hadn’t taken a DBMS course yet, so how these systems worked under the hood was a large mystery. Beyond some simple explain commands and adding indices, my technical ability to troubleshoot these problems were limited by my lack of foundation. Naturally, my curiosity towards the internals of DBMS gradually grew over time. I knew I had to learn what goes into building it.

SingleStore

After returning to CMU, I took Andy Pavlo’s Database Systems course where most of my questions from VoyagerX were answered. You might notice a pattern now, I had a blast! For one I got a chance to listen to Pavlo’s lectures live, that in itself is an experience. But I also got exposure to all the concepts like buffer pool, join algorithms, query optimizer basics, WAL, MVCC, and more. Around this time I had to make a major decision on where to work after graduation between SingleStore and a big tech company. The nature of the work between the two could not have been more different. At SingleStore, I would be working on a proprietary database engine code, and at the other big tech, I would be working at their e-commerce platform.

At this crossroad, I had few agonizing days of reflecting on what I wanted out of my career. This was an easy call from my parents’ perspective. For South Korean parents, working at a big company is the end game for their children. My mom knows the big tech company from the brand name, but didn’t know anything about SingleStore, so her recommendation was rather clear. Funny side bar, when I told her I would be working at a database company, for months she thought I worked at a company called “Database”.

Though I struggled to decide at a surface level, I knew deep in my soul that I wanted to work at SingleStore. I wanted to work at a low level full time. I wanted to think about memory. I wanted to think about performance. I wanted to keep exercising that muscle that makes me feel alive. I wanted to learn systems and experience what it’s like to work on database internals. Silencing the noise of pushback from parents and the relative lack of so called brand name was easier than I thought when I knew I had to be who I was. That winter I signed with SingleStore, filled with excitement and a pinch of fear of what my next chapter of my career would look like.

Fast forward 2 years, I’ve learned and grown so much working here. I think about every single allocation and all the odd concurrency problems on a weekly basis. Trying to “debug” optimizer behaviors in a complex 20-table join query is a head scratcher. Every day I discover a new part of the engine that forces me to learn something new. The work is challenging and pushes me, in a very satisfying way. On top of the technical challenge, I’m at an environment where I can see the user side of the business at a relatively early career stage of an IC, that scratches the product side of my career aspirations. I’m happy I made the decision to come here 2 years ago.

The joy is in the process of learning and de-mystifying a complex systems software.

At the core of a behemoth, there are fundamental systems concepts that enables high performance and resilience under high traffic. That’s what I love about low level engineering - basic concepts coming together in a clever way to build something incredible, the same spark I felt when I first learned about linked list.

I hope you follow along my journey of learning and having fun for months to come. If you’re working in systems software, I would love to learn about what you work on, and thoughts on what topics I should explore. If you’re a builder, your projects and creations inspire me - I’ll also be building fun weekend projects and hope you enjoy that aspect of my blog as well. Thank you for reading through my ramblings :)

---