I especially love the third task because that’s exactly the kind of shit you get thrown on your plate in the field as a SWE.

There’s been some work an old member of our team did a year ago. No one remembers what it was, but it is important. Please do something.

That’s almost exactly one of the first tasks I got as an intern when I was starting out. THIS is what they are saying LLMs are going to replace.

It occurs to me that this audience might not immediately understand how hard the chosen tasks are. I was fairly adversarial with my task selection.

Two of them are in RPython, an old dialect of Python 2.7 that chatbots will have trouble emitting because they’re trained on the incompatible Python 3.x lineage. The odd task out asks for the bot to read Raku, which is as tough as its legendary predecessor Perl 5, and to write low-level code that is very prone to crashing. All three tasks must be done relative to a Nix flake, which is easy for folks who are used to it but not typical for bots. The third task is an open-ended optimization problem where a top score will require full-stack knowledge and a strong sense of performance heuristics; I gave two examples of how to do it, but by construction neither example can result in an S-tier score if literally copied.

This test is meant to shame and embarrass those who attempt it. It also happens to be a slice of the stuff that I do in my spare time.