Usually Pragmatic

Product Thinking in an AI World

Mon, 24 Jun 2024 00:00:00 +0000

I’m writing this in mid-2024. GPT-4, Claude 3.5-Sonnet, and Google Gemini 1.5 Pro are the world’s leading LLM models, and the ever rumored Llama-3-400B is supposedly just around the corner.

The LLMs are wildly capable: they don’t just write sonnets, or teach you about history, but they’ve even turned into amazing coding assistants. Sure, they still hallucinate sometimes, but most of the time it’s not worse than your average person (or stack overflow answer).

The problem is that no one knows what the future is going to bring–will RAG rule the day? will we need new models with flexible depth-of-compute as per Yann LeCun? will the scaling laws finally fizzle out? or will we simply run out of voltage transformers?–just that it’s going to happen soon. And with tens, if not hundreds, of billions of dollars in play, the mega AI labs feel like behemoths, ready to wipe out entire business models in a glance, or at least a model update.

So what do you do if you still want to play in this space?

The way I see it, there are three main options: find a defensible niche, adapt at the edge, or build foundations.

Find a defensible niche

There are some industries that are hard to enter, or might be too small to be worth it for the big players, or require some kind of specialized knowledge, infrastructure, or process to tackle. Think industries like healthcare (especially with HIPAA), defense tech (clearances, govt contracting), or autonomy (specialized training datasets).

The fundamental bet here is that the extra barriers to entry will give you enough headroom to establish a sustainable business.

Adapt at the edge

If you’ve got the network for it (or the dedicated manpower, or connections, or pre-existing business lines), it might be possible to build a reactive business. That is, to always be on the look out for “the next step” and to simply build out the new products faster than your competition (once it’s clear what they should be!).

This is a road of constant exploration and vigilence. It means you’ll need to do a lot of networking, outreach, and learning, and probably also have your own internal AI R&D effort, even if only for the second-order benefits of improved market insight. You’ll probably also end up sponsoring events and truly trying to build a community. After all, you won’t know precisely where the next product signal will come from… only that you’ll have to find and act on it faster than your competitors.

Build foundations

There are a lot of people within the AI space. I just got back from Seattle, where CVPR broke 12,000 attendees. And yet, that’s a small fraction of the population. For everyone right now pushing the edge of AI systems, there are a thousand more who know they should care, who know they’ll need it, and who wish it was within their reach, but who simply don’t know how to get started. How to put the pieces together and make things happen.

Building foundational tools–note! not foundational models!–for these people will be a great way to bring value and earn happy customers. But what counts as a foundational tool?

The operative question is, for any particular customer segment,

“what can you build that will embody the lessons-learned from the experts-at-the-edge, so that your customer can build their own systems successfully, and without needing to pay the cost to learn those same lessons first-hand?”

My favorite example here is not a software product at all, but the bodies of documents we call building codes. Many lessons have been learned over the years about fire safety, earthquake sturdiness, handicap accessibility, and more, and the beauty of building codes is that they oblivate the need for individual builders to learn those impossibly-many items for themselves. Instead, they establish a standard framework of “18 inches on-center”s and “standard weight Schedule 40 wrought iron”s that empower the end users to put the pieces together for themselves faster, and more confidently and easily, than would otherwise be possible.

Of course, this works for software just as well. “Build the systems that let people build systems” is a time-honored strategy.

Python decorators for fun and profit

Mon, 22 Apr 2024 00:00:00 +0000

Two python functions:

1
2
3
4
5
6
def print_status_and_execute(f):
    print("running function " + f.__qualname__)
    return f

def add(x, y):
    return x+y

You might want to wrap them like:

1
status_printing_add = print_status_and_execute(add)

Which is kind of ok, except that now you have to change all your code wherever you used to use add to use the new status_printing_add function.

Instead, you can just decorate the function definition of add like so:

1
2
3
4
5
6
7
8
9
def print_status_and_execute(f):
    def fn(*args, **kwargs):
        print("running function " + f.__qualname__)
        return f(*args, **kwargs)
    return fn

@print_status_and_execute
def add(x, y):
    return x+y

And now every instance of add behaves as if it is automatically transformed: add(a, b) -> print_status_and_execute(add)(a, b)

You can even make decorators themselves depend on arguments, though it requires an extra level of abstraction:

1
2
3
4
5
6
7
8
9
10
11
12
13
def print_n_times(n):
    def decorator(f):
        def fn(*args, **kwargs):
            for i in range(n):
                print(*args)
                print(**kwargs)
            return f(*args, **kwargs)
        return fn
    return decorator

@print_n_times(5)
def say_hello(name):
    print("hello " + name)

Or do fancier things, like create and maintain extra data structures available to their context across function calls:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
def memoize(fn):
    prev_res = {}
    def mem(num):
        if num in prev_res.keys():
            return prev_res[num]
        else:
            val = fn(num)
            prev_res[num] = val
            return val
    return mem

@memoize
def fibb(n):
    if n == 0:
        return 1
    elif n == 1:
        return 1
    else:
        return fibb(n-1) + fibb(n-2)

Course Notes -- Fast.ai

Tue, 19 Mar 2024 00:00:00 +0000

Intro

I’ve been meaning to run through the Fast.ai course on practical deep learning for about a year now, since I first learned of it, and it’s finally time. As usual, I’ll be burning down through this quickly, and thankfully have a pretty deep background in the gnarlier, more mathematical aspects of the field… (I’ve derived and implemented backprop by hand a number of times, and most recently on the professional side, was SVP product for a startup building LLM Inference HW, so I’ve become quite familiar with the underlying theory and architectures).

But now it’s time to roll my sleeves up and learn the hands-on tools.

I’ll be using this space to track and update any particularly interesting things I learn along the way.

Part 1

My goal here will be to go through the fast.ai Part 1 sequence and produce and deploy an ML model accessible directly from this project page.

Things I learned:

yt-dlp is a great command line tool for downloading youtube videos. Highly useful for making it easy to adjust playback speed (using [ and ] keyboard shortcuts) with mpv (a close analog to mplayer with good mac support).
timm is a pytorch-based deep learning library collecting a number of pre-existing image models.
I cannot believe I hadn’t encountered python’s functools.partial() before. I’ve been used to this sort of functionality since my days messing around with making a solver for hateris in haskell ages ago, and I’ve always rolled my own in python using lambda functions. But of course there’s a built-in for that now.
ipywidgets.interact is another major quality of life improvement. However, as a reminder to myself, this is not enabled by default in jupyterlab.
Python decorators, which have been on my “to-learn” list forever, and have finally drifted to the top.
The use of * and / in python argument lists in order to enforce the allowable order of positional vs keyword arguments.
Nice to use log() for reducing the domain of distributions. (But more important to just be aware of, and interrogate, your data distributions!)
Somehow, I missed that list comprehensions in python also extend to dictionaries and sets! Cute.
I had not encountered the book Python for Data Analysis before, but it’s a solid resource on some of the internals and tools (especially in pandas) that I was less familiar with.
I was not previously familiar with SymPy at all. Seems legit.
Random Forests: I had heard of these before, but never actually learned them. To be honest, I’m a little disappointed. The process is cute, elegant, and simple. But damnit, it’s about as crude as a blunt rock. Chop your dataset into random subsets, each with a random subset of all of the features, and train a bunch of decision trees (one on each subset of your data). For predicting, take the average value of all of them. Or maybe the mode, depending on whether you want a quantized result or not. It works, but it’s literally just duct-taping random shit together. Though sometimes, that’s all you need.
Test Time Augmentation is a cute trick that seems especially amenable to image models for potentially improving output accuracy.

Part 1 - Project

This is a transformer-based number-classifier, trained on MNIST. More details about the specific architecture I experimented with below. To use, draw a digit 0-9, and click submit.

0: 0%

Recipes

Mon, 12 Feb 2024 00:00:00 +0000

Shortbread Cookies

Makes about 3-4 dozen cookies

226g unsalted butter
90g powdered sugar
1 tsp vanilla extract
280g AP flour
25g cornstarch
1/2 tsp salt

For best results, measure ingredients with a scale by weight.

Mix together and refrigerate dough.

Bake at 350degF (175degC) in a preheated oven for about 13-15min or until slightly golden brown.

Sernik (Polish Cheesecake)

1.37kg twarog (farmer’s cheese)
100g sugar
somewhere between 1 tsp and 1 tbsp vanilla extract
5 eggs
50g corn starch
200mL heavy cream

Put the twarog in the mixer, add sugar, and vanilla extract, and mix. Then add eggs, and mix. Add corn starch, and mix. Add heavy cream, mix.

Take a 12”x8”x2” baking form, coated with butter, and pour the mix in (no higher than 1/2 to 2/3 of the height of the pan).

Bake at 170degC (340degF) for 90 minutes (or until light brown on top). After 90 min, turn the oven OFF. Do not open the oven until it cools down!

Cornbread

Makes one small lodge griddle

4 tbsp unsalted butter
1/2 cup cornmeal
1/2 cup all-purpose flour
2 tbsp sugar
1/2 tsp kosher salt
1 tsp baking powder
1/8 tsp baking soda
1 egg
4 oz greek yogurt
6 oz buttermilk

Preheat oven (425F) and brown butter, mix ingredients by hand (no blender), and bake for ~20min. Works great in a cast iron griddle.

Banana Bread

Work in progress.

Mochi Waffles

Makes 4 waffles / one ‘square’ waffle-maker load

Dry Mix:

1/4 cup oat flour
1/4 cup AP flour
1-1/4 cup mochiko
2 tsp baking powder
1 1/2 tsp salt
1/2 tsp cornstarch

Wet Mix:

1/2 cup grass-fed whey isolate
1-1/4 cup whole milk (mix the above and let the protein powder hydrate) then…
1/2 banana
1 egg
a dash of vanilla extract
1 tsp lemon juice
2 tbsp melted coconut oil (or butter) (blend all of the above)

Fold the resulting wet ingredient blend into the dry ingredients and mix until just combined. Pour into hot waffle maker and enjoy.

Notes: Adjusting this again, to try to improve moisture while maintaining chewiness

Athletics

Thu, 04 Jan 2024 00:00:00 +0000

Update - 1/04/24

Well, this is a few years out of date. The whole pandemic lasted a lot longer than I think anyone really expected. Especially with the California lockdowns.

I wish I could say that left me with smooth and continuous progress, but it wouldn’t be true. We moved, had a kid, and I encountered a small host of minor, but lengthy, injuries. Along the way, I ended up making some changes to my routine.

For lifting, I switched out of the tactical barbell program (with its focus on 1RM testing), and into a more “bodybuilding” style of lifting. 5 week long phases, with a lot more volume (sets of 10+), and a gradual peaking followed by a recovery week. I started this new routine in mid-December, 2020, and am currently halfway through Phase 19. At 5 nominal weeks / phase, that means I’ve been about 60% efficient with my lifting compared to an ideal schedule.

I’ve also ended up with a better table tennis setup at home. I’ve now got a table, robot, and most importantly, enough room to use it, so drills with that have become a regular part of my routine. I’ll write about my shot accuracy tracking system in a later update.

Overall, my best guess has me in pretty good overall shape. I definitely don’t have the level of conditioning that I did when I was spending 10+ hours a week at the table tennis clubs pre-pandemic, and I definitely don’t have the level of raw, 1RM strength that I did at the peak of that training, but I’m at an acceptable middle on both of those, with a clear improvement in strength-endurance (from the focus on greater volume of reps) as well as body composition.

In the meantime, I’ve got clear directions to go for improvement, and the most important step is the next one.

Update - 10/27/20

Latest graphs at the end of a bulk cycle.

Bench press 1RM stayed the same, but my shoulder injury is almost totally healed, so… win some, lose some?

Not pictured here is my progress with kettlebells which I’ve added as a recurring part of my routine (because really, who computes 1RMs for kettlebell swings?). In any case, I’ve worked my way up to 2-hand swings with a 24kg bell, 1-handed swings with a 20kg bell, and Turkish Get-Ups with a 20kg bell. Unfortunately, I’ll pretty soon be maxed out with the kettlebells that I have (only have up to 24kg), so I’ve started to keep an eye on the usual suspects (Rogue and Vulcan, I’m looking at you!) for their next restock.

Update - 9/2/20

Latest graphs at the end of another 6-week Tactical Barbell cycle.

My bench press dropped a bit due to sustaining and recovering from a minor shoulder injury.

Update - 6/14/20

Latest graphs at the end of another 6-week Tactical Barbell cycle.

Where I began

As a child, I never really did any form of organized sports. Sure, I did some karate classes, and even participated in a t-ball league for a few months in… first grade?, but I have almost no memory of those times, and it wasn’t sustained enough to count.

My first real experience with athletics instead came from joining the cross-country and track teams during freshman and sophomore year of high school.

To be honest, I think my main reason for joining them was because that’s what the nerds did at my high school. Everyone knew that PE sucked, and for some reason or another, there was only one alternative for nerds: cross country in the fall, and track & field in the winter and spring. So that’s what my friends and I did.

I hated cross country. Running long distances was, and still is to me, an exercise in ever-increasing boredom and pain. I learned that I could do it, as in, I could physically move my body through space for the eight or nine miles of any given run, or the more intense but shorter 3 mile races, but I also learned that I didn’t enjoy it. Not by a long shot.

Track was better. You could choose your events! They had sprints! I primarily ran hurdles (I was tall) and the 400m. And unlike cross country, since everyone wasn’t competing simultaneously, you could actually go watch and cheer on your friends. Everything about this was fantastic.

Track also taught me some valuable, though somewhat tangential, skills: how to cheer really, really loudly; how to mentally cope with interval sprints; and that if you intentionally lie to your friends by calling out their split times a few seconds slower than they really are, you can get your friends to try harder and thereby set new PRs, and while they’ll thank you afterwards, they’ll also think you’re a maniac.

Once sophomore year was over, I had fulfilled my 2-year “PE” requirement, and I dropped the organized athletics entirely to focus on the more important math and science competitions in my life.

Throughout this period, I had only minimal exposure to strength training. I had seen the inside of a weight room before, and of course I had seen any manner of targeted weight machine, but I did not understand them, nor did I feel even remotely comfortable with them.

College years

Going into Caltech, I wanted to get back into athletics, and I had an growing interest in strength training.

Over the summer before my freshman year, I had picked up a copy of Arnold Schwarzenegger’s “The New Encyclopedia of Modern Bodybuilding” which was, by weight, 99% an encyclopedia of every exercise you can imagine, and 1% thoughts about how to structure them together as a coherent program. Nevertheless, once rotation was over and all the chaos of move-in and adjusting to the new world had settled down, I began to hit the weight room—and track, for more of those interval sprint workouts!—on an extremely irregular basis with a variety of friends over the years.

Notable memories include watching a post-doc I knew doing bench press with ~140lbs on the bar—he was using the big plates!—and visiting the much less-well air conditioned, much grittier, but much more viscerally enjoyable weight room at Los Alamos during the summer research fellowship I did there.

By far, though, the most major thing to happen to my athletic trajectory during my time at Caltech was about two weeks before I was supposed to move in early for pre-season track & field training (hurdles again!), when I was flipping through the course catalog and saw a listing for “table tennis”. Three thoughts passed through my head in quick succession: college is about trying new things, right?; I enjoy playing ping pong in my back yard…; and “you know… hurdles are kind of boring…”.

I switched over.

In pleasing symmetry, there were three important things that I didn’t realize at that point:

I had accidentally signed up for the advanced section.
The coach, Wei, used to play professionally in China, and played on the US Olympic squad after emigrating.
Table tennis was a sport that many learned to play from a very young age, and most of those in the advanced section had been playing for years.

It was immediately obvious that I had no idea what I was doing.

After the first session, the coach came up to me and asked if I was sure I had signed up for the correct section, and explained that maybe I would be better off switching on one of the beginner/intermediate groups. Not to be deterred, I decided that instead of switching sections, I would simply attend all of the sections every week. This would prove to be one of the best ideas of my life.

Over the next four years, I became, though certainly not the best, one of the better players at Caltech, and built wonderful, and enduring, friendships with many people through the table tennis community. My lengthy discussions about technique with Wei easily taught me as much physics as several of the courses I was taking for my degree, and the stability and sleep-requirements of that training became a rock that I could lean on to ground and support myself through the harder trials of those years.

Notable memories include one exhausting day in San Diego when we decided to field three teams amongst a total of 6 people at a NCTTA tournament; getting to coach our Women’s team at nationals for two years in a row (first in Columbus, OH, and then in Rochester, MN); and traveling up to Stanford in 2007 to watch the World Junior Championships being hosted there.

2008-2010

The two years after I graduated were more tumultuous for me. I moved across the country twice before finally settling in Mountain View, I switched jobs several times, and finally ended up diving into the startup world while working on natural language processing applied to email database analysis.

During this time, I continued to play table tennis at whichever club was most convenient for wherever I was living, but the organized training I enjoyed while at Caltech was largely absent.

Meanwhile, I had no regular gym/weight room access, so that ended up on the back-burner.

2010-2016

Before moving up to Silicon Valley in 2010, I became more seriously interested in strength training, picked up a copy of the Starting Strength book, and a set of Ivanko weights and barbell off of someone on craigslist in Los Angeles. When I finally moved into a place with a garage, I built a squat rack and a bench—both wooden! I know!—and got to training. The bench was a beautiful piece that I still have and use as regular furniture. The squat rack was a wobbly, scary piece of kit with only two hook positions available (one for bench press and one for squats), but hey, at least it had safety rails. And it was cheap.

It was with that setup that I set a number of PRs. My working weight for three sets of five reps passed 250 lbs on squats, 150 lbs on bench press, 115 lbs for standing press, and 300 lbs on deadlift (ok, just for one set of five reps, not three sets on deadlift). During the later part of this period, I moved again, no longer had a garage, and joined Boss Barbell Club, where in that impressive company, I notably improved my working weight for squats up to my all-time PR of 275 lbs. You learn a new kind of respect for gravity when you pass the point of having more than your entire body weight on a barbell on your back, so I can only imagine the focus and dedication required of those athletes at BBBC whom I saw squatting 500, 600, and even 700+ lbs.

This time period also led to a renaissance in my table tennis training. If you’re familiar with table tennis in the US, you know that California is one of the hot spots. And if you’re familiar with table tennis in California, you know that the bay area, in particular, is Mecca.

If you’re not familiar with the scene, understand that in most areas of the US, and even elsewhere in the world, table tennis clubs are often share their facilities part-time with other groups, either as part of independent businesses, or as part of a community recreation center. Thus, while it’s not unheard of, it’s also not the most common thing to find a club with a dedicated space. Even once you find such a club, it’s rarer still that they will be set up and equipped with professional-grade flooring and lighting—things that make a significant difference for the quick, lateral movements of the sport, or when lobbing and smashing high balls.

The bay area is composed of many small cities, and each city has a dedicated club with professional flooring and lighting. Some cities have several. Opposite the situation elsewhere, it is rare here to find a club that only has a part-time facility, or one that subjects players to playing on wooden or tile flooring. Professional-grade equipment, dedicated facilities, and top-tier players are de rigueur.

For further comparison, in last US National Championships in 2019, every one of the four finalists in both the women’s and men’s singles came from a club in the bay area, and in fact, from a group of three clubs, all within a four-mile-radius circle on the map.

Playing table tennis here has been fantastic. After browsing between several clubs, I picked one and have been a regular at their trainings and league events (though I still drop in at other clubs, time permitting), and my technical understanding and ability with the game have continued to improve.

2016-2017

These years were challenging for me due to an incidence of back injury. On several occasions, I ended up tweaking my back to the point where I could not move out of bed the next day without excruciating pain. I reset my working weights for lifting back down to zero several times, and I had to take care to slowly reintroduce even the intensity of my table tennis training.

I struggled for a solution until I encountered the work of Stuart McGill, who I eventually ended up visiting in his home office in Canada. His life’s work is understanding the biomechanics of the back and torso, and working with individuals and athletes of every caliber (including many olympians and world record holders) to rehab and return to form after injuries.

His methods were magic. Not in the sense that he was able to speak a short incantation, wave his hands, and bam my problems were resolved, but in that unlike any of my prior consultations and appointments, he went into great depth to understand the specific issues I was facing, and his recommendations, like the McGill Big 3, when applied diligently, finally and completely resolved all of my problems.

After that, I also adjusted my strength training routines to accommodate my McGill-inspired focus on resiliency and stability: I traded barbell squats for Bulgarian split squats, and moved all of my deadlifting to sumo-style.

2018-2020

The last few years have seen, by far, the greatest progress in athletics of my life. This comes down to three things:

A change in my responsibilities with my job meant that I no longer had to travel to Florida one week out of every four. Hello improved consistency!
I changed strength training programs. After much searching, and some trial and error on my own, I settled into a routine following the progressive training templates in Tactical Barbell. The problem prior to this is that the energy demands of my lifting and table tennis training in parallel had grown to be too large, and so I would often find myself either tired at table tennis from my lifting the night before, or experiencing slower gains in lifting due to the long sessions of table tennis. The Tactical Barbell program is the best I’ve encountered yet for structuring strength training as a supplementary activity to your main sport, and has supported my progress in both, wonderfully.
The RP App. A close friend of mine went to work for that company, and recommended I try out their app. Now, I had done various forms of diet tracking before, following templates, and even building an eventually-surprisingly-exhaustive database of nutrition facts for Trader Joe’s products, I had only really had relatively middling success. I often had difficulty maintaining the trajectories of my bulk or cut diets, and attempting cut diets, especially, was always miserable.

Enter RP. Like you might expect, to use the RP diet app, you have to tell it what kind of a diet you want to do (muscle-building, maintenance, or fat-loss), and some basic things about yourself and your schedule (height, weight, daily wake-up & sleep times, and workout timing and duration). What you—and I, certainly!—might not expect is that not only will it tell you how much protein, carbs, and fat to eat each day, but it will also tell you when to eat it. Things like “1:15pm - 30g protein, 15g fat, 20g carbs”. This makes a huge difference. By getting your individual meal timing and sizing right, you can optimize for your individual metabolism, and, when on a fat-loss diet for example, can give your body the regularly-spaced doses of protein it needs to maintain muscle mass, while still being in a caloric deficit to encourage fat loss.

Bonus points to the RP app for also giving you an easy way to select what you’re about to eat from a whole range of different foods, cooking styles, and even by scanning barcodes, and then automatically calculating what total weight of foods you should consume to hit your target macros.

So that brings us to roughly today, and the events of my Quarantine Projects.

Finally, I’ll end with some progress graphs from my use of the Tactical Barbell routine:

Thoughts on "The Twelve-Factor App"

Tue, 02 Jan 2024 00:00:00 +0000

These notes are a stream-of-consciousness response to reading through The Twelve-Factor App at the suggestion of a good friend of mine.

Introduction and Background

Nothing particularly special here. At this high of a level of abstraction, everything sounds like a good and reasonable idea!

Section 1: Codebase

Not too much controversial in here (I hope!). I like the emphasis on strict delineation of codebases between apps, and factoring out shared code into separate libraries under dependency management.

Section 2: Dependencies

This one requires some rigor and automated test tools for enforcement!

Their ending note about extending the philosophy to system tools makes plenty of sense, though it does start to make me wonder how they would handle system tools that might need to be rebuilt for different micro architectures… seems troublesome to end up with nested system compatibility issues…

Section 3: Config

This one is an interesting distinction. Of course you don’t want to save credentials and the like into your code repo, but storing all config as environment variables feels a little like you’re just pushing the problem down the road… you still need tools to set up and manage those env vars for any sufficiently complicated setup, and especially if a single user might need to switch between various configs (when, say, deploying to dev vs test vs qa) it feels like it would be very easy to accidentally end up in a world where your config is now stored in a management program or downloaded from a source location.

Section 4: Backing Services

This is a reasonable set of definitions and goals, but leaves a little bit of documentation and organization discussion to be desired. For example, if you’re using a SQL database as a resource, you may also need to track specific version requirements for that database implementation–especially if you’re relying on specific and/or unusual functionality–but even in the general case, you may want to specify version requirements to align with your testing environment, even if only to make sure to be able to replicate behavior where bugs may be concerned.

These sorts of situations would require some extra state to be stored somewhere beyond just the resource handle.

Section 5: Build, release, run

Again reasonable, with some additional potential trickery around dealing with exploding database sizes in cases where the release may include large binaries / etc.

Section 6: Processes

Very reminiscent of my time with Haskell. I dig it, but there are probably consequences of this approach that I’m not immediately appreciating.

Section 7: Port Binding

This is one I’ll tag for deeper discussion. There’s definitely some stuff I don’t fully grok about sentences like “does not rely on runtime injection of a webserver into the execution environment to create a web-facing service”.

For example, what would doing so even mean or entail? Why might you be tempted to do so?

Section 8: Concurrency

A few questions in here also. What does “daemonizing” mean? What are PID files? Is there some formalism for “process types” that I should be aware of, or is it just a question of nomenclature, fully under the control of the devs?

Section 9: Disposability

This again seems like a “mathematically pure” thought, but one that’s devilish to police via testing in practice. In particular, the linked wikipedia page on “reentrant” was surprisingly subtle.

Section 10: Dev/prod parity

Nice, here’s some discussion of the issues I was thinking about from the “Backing Services” section. Overall this seems like an obviously good idea. I wonder what situations would make it maximally difficult.

Section 11: Logs

Huh. This is definitely an idea in the same vein as the previous ten sections, but it surprised me on first read nonetheless. Just logging everything straight to stdout and leaving log routing and collection to the execution environment definitely makes sense given the rest of this framework. I’ll have to chew on this one a little more to see what I really think of it, but so far no objections, even if it is a change in how I’ve been thinking.

Section 12: Admin processes

This doesn’t actually seem like it’s saying much of anything at all. Basically the message is “standardize everything you can, and use tools to make sure of it”.

Is there something more to this that I’m missing?

London Recommendations; pre-pandemic

Sat, 11 Dec 2021 00:00:00 +0000

I know some people headed to London soon, so for them and anyone else, here are my London recommendations. Do note that these all date from my trips before the onset of covid-19, so you should check the actual availability / openings of things for your specific visit.

Cocktail Bars

Dandelyan is absolutely one of the best bars in the world, and certainly the most pioneering of the cocktail bars I’ve presonally visited. These are the sort of people who don’t just use a banana syrup in their drinks, they spend weeks making their own banana syrup, from a plethora of different banana species and individual fruit at different levels of ripeness, each ingredient prepared individually to highlight certain flavor elements in the final syrup. The correct way to visit Dandelyan is to show up as close to their opening time as possible—early in the day!—and grab seats at the bar. Then talk the bartender’s ear off. Ask them about the background behind the drinks. Ask them about the ingredients. How they refined the recipes. What surprised them the most. Order whatever you want, but don’t feel pressured to finish everything, or even anything! The best visits here will give you a chance to sample flavors you simply won’t find anywhere else, and it’s perfectly fine to leave half-finished drinks when you’re done. Delight your taste buds. Save your liver.
Mr Fogg’s are top-notch cocktail bars with a lean towards flair and adventure. If Dandelyan is a science lab, then the Mr Fogg’s bars are Disneyland. Tons of fun, really good drinks, and plenty of atmosphere, theme, and delight. You simply can’t leave without a smile. Mr Fogg’s are best visited a little later in the evening. Your goal shouldn’t be to grab time with the bartender while it’s still uncrowded; instead, get there only a little bit on the early side, so that you can watch it fill up, kick back, and enjoy the celebration that’s sure to surround you.

Restaurants

Hoppers is one of my favorite restaurants in the entire world. It has a focus on Sri Lankan and South Indian food, so expect bold flavors! But everything, everything, is delicious. Special shout-out to the bone marrow.
Dishoom is a series of Indian restaurants in London. They’re much bigger than Hoppers, and tend to have somewhat less bold and specific flavors, but are still delicious, and just extremely solid for a meal.

Other

Timber Yard Seven Dials is my favorite coffee shop in all of London. I love the crowds. I love the coffee. I love the location. And I love the snacks. I’ve set up shop there for days at a time, speed-running an online course, just as easily as I’ve used it as an excuse to step in from the cold and enjoy a quick, warm drink.
Borough Market is the best damn food market I’ve ever encountered, only Torvehallerne in Copenhagen comes close. Come for the duck wraps. Come back for the charcuterie. Come back again for the cheeses. Come back yet again for the baked goods. Come back one more time for whatever else you feel like. It’s all great. Then hit up the hot chocolate from across the street. It’s all fantastic.
High tea is worth doing at least once. On every week. Of every trip. Honestly, it’s a little kitsch, but it’s delicious, it’s fun, you’ve got a billion excellenet choices to pick from, and it’s so London. Note that some of the more expensive options have a refill policy that can actually be quite a good value for your money if you’re hungry enough. For example, I have definitely not ever taken advantage of the Langham’s infinite refills on sandwiches to consume in excess of 23 delicious little tea sandwiches in a single sitting.

Non-Food Thoughts

Have at least one set of nice clothes. There are plenty of places in London which, unlike California, actually have dress codes. This means at least pants, a button-down shirt, and a blazer for men. Flip flops and sneakers need not apply.
It’ll probably rain. You can deal with it. You won’t melt. And honestly, while you should pack a reasonable coat, you probably don’t need to bring an umbrella. If it’s raining that hard, you can always buy a cheap one when you’re there, or borrow one from your hotel.
Shows are a lot of fun! You should try one!
Festivals / exhibitions / etc are worth watching out for. Something or other is always passing through London, and you can easily lose a day—or more!—in any of the excellent museums.
London has a bunch of great bookshops! Even though I do almost all of my reading on something with a screen these days, I love browsing around and picking up something with pages every once in a while.
Yelp has been really quite reasonable for me in London. Also Uber.
There’s really not much need to plan day trips in advance if you don’t mind taking the train (and you shouldn’t! it’s great!). Just show up at the station, buy a ticket, and grab the next one.
If you’re visiting London for a while, dealing with laundry can be surprisingly tricky. Hotels often charge very high rates, so you may be best served with a local wash-and-fold service.
The parks and walking paths are great. There are few things I enjoy more than meeting up with friends in London and catching up over a walk. In any direction. From anywhere. To anywhere. It’s great! Bonus points if you end up passing through one of the many parks, which are always lovely.
Shopping is… a thing? I guess? Fortnum & Mason has great tea, and we always bring back a bunch with us whenever we go. Beyond that, London is certainly a major metropolis, with more than enough wealth and fashions to support just about any sort of shopping experience you’re looking for.
There are tourist attractions. They usually have lines. At least, I’ve usually seen lines whenever I’ve walked by them. I dunno, it’s not really my thing. I’d much rather wander through Soho with a cup of coffee, or crash a table tennis club somewhere.
Speaking of which, activity clubs are myriad in London, and well worth looking up for your trip. Whatever your hobby, you’re very likely to be able to find some club or gathering for it in the city during your visit, and that’s a great way to make friends, have a great time, and get the inside scoop on whatever you want from some locals.

Course Notes -- Coursera - Andrew Ng's Machine Learning Introduction and Deep Learning Specialization

Mon, 16 Aug 2021 00:00:00 +0000

I’m back with another round of course notes, this time from Coursera’s Machine Learning, by Andrew Ng, as well as the entire Deep Learning Specialization sequence.

My history with these courses and material was curious. Much of the background mathematics, including calculus, I became extremely familiar with through the course of my physics degree years ago. Then, back around 2011, I went through the first run of Caltech’s Learning From Data online class which was, as is usual for Caltech, a tour de force of even more math, to really drill to the core of some of the most fundamental concepts in the field of machine learning. But at the time, it was obviously missing some of the more recent focus on neural networks, so I put in a mental bookmark to return to the subject and catch up on more recent developments once I had the opportunity.

That opportunity came recently, as a friend and I worked through the OG Machine Learning Introduction class on Coursera by Andrew Ng. This was a really great survey, but still felt a little dated, and certainly lacked many of the more recent developments in Deep Learning, so I still wanted more.

Done with the intro class, I started looking at Coursera’s Deep Learning Specialization sequence: a series of 5 courses, focused directly at deep learning techniques up to about 2017-2018.

The fateful moment came when I noticed that Coursera wanted a $49/month subscription¹ to access the course materials, but gave a 7-day “free trial” period.

Now, if you know me at all, you know that I don’t go for half measures of “slow and steady” progress. I’ve even written an essay on the subject called Slow and Steady is Bullshit.

So I saw the words “7-day free trial”, and read them as a challenge. All I’d have to do would be to exceed my usual pace of 1-week of a course per day, and accelerate it to 1-entire-course per day. Keep up that pace for five days, and I’d be done with time to spare.

If this sounds insane, that’s because it probably was (at least a little bit?). It was also only conceivable through the confluence of several factors: my previously-established deep familiarity with mathematics, my fluency with the python programming language (which was also the language of choice for the courses), the fact that the courses in the specialization sequence were shorter than is usual for semester-length classes (~2-4 weeks of material each, rather than ~12), and the browser console one-liner $('video').playbackRate = 3.5 to override the default maximum playback speeds of the lecture videos.

So I targeted a weekend, took a couple days off on either side, canceled or turned down all of my other plans, and got to work.

Fallback: this browser does not support PDFs. Please download the PDF to view it.

Download my course notes as a PDF.

Fallback: this browser does not support PDFs. Please download the PDF to view it.

Download my course notes as a PDF.

Long story short, I succeeded right on plan at a pace of 1 course per day, finishing the entire Deep Learning Specialization in 5 consecutive days. This was intense, and by the end I was literally dreaming about multiheaded attention and LSTMs. But, on the other hand, I’m now psyched and ready to dive into even newer developments in the field of deep learning since the 2017-2018 cutoff dates of the specialization.

I don’t know that I would recommend this extreme speedrun approach to others. In my case it was literally only possible because I already had a very solid background on all of the tools necessary for the course, and heck, it even helped that I had already spent a while getting used to the sound of Andrew Ng’s voice on significant speedup during the Machine Learning Intro class!

That said, if you want to learn through these courses yourself, figure out what sacrifices you can make, and then do it as fast as you can. Slow and steady is still bullshit.

$49 / month is really a very reasonable amount of money to charge for access to the quality of material in the Deep Learning Specialization. No qualms there, and hats off to Coursera for providing such excellent lectures, notes, exercises, programming assignments, and exams.

Unit Vector Orthogonality and What It Means For You

Sun, 04 Jul 2021 00:00:00 +0000

I talk a lot about comparative advantage, and finding ways to trade on it. But one of the things I’ve never directly defended is the idea that strong comparative advantages can exist, and that you can improve your chances of developing such comparative advantages by studying more or less anything at all.

So, here’s a quantitative estimate.

Imagine forming a multi-dimensional vector representing your level of productivity on various tasks. Now, asking whether you are likely to have serious comparative advantages compared to someone else is akin to asking whether your productivity-vector points in a different direction to theirs. What are the odds of that happening, even in the worst case scenario, where your study is left to chance?

To get an idea of the numbers, we’ll pick some different dimensionalities, generate a bunch of random unit vectors of each dimensionality, and just look at the angle between the vectors of the same dimensionality. To make things even more concrete, we’ll pick a cutoff (in this case, ±2°) and ask “What is the chance of two randomly generated unit vectors of a given dimensionality being within ±2° of 90°?” This will tell us not only the chance of having a significant angle between the two vectors, but the chance of the two vectors being nearly maximally un-aligned. Or, in other words, the chance of really significant comparative advantages existing.

So there it is. Study things—almost anything at all, and the more the better—and you’ll be practically guaranteed to develop significant comparative advantages that you can use for mutually beneficial trade.

Communication, Management, and Leadership in Startups

Thu, 01 Jul 2021 00:00:00 +0000

– tl;dr –

To communicate well, apply techniques of error-correcting conversations, especially whenever someone says something that sounds dumb, or otherwise doesn’t make sense:

Make sure you’ve heard their words correctly. “Did you just say <the words you heard>?”

Use your understanding to create a prediction for an implication outside the immediate bounds of what they’ve said, and see if their understanding agrees with your prediction. “Does that mean that we would also expect X/Y/Z?”

Rinse and Repeat.

Management is about developing comparative advantages, both within yourself, and among your teams. The goal for yourself is to minimize context switching costs on a daily basis. The goal for your teams is to ensure that no matter what surprise tomorrow brings, your team has a spread of comparative advantages that will let it tackle the work with efficiency and effectiveness.

As an addendum, when hiring, forget the adage about “T-shaped people”. Instead, seek out and hire Koosh ball people: people with multiple, separate areas of deep skill. This maximizes the chance of your team members having really distinct and applicable comparative advantages, no matter the challenges ahead.

Leaders have two jobs: to cultivate pervasive trust, and to delegate well.

There’s a lot said about culture, and most of it is BS. The only thing that matters in a startup’s culture is whether or not all members of the company share a deep trust with each other. The operative question is “Would they take a bullet for me?”. When asked from anyone’s perspective, and directed at anyone else, the answer had better be “yes”. As a leader, judge every action, policy, incentive, and plan with regards to whether it deepens or depletes trust.

Delegation is a two-way contract. It must have a clear vision and strategy defined from above, and tactical authority and true acceptance from below. The delegator specifies the goal and the delegatee specifies what they need to accomplish it. Vitally, the delegatee must truly be able to say no if they deem the task impossible given the resource constraints.

Startups are hard. These are my guidelines for making them a little bit easier.

These thoughts are, of course, applicable to many other situations in life as well, but by using the word “startup”, I hope to attract the attention of people in startups, for whom this advice may make a tangible difference between success and failure.

This is the most directly actionable advice I have, distilled from more than a decade in the field, and I wish someone had presented it to me, in just this way, when I was new.

Communication

The typical communication involves three distinct translation steps.

First, an idea in one person’s mind must be translated into words that are spoken.

Second, the spoken words are translated into heard words.

And third, the heard words are translated into an idea in the second person’s mind. Each of these translation steps is an opportunity for errors to creep in, and miscommunication to result.

this happens to everyone

This is the origin of a peculiar paradox: everyone has heard plenty of other people say dumb things, yet when asked, everyone will tell you that they try very hard not to say dumb things themselves!

The solution is that most of the time when something sounds dumb, it’s because the idea that you’ve ended up with in your mind (after three steps of potential translation error) doesn’t match the idea that was in the speaker’s mind. Thus, they may have been trying to express an idea that made perfect sense to them, but by the time it’s reached you it just seems sort of stupid… because it’s not the same idea.

Most of the time you can resolve these sorts of issues by borrowing a page from the scientific method.

Step 1: Clean your data. Make sure you’ve heard them correctly by just repeating their words back to them and asking if that’s what they’ve said.

Step 2: Come up with a prediction about an implication of the idea that’s outside the direct bounds of what they’ve just said, and see if they agree with your prediction.

Step 2 is usually the tricky one. The idea here is that since there’s no way to directly compare the neural pulses in your brain with those in theirs, the best you can do is try to make sure that all of the predictions that arise from the idea in your mind match all of the predictions of the idea in their mind. If this is broadly true, then for all practical intents and purposes, the two ideas are the same.

Here are some examples of what this might look like:

Person A: “Unemployment is bad.”

Person B: “Do you mean that no one should be allowed to quit their job and take time off?”

Person A: “No! Just that if too many people want jobs and can’t find them, it’s not good.”

– or –

Person A: “Facts aren’t always true.”

Person B: “What? Like linear differential equations don’t have unique solutions?”

Person A: “Naw, just that when someone shows up with some data, you have to question it! How was it collected? Is there a bias towards one viewpoint above others? Which industry group paid for it?”

– or even –

Person A: “This surface needs to be smooth.”

Person B: “So you mean you want it to be polished so it doesn’t feel rough?”

Person A: “More than that! We have to make sure that reflections on any part don’t have sudden changes!”

Management

Management is about developing and ensuring comparative advantages, so that tasks can get done as efficiently and effectively as possible.

If you’d like a refresher on comparative advantages and why they’re important, check out the beginning of my essay on Comparative Advantage and the Theory of Intertemporal Self-Trade.

For managing yourself, you can create comparative advantages between different days by trying to arrange your schedule to minimize context-switching costs. This is related to Paul Graham’s idea of the Maker Schedule vs Manager Schedule, but slightly more expansive.

Of course, a la PG, it’s better to schedule meetings together in a concentrated batch, and ideally to keep them fully separate from blocks of time dedicated to deep work. But we can go further than this and say that even among blocks of deep work, it’s usually more efficient to focus on just one thing at a time, for as long as possible; so if you’ve got a to-do list that consists of a database migration, refining a new SVM model, and setting up A/B tests for marketing copy, for the love of god, do them one at a time if at all possible.

Managing teams is a little different. The focus becomes learning the distinct comparative advantages of your team and ensuring that your team members are free to dedicate themselves to work that aligns closely with their comparative advantages. Most teams get this right to first order—have the database experts work on the database, and the UI designers focus on the frontend—but in the case of startups, first order correctness doesn’t help much when you’re working with fewer engineers than you’d like, most of them generalists, and you certainly don’t have entire departments dedicated to specific functions.

In those cases you’ve got two options, and you should pursue them both simultaneously.

On one hand, get to know your team members. Watch their performance on various tasks closely and build up a good mental model—as accurate as you can!—of their comparative advantages. Figure out who’s the fastest for exploratory work, and who’s the most thorough for vital functions. Determine who can hold their own in open presentations, and who prefers communication through written documentation. And do your damndest to make sure that every one of your team members gets to focus on their comparative advantages.

On the other hand, be open with your team members and talk to them, often, about where they see their current comparative advantages, and what new ones they’d like to develop—they might surprise you! And of course, it’s vital to not only encourage your team members to develop new or deeper comparative advantages, you must make doing so part of their job. This means that continuing education can not be optional; time must be formally and officially set aside for it.

Continuing education can be tricky to mandate. Too often, it turns into useless corporate classes and gets viewed as a chore. Instead, it’s best to leave the direction and method of the learning completely up to the individual doing the learning. If they want to audit a class online, then they get to audit a class! If they want to study a book, then they get to study a book! And if their chosen topics don’t align directly with the work they’re currently doing, so much the better: it just means that when new, unexpected challenges come down the line tomorrow, you’re more likely to have someone able to tackle them.

To judge their efforts, simply suggest some sort of presentation on the learnings at the end. This can be as elaborate as showing off a small project to other teammates, or as simple as a verbal discussion with you (their manager) or another colleague. The important part is to ensure interaction with an outside voice that can ask questions, for that is when some of the most valuable learning takes place. Study groups, in particular, are excellent at this.

A note on hiring…

The goal in hiring is to bring in new team members who have sets of comparative advantages that serve as good complements to those possessed by your current team members. The problem is that while you may know what tasks need doing today—and so you can interview for them—you will not generally know what tasks need doing tomorrow.

The solution is to try to pick candidates who have significant additional comparative advantages outside of the set that you directly need for your current tasks. Koosh ball people instead of T-shaped people.

The way I do this is to ask candidates in interviews to “teach me something technical”. I tell them to pick a topic they’re very comfortable with, and then as they teach me about it, I will ask questions about related ideas or corner cases that hopefully they haven’t encountered directly before. I even tell them exactly what I’m looking for: not particularly correct answers (because I’m not likely to be a good judge of correctness on a topic I’m probably unfamiliar with), but instead that I’m looking to understand their thought process and approach when I throw them a curveball.

Once they pick a topic and dive in, I let them talk for maybe fifteen minutes or so, asking only clarifying questions at first, so that I understand what they’re telling me. Then I take a moment and come up with my curveball. This is almost always just asking them to reconsider the topic with some change to one of their fundamental assumptions: either a difference in constraint, a difference in the scale of some number, or similar.

For example, here are some descriptions of actual interviews:

Candidate A told me about their time working on nuclear power plants, and described in detail the challenges associated with the cooling systems. I asked them “Imagine that word of your expertise gets around, and tomorrow NASA shows up at your door, wanting a nuclear reactor for a new space station. How does your cooling system change without gravity?”.

Candidate B told me about a mesh sensor network that they put together for their garden. I asked them “How would you tackle this if your garden was huge and you still wanted to instrument every plant? What if it wasn’t just a garden, but a full scale farm?”

The point is not what solution they give to the curveball question, but how carefully they think through it. A bad answer will be simple: “Oh, I would just change X and that would be it”. A great answer will show them progressing down, discovering and resolving issues as they consider the ripple effects of your modified assumption: “Well, the biggest effect from that change would be X, which we could try solving by adjusting Y, but then that brings up the question of Z…”.

There is no topic that can survive a change in scale of multiple orders of magnitude, or the switching of a basic assumption, without triggering a fractal cascade of effects. The only metric is how far down that chain they can go.

The reason I do this is to learn how deeply the candidate digs into topics that they’re faced with. If the answer is “very deeply indeed”, then you can be pretty confident that they probably bring this behavior to lots of topics, and so they’re almost certain to have plenty of deep comparative advantages to bring to the table outside their direct job description.

In a startup, you need these people. They’re the only way to get ahead of the curve and give your team the best chance possible of having suitable comparative advantages already in place when unexpected challenges show up¹.

Leadership

The only thing that matters in a company’s culture is deep, mutual trust between all of the team members at the company, and as a leader, you are responsible for cultivating that trust.

This is important, so I’ll say it again.

No one actually cares if you have ping pong tables, dogs in the office, weekly happy hours and board game nights, or even an on-site masseuse. Those are all bullshit and meaningless towards the success of your startup if you don’t have deep, mutual trust. Similarly, if you do have deep, mutual trust, you can be working out of an unfurnished basement for all anyone cares, and still have full engagement from your team and produce amazing results.

The metric for this trust is the question “Would they take a bullet for me?”. This doesn’t need to be literal, but the gist is that anyone in your company, thinking of anyone else in the company that they interact with, should feel assured that the other person would be willing to take risk, flak, or trouble onto themselves for the sake of the questioner.

This means managers taking the heat when one of their team screws up. It means colleagues giving credit to otherwise invisible advice from others when it helps their project succeed. And it means the CEO rolling up their sleeves and turning a god damned wrench if it’s truly “all hands on deck”.

As a leader, you need to evaluate everything around you for whether it encourages or diminishes this kind of trust. Get it right and prosper with engaged colleagues, good retention, and full buy-in to the mission. Get it wrong and suffer.

For example, if you want to make sure that your remote employees feel like a full part of the team, don’t bring them into HQ for a bonding offsite, but send your HQ team on an offsite to the remote location.

Another: don’t expect your team leads to take flak for their reports if the VPs don’t take flak for their team leads in turn. And make this metric a part of their performance reviews!

Jeszcze raz². If the CEO is leaving the office while other team members are still crunching, then the CEO had better offer to get them coffee on the way out.

Delegating well

Too many people think that delegation is simply the act of telling someone else what to do. This is a recipe for disaster, missed deadlines, and a breakdown of trust.

Instead, delegation must be a two-way contract.

The delegator is charged with maintaining a comparative advantage in understanding the bigger picture, and specifying the desired goal.

The delegatee should have a comparative advantage (and freedom of choice!) in the tactical implementation required to achieve the specified goal, and crucially, must be responsible for specifying the needed resources (people/time/material)³.

Finally, the delegator and delegatee must reach an agreement as to the final nature of the goal, and resources available to reach it. This must be a real agreement. That means that the delegatee must be able to say no.

Without the ability to say “no”, it is far too easy to default into an endless stream of under-resourced death marches, chasing impossible goals and achieving little besides record burnout and employee churn.

Do note that the ability to say “no” doesn’t mean that you have to arrive at “yes” on the first pass. It’s completely reasonable to take a couple rounds of back-and-forth between the delegator and delegatee, carefully refining the precise definition of the goal, and likewise the precise resources needed to achieve it. This is an incredibly valuable process⁴.

Delegation and trust are tied together. Proper delegation enhances trust. Poor delegation destroys it.

One final note: much of this last section on leadership is distilled from the insight of two sources: Robert Coram’s biography of John Boyd and Chet Richards’s book on applying the strategy of John Boyd to business. Both are excellent reads.

Or, as we call it in startups, “on Tuesday”.
“Jeszcze raz” – roughly “once again” in Polish.
This is the only time I will come close to referring to “people” as “resources”. I hate it when this is done, and only do it here because I’m using “resources” to mean a broader set, including also time, equipment, tooling, money, etc. If you use the word “resources” when all you really mean is “people”, then then only thing you’re doing is obfuscating and anonymizing the fact that you’re referring to human beings, and there is surely a special place in hell for you… right next to people who talk at the theater.
Also known as “planning”.