If you hang out with AI nerds, you’ll hear them drop the phrase “bitter lesson” roughly once an hour. They’re referring to a 2019 post by noted computer scientist Rich Sutton. It suggests that:

The biggest lesson that can be read from 70 years of AI research is that general methods that leverage computation are ultimately the most effective, and by a large margin. … Seeking an improvement that makes a difference in the shorter term, researchers seek to leverage their human knowledge of the domain, but the only thing that matters in the long run is the leveraging of computation.

…

The bitter lesson is based on the historical observations that 1) AI researchers have often tried to build knowledge into their agents, 2) this always helps in the short term, and is personally satisfying to the researcher, but 3) in the long run it plateaus and even inhibits further progress, and 4) breakthrough progress eventually arrives by an opposing approach based on scaling computation by search and learning.

More concretely: if you’re building AI to play chess, you’d probably assume that it would make sense to architect the AI around how we humans think about chess. We know a lot about chess already, so we should use that knowledge! You might build a part of the model that focuses on opening moves, or one that encodes the relative values of different pieces as we understand them.

Doing so, Sutton suggests, will help in the short term and probably be satisfying to you, but (this is the bitter part) it will ultimately inhibit progress. In the long run, you’d be better off basically just throwing enormous amounts of data into the model training process and letting it figure it out itself, without your pesky human opinions (even if they’re correct). Says Sutton:

We have to learn the bitter lesson that building in how we think we think does not work in the long run.

Empirically, Sutton seems to be quite right with respect to training AI models. And I humbly propose that the bitter lesson applies just as well to life at large.

Think of the clay pot parable from Art & Fear:

[A] ceramics teacher announced on opening day that he was dividing the class into two groups. All those on the left side of the studio, he said, would be graded solely on the quantity of work they produced, all those on the right solely on its quality. His procedure was simple: on the final day of class he would bring in his bathroom scales and weigh the work of the “quantity” group: fifty pound of pots rated an “A”, forty pounds a “B”, and so on. Those being graded on “quality”, however, needed to produce only one pot — albeit a perfect one — to get an “A”.

Well, came grading time and a curious fact emerged: the works of highest quality were all produced by the group being graded for quantity. It seems that while the “quantity” group was busily churning out piles of work – and learning from their mistakes — the “quality” group had sat theorizing about perfection, and in the end had little more to show for their efforts than grandiose theories and a pile of dead clay.

The bitter lesson for pottery! Don’t think about how to make a good pot — make many pots!

So too for so much else in life. Talk to the girl. Better: talk to many girls! Go on a run. Go on many runs! Reach out to the candidate. Reach out to many! You could read up on how to talk to girls; research running shoes; figure out how to perfectly articulate your pitch. Or you could just go, do, and begin the compounding of lessons and successes.

The bitter lesson is exhibit #4,672 of the midwit meme. We’ve learned there are no silver bullets or magic pills in AI, and it’s the same for life. Sometimes “tricks” work for a bit, but they are short-term strategies and lose out to repeated, high effort inputs (see previously: On the application of high effort).

This, of course, doesn’t mean “don’t learn anything or try new approaches.” AI research itself is 1) filled with new techniques that advance the field, and 2) mostly trained on the outputs of human knowledge in the first place. But as Sutton says (emphasis mine):

Seeking an improvement that makes a difference in the shorter term, researchers seek to leverage their human knowledge of the domain, but the only thing that matters in the long run is the leveraging of computation. These two need not run counter to each other, but in practice they tend to. Time spent on one is time not spent on the other.

So, sure, studying doesn’t necessarily make you worse at doing. Could even make you better. But when faced with a choice between the two, take the bitter lesson and Just Do (More Of) It.

p.s. Rich Sutton if you’re reading this I’d be thrilled to sponsor an SSL certificate for your site so I don’t need to click through a security consent every time I reference your article :)

Looking for more to read?

Want to hear about new essays? Subscribe to my roughly-monthly newsletter recapping my recent writing and things I'm enjoying:

And I'd love to hear from you directly: andy@andybromberg.com