How to Think Empirically Part 5: Keep Track of What You Did and How it Went

“Combine the ingredients and cook in the black pot.”

My mom was recently making a stew recipe passed down from her grandmother. This detail mattered enough for my great grandmother to include it in the directions. But 100 years later, without the “black pot”, we were left unsure about the best replacement to use or how important it was to perfecting the stew.

This story highlights something that scientists care deeply about called replicability. 

At its core, replicability comes down to a simple equation: attention to detail plus documentation. Without a clear enough record of how to reproduce the stew, some of the knowledge my great-grandmother gained through hard work and experimentation was now lost.

Document what you did

Let’s return to our pasta sauce experiment. There are a few key things we need to keep track of as we go:

• Materials: which ingredients did you include, and in what amounts? What pots and pans are needed?
• Procedure: how did you prepare the ingredients, and in what order did you combine them? How long did you cook?
• Results: did you like this version better than the previous one?

Remember that the goal is replicability – you want to ensure that the next time you make this recipe, you can achieve the same outcome by faithfully recreating the steps. It’s helpful to think about the details someone else would need to recreate the sauce to make sure you’re capturing the right level of detail:

• instead of “cook in the black pot”, try “cook for 30 minutes over medium heat using a 6.5 qt cast iron dutch oven”
• instead of “add 1 teaspoon of oregano”, try “mix 1 teaspoon of dried oregano with basil and garlic and stir for 1 minute”

These details may seem minor. But small differences can easily sneak into your recipe and change your results when documentation is unclear. If you intended to test “removing red pepper flakes” but you accidentally switched from a white to yellow onion at the same time, you’ve accidentally changed two variables and will have trouble learning from your experiment.

Record what happened

You’ll also want to make note of whether your new version of the recipe worked or not. A common framework in business experimentation is the champion vs. challenger model. In this set up, you continue to use the best performing version of your recipe (in our case) as the baseline until some challenger comes along that can outperform it – that then becomes the new champion.

You need to keep track of which version is the champion, but there are many simple ways to do this for our pasta sauce example:

• tasting notes: description of the new sauce and comparison to the champion
• Thumbs up / thumbs down: was the challenger better or worse than the champion?
• Rating scale: gives more nuance to tracking the outcome vs. thumbs up/thumbs down

The exact system doesn’t matter. What matters is preserving the knowledge you gain from each attempt and setting yourself up for the next round of experimentation.

Detailed documentation is the most time consuming part of experimentation – just ask any scientist how much they enjoy preparing reports for publication! The most critical thing is to capture the essential details, and avoid burning yourself out on things that may not matter much (should it be store brand or name brand oregano?). 

And with that, you’ve completed a testing cycle and are ready to return to goal setting for you next attempt. By following a systematic process we’ve gained clear, reusable insights from all the effort we’ve put in, and are ready to continue on with more confidence in what we know.