Let’s do more experiments!

A quick poll: do you pull bandaids off slowly or rip them off quickly? Denise tends to go slowly on the bandaid front (although bear in mind that if there’s any blood, then it’s someone else’s job not hers!). And John says rip them off quickly! So then the question is: does it actually make any difference, and if so, which is less painful? Believe it or not, but a university study has been conducted on this very topic, and we will come back to that later in this post.

We think it is easy for us to keep the status quo, and keep doing the same things day in, and day out. It’s comfortable, it’s easy to plan, and we know what to expect. But what might happen if we were to change something? In this post we are going to explore the scientific method and see how we can achieve greater results through greater experimentation. 

Many of us probably studied science at high school or university, and have vague recollections of the scientific method, but can you remember what the actual steps are? If not, let’s go back about 500 years. The original components of the scientific method were developed by Sir Francis Bacon, way back in 1620.   

Here is a quick refresher…. the scientific method has six steps. It begins with a question, followed by research that leads to a hypothesis, which is then tested with an experiment, the results of which are analysed, and this leads to a conclusion. It sounds simple, yet this approach has led to astonishing advances in science for centuries. 

Some suggest we can track experimental design back to the book of Daniel, which was written around 165 BC. It describes an experiment to test the difference between consuming meat and wine versus pulses and water for ten days and then observing the differences. if you are curious about this, you can read an interesting description of it over on the ICR website

While that was a controlled experiment, it was not a randomised control trial, which is often abbreviated to RCT, and is considered to be the gold standard for effective research. This is a specific type of scientific experiment used to control factors that are not under direct experimental control. Clinical trials are a good example, where doctors study the effect of drugs, surgical techniques, and medical treatments. 

The participants who enroll in the RCTs are obviously not identical, and have various differences in known and unknown ways that can influence the outcome of the studies. But by randomly allocating the participants among the treatments being compared, an RCT enables statistical control over these influences. The randomisation balances out the participant characteristics (both observed and unobserved) between the various groups, allowing any differences in outcome to be attributed to the particular intervention. Provided the trial is well designed, conducted properly, and involves enough participants, an RCT can achieve sufficient control over these confounding factors to be able to deliver a useful comparison of the treatments studied.

In his book Randomistas, Andrew Leigh describes how radical researchers can change the world. He even provides examples of how he lives his life guided by RCTs. He stopped taking daily multivitamins after an RCT study found them to be ineffective, and he no longer attends annual medical check-ups after seeing evidence that suggests they do not reduce the chance of falling ill. Perhaps not surprisingly, the subtitle of the book was chosen based on the result of an RCT. 

He begins with the gripping tale of how ship surgeon James Lind in 1747, discovered that citrus fruit could prevent thousands of seafarers from dying of scurvy. John confesses that he thought people with scurvy just went to sleep one night and did not wake up the next morning. However it turns out to be a rather gruesome death, as it affects the body’s connective tissues. Initially, those with scurvy feel tired and become uncoordinated. They bruise easily and their legs begin to swell. Then their gums become inflamed and rot, oozing black putrid blood. Their thighs and legs also go black and gangrenous. Apparently in the final stages their gums swell up so much that they can no longer eat and they soon die from internal bleeding. (At this point Denise has stopped contributing to this blog post and is trying to think about chocolate…) Scurvy was so bad that a British sailor in the Seven Years’ War had less than 1 percent chance of being killed in action, but a horrifying 72 percent chance of dying from scurvy. 

But back to James Lind, the ship’s surgeon. Out of frustration of having to treat so many people for scurvy, he conducted a randomised trial to test six treatments. Fortunately one of those included citrus fruit, which made the greatest difference. But unfortunately it then took Lind six years to write his 456 page book A treatise of the scurvy, which was generally overlooked. It was not until 1795, almost 50 years later, that lemon and lime juice began to be routinely dispensed to sailors. Leigh graphically describes all this (and many other stories) in his book. Interestingly, Andrew Leigh is now the federal member for Fenner, in the ACT, Australia. Before being elected in 2010, he was a professor of economics at the Australian National University. 

But what does all this have to do with us as enablers of change? Plentyas we always need to be looking for the best evidence-based approaches for our work. As good scientists we need to be questioning the content and process of our day-to-day work. We should be questioning why we do things and how we do them. Let’s not be like Einstein’s definition of insanitydoing the same thing over and over again yet expecting different results. 

While running experiments in the physical world can often take weeks or months, doing them in the digital world can bring results surprisingly quickly. As an example, if you send electronic newsletters using a program like Mailchimp, and you are on their Premium plan, you can change a number of variables and see the effect. It will allow you to run what they call an A/B testing campaign to see how small changes to the content can impact the results. You can change things like the subject line, the sender’s name and the time the campaign is sent. Their system then tracks when the messages are opened and what links are clicked. You can choose what percentage of your mailing list to use for the test, and once the results are obvious, it can automatically use the best result to send the best message to the rest of your subscribers. 

On the Mailchimp website they give the example of using three slightly different headlines. John has used tests like that before and found his audience more quickly opened the messages if the subject line had a short, interesting description, rather than a bland “Newsletter number 32” style headline. We have noticed some people like to mailmerge the recipient’s name into the subject line, such as ‘Jo, here’s something you’ll find useful’, but we feel that sometimes this can be a bit cheesy. 

Towards the end of his book, Leigh provides the following 10 guiding principles for conducting RCTs. 

  1. Decide what you want to test
  2. Think creatively about how to create a random difference in the program
  3. Consider what the control group will do
  4. Choose which outcomes to measure
  5. Select the level at which to randomise
  6. Ensure your study is large enough
  7. Register your trial and get ethics approval
  8. Confirm that the key people understand and support randomisation
  9. Use a truly random procedure to split the sample, and finally
  10. If possible, conduct a small-scale pilot study.


Of course it is not always possible to run a full RCT, but that should not stop us from experimenting and trying new things. Our approach with the Enablers of change YouTube videos has been to start with what those working in startups would call a minimum viable product or MVP. That is the cheapest, easiest thing that works sufficiently well to demonstrate an idea. We have both read the book Ish by Lynne Cazaly and keep telling ourselves that “Done is better than perfect”! 

We keep doing little tests along the way to see what works best. When we first started out we were disappointed with the recording quality of 720 pixels or less that we were getting with our Zoom recordings, as back then they were promoting 1080p recordings. We kept using the lower definition ones, but in the background John reached out to Zoom and somehow convinced one of their engineers to help us. He spent three or four months working through various options, including buying different webcams and upgrading his computer, only to find that for whatever unknown reason, we could not record in high definition. That was about three years ago and since then with the explosion of users and increased usage of bandwidth, we sometimes only get only 360p. Anyway, suffice to say we keep testing different things to improve the user experience.

Those of you who listen to the podcast or watch the video episodes may have noticed that we moved the intro music for this episode. We know many people are time poor, and while the 10 seconds of intro music right at the beginning is not long for a podcast or YouTube video, we know people are becoming increasingly impatient. We could not think of an easy way to do this as an RCT, so let us know in the comments under the blog post whether you noticed and whether you liked it or not. 

And that is the other easy way to get feedback on tests you might run when you do not have the resources to run a full blown RCT. Do something differently and either wait to see if anyone notices, or just ask them! You could for example send a short eSurvey asking a couple of quick questions. We have done that and get surprisingly high response rates, as people appreciate that we’ve taken the time to seek their opinion.

But before we finish, let’s go back to the bandaid dilemma. Is it less painful to remove them quickly or slowly? As it happens, an RCT run with 65 volunteers from the James Cook University in Queensland showed that quick wins the day! The overall mean pain score for the fast bandaid removal was 0.92, while for the slow bandaid removal it was 1.58. This represents a highly significant difference and if you are curious to know more, the reference to the article is at the end of the blog post.

So that’s it for this post! If it sparked something for you, add a comment below and tell us about your experiences with using RCTs, including any tips and further ideas about them. We don’t want this to be just a one-way conversationjoin in by sharing your thoughts and ideas with us! 

And if you liked this post, take a moment to give it a rating using the handy widget at the end. Just click the number of stars to indicate your rating. That’s another little thing we are trialing!

Resources

Furyk, J.S., O’Kane, C.J., Aitken, P.J., Banks, C.J. and Kault, D.A. (2009), Fast versus slow bandaid removal: a randomised trial. Medical Journal of Australia, 191: 682-683. Available online.  

Leigh, A. (2018). Randomistas: How radical researchers changed our world. Australia: La Trobe University Press.

4.5 6 votes
Article rating
Subscribe
Notify of
guest
6 Comments
oldest
newest most voted
Inline feedbacks
View all comments
Chrissy Stokes
Chrissy Stokes
9 days ago

Obviously your subject line got me this time guys 🙂

Denise Bewsell
Denise Bewsell
23 hours ago
Reply to  Chrissy Stokes

Excellent!!

Helen Newman
Helen Newman
6 days ago

I noticed the missing music at the beginning, I started to hum the tune tune and then was surprised by a different beginning 🙂
Excellent work as always
Thank you

Denise Bewsell
Denise Bewsell
23 hours ago
Reply to  Helen Newman

Thanks Helen!