Unlocking personalized design with diary studies

While researchers agree that there is no gold standard for how to design a diary study, there is consensus that these studies are highly flexible to the question at hand. 

Diary studies can be organized and adapted into a variety of different experiments. For example, you can randomize your participants into different treatment groups and you have yourself a controlled study. Or, you can pair survey-style check-ins with interviews or observations for a mixed-methods framework that balances breadth and depth. 

Advances in tracking and AI technologies present new opportunities for diary studies. 

• Get better measurements: Data from health tracking technologies can be used to gather measurements that supplement self-reported data. 
• Chat with your participants: AI chat models can be programmed into your survey (for example, using OpenAI’s Chat Completions API) to pose responsive follow-up questions to participants, creating a more dynamic study experience. 
• Create realistic recommendations: Likewise, AI text generation models can be used to transform preliminary, basic recommendations (“exercise for 15 minutes”) into recommendations that are closer to what a future product would generate (“We see you’ve got a lot on today, but a little goes a long way. How about squeezing in a 15-minute workout to lift your mood?”).

In many parts of life, what you say you’ll do matters a lot less than what you actually do. When it comes to comparing these intentions against real actions, diary studies are an ideal tool. Moreover, diary studies are adept at identifying opportunities to instill habits, offering a better understanding of the key moments where users can be nudged toward more consistent engagement. 

The persistence of the intention-action gap and the power of habits are both never more relevant than in the context of exercise. So, when faced with the task of testing personalized, evidence-based exercise recommendations with prospective users, we chose to use a diary study. Our study followed 30 active North American adults, with half of the group receiving personalized exercise recommendations, allowing us to measure the impact on their exercise behavior. 

For two weeks, we checked in with participants three times a day according to their usual exercise schedule: 

1) Early check-in: Understand what exercise they intended to do that day.

2) Pre-exercise check-in: Ask how they were feeling once the day got going and provide an exercise recommendation based on their response.

3) End-of-day check-in: Request a report on what exercise they did that day.

With this setup, we were able to answer questions like:

• What were the key personal or contextual factors that made users more receptive to our recommendations? Our recommendations were particularly effective at changing users’ minds when they didn’t intend to exercise at all. This was especially true if users were feeling self-conscious or sore.
• What recommendations were users most receptive to? Context is everything when it comes to answering this question. When users felt capable of taking on a new challenge, a recommendation to do a technically difficult exercise got more users sweating. Meanwhile, when users felt stressed, a recommendation to do a high-intensity workout was more likely to get them moving.
• What contextual factors called for a more sensitive approach? People who had been inactive for a long time on a given day proved particularly difficult to budge. This group also tended to be stressed, self-conscious, and sore, suggesting that inactivity was the symptom of something bigger going on. Since recommendations to get active were not effective, our results suggest that this is the perfect opportunity for a self-compassionate approach that builds rapport with the user, such as reminding users that it’s okay to take a rest day or suggesting that they postpone their planned workout to another time.

Based on insights like these, the diary study provided recommendations targeted at making a product more responsive to changing moods, pressures, and sensitivities. Perhaps even more importantly, the results highlighted the contextual factors that a personalization system must know about (and therefore measure) to respond helpfully and appropriately to its target demographic. 

While insightful and worthwhile, diary studies do require a certain amount of respect. Not only are you gathering potentially personal information, but you are also reliant on your participants remaining engaged. In our experience, the best insights are reached when diary studies are rewarding for both participants and researchers alike. With that in mind, here are a few must-haves for a well-run diary study:

Pay well

This is especially important if you are asking users to report on a sensitive topic. Previous research with our partner, Tremendous, shows that sensitive topics need to be compensated at a higher rate to engage participants, in particular for longer studies. Keep people engaged by spacing payments throughout the study, and show good faith by sending the first payment early on. 

Connect. 

Diary studies require users to stay engaged over longer periods of time and often call for closer support for participants. To build a connection with participants, try unveiling the human behind the Google Form by recording an onboarding video or offering a quick one-on-one catch-up with participants to answer any questions.

Remind people! 

Let’s face it – filling out a 5-minute check-in isn’t at the top of everyone’s mind. Support your participants by reminding them to fulfill their diary study tasks. Even better, allow participants to personalize their reminder schedule and method. Keeping tasks short will encourage people to respond to reminders straight away, rather than them putting them off, or even worse, completely forgetting them. 

Make the study relevant. 

Don’t underestimate the importance of reciprocity. Demonstrate that you are committed to making the experience positive and useful for participants in whatever way you can. As a starting point, this study suggests giving the gift of knowledge by providing personalized feedback reports to participants. 

Following these recommendations, our study on exercise recommendations achieved a 100% retention rate and glowing participant feedback. Beyond tolerating our daily intrusions, participants actually valued and enjoyed the experience; it provided accountability, motivation, and an opportunity to reflect on how they were truly feeling about their barriers to exercise. 

While undoubtedly instrumental in the high engagement we experienced during that study, this positive participant feedback does raise an important point about diary studies. Recording how you feel and what you do is an inherently introspective exercise, and one that may not reflect “business as usual” for most people. Depending on the diary study, taking part is itself an intervention with potential impacts for participants. 

This is important to keep in mind both when designing the study and analyzing the results. Introducing a control group that undertakes a similar diary entry exercise but is not exposed to personalized input is one way of helping to tease apart these effects.

While personalization is often based on masses of big data, diary studies build a foundation for products that don’t just know users based on algorithms, but, vitally, understand them on a personal level. They shed light on the current and potential role of a product or service in users’ lives. Through diary studies, we learn how products can be more useful and better accompany users in overcoming day-to-day challenges and achieving their goals.

The Decision Lab is a behavioral consultancy that uses science to advance social good. We have created bespoke diary study designs using robust experimental methods to probe deeper into the experiences of real users and support the design of more useful and impactful products. If you are eager to explore how we can create tailored solutions that drive positive change in the world, contact us.