Results
Outcomes are the focus of any behavior change work. I have been fortunate to work with teams across a wide variety of areas on a diverse mix of behavioral problems, and we have generally been successful on the outcomes scoreboard, though I certainly cannot take all the credit. Some examples are below.
All Evidence
Civic Outcomes
“The polling location messages increased turnout by 0.2 percentage points (p < 0.05) relative to the no contact control group, yielding 1,473 net votes at a cost per net vote of $128 (or about 8 votes per $1,000 spent).”
“The text program also boosted voter turnout by a statistically significant 0.3 pp (p <0.01), yielding votes at cost of $102 per net votes.”
“The SMS program increased voter registration by a statistically significant 0.3 pp (p < 0.001), generating 1,601 registrants at a cost of $82 per net registrant.”
“Not only were we able to register 26,000 hard-to-reach-voters, but we were able to do so at a cost of $10 per registration form, which is cheaper than site-based registration. What’s more, our newly registered voters actually turned out to vote. The cost per net-new-vote was $77, which means for every $1,000 we spent, 13 people voted who would have otherwise stayed home.”
“Irrational Labs found that the new sharing prompt led users to share misleading videos 24 percent less often.”
“The intervention reduced shares of flagged content by 24%. The intervention was more likely to work on older users, decreasing shares by 34% for users 35+, compared to a 21% reduction for users 18-24 and 25% reduction for users 25-34. In fact, when we looked at the data by looking at ages 18+, the intervention cut in half the number of shares relative to the control, with the video prompt being the main driver of this additional effect.”
Finance Outcomes
“When partnering with Chime to have clients automatically save 10% of their direct deposits, those that used the feature saved an average of $200 per month.”
“We also worked with Chime to launch their 10% auto-savings tool. Through this tool, users can opt-in to automatically saving 10% of any direct deposits into their checking accounts. This tool led to more than $8 million in savings in the first few months.”
“In terms of engagement with the budget, about 10% of the users in the experimental condition saw the budget 8 times or more, while a majority of the users (84%) saw the budget 5 or fewer times over the experimental period. Both budgeting variants statistically significantly increased engagement over the control from once every 4 weeks to once every 3 weeks (p < 0.001).”
“The interventions led to an 18% increase in recurring transfer setups and a significant increase in direct deposit interest.”
“Pre-commitment increased tax time savings from 17% to 27%, a 58% increase.”
“Our experiment led to $1,000,000 in savings. Nearly 85% of the savings were still in the savings account 3 months later.”
“Of everyone who responded to the text, on average people wanted to save 10% of their tax-refund in the control condition, and 15% in the experiment condition. When we looked at those that responded positively to the text and wanted to save, our experiment doubled savings rate to 22% from 12% in the control condition.”
“DoubleNet Pay enables workers to contribute directly to a goal-based savings account and pay bills automatically via payroll deduction. This addresses two challenges that arise from volatile incomes: automatically depositing part of a paycheck into savings increases workers’ ability to save successfully, and paying bills through payroll also leaves the worker with more certainty about how much money they have and how much is safe to spend.”
“Approximately 37% of web sessions during which individuals used the sliders resulted in savings rate changes. Notably, 75% of the changes were increases in savings rates, with an average increase percentage of 18.8%.”
“This results-driven philosophy begins with a personalized view that translates savings into estimated monthly income in retirement and offers next steps to help boost income in retirement. Of the participants who have used our experience, 39% changed their deferral rate and the average deferral rate increase was 25%.”
“This “round up” framing led to a jump from 11% to 15% of users increasing their payments; that is a 35% increase in people paying off their mortgage faster.”
“We included ~80,000 Steady users in the experiment, with ~40,000 in the control, and ~20,000 in each of the Goals variants. The bank account link rate was ~36%. We ran the experiment over 4 weeks, so over the course of that time frame ~14,400 users increased their incomes $28-$80 (2-6%) over that month, for an overall increase in income of $400,000-$1.15M due to the Goals feature.”
“We found that both experimental conditions substantially outperformed the control in CTA acceptance (forced choice +71% and missing info +80%). The effect size of the forced choice (60%) and missing information (63%) conditions were similar but still statistically significantly different from the control.”
“In the K2C analysis, after controlling for school- and community-level factors, there is no statistical difference in total contributions in years one through three between high-poverty and low-poverty schools. […] It appears that giving families equitable access to savings vehicles can narrow the college savings contributions gap between low-income and high-income families, at least over several years.”
“San Francisco’s Kindergarten to College has achieved close to 100% enrollment using its automatic enrollment approach.”
“Findings from K2C indicate that 15% of students in high-poverty schools made at least one contribution, compared to 20% of students in low-poverty schools.”
“[…] when focusing only on students with at least one contribution, the average contribution value and the average total value increased as the CSA ages (from $247 in the first year to $709 in the fourth year and from $420 in the first year to $907 in the fourth year, respectively; Figure 4).”
Health Outcomes
“8 out of 10 patients reported improved mental health symptoms after treatment, and 84% reported they’d recommend the service to others.”
“Genetic testing rates for sites deploying a (CancerIQ) POC-T workflow were more than double that of the other 3 workflows, averaging 35% across all the 9 POC-T sites (range 14-53%). ”
“The results of the six-month pilot showed a 20 percent decline in caregiver burden, such as a reduction in their physical and emotional stress; a 13 percent increase in caregiver confidence; a 17 percent increase in caregiver knowledge; an 18 percent decrease in hospitalizations; and a 51 percent drop in emergency department visits.”
“[…] clinicians reported the video component to be the most helpful part of the Cognoa evaluation. Overall, 88% found the summary report helpful, and 76% found that the report and videos alerted them to look for particular behaviors.”
“Comparison of device results to specialist diagnosis found the PPV: 80.8% [95%CI, 70.3%–88.8%], NPV: 98.3 [90.6%–100%], sensitivity: 98.4% [91.6%–100%], specificity: 78.9% [67.6%–87.7%] for subjects with determinate device results. There was no evidence that device performance significantly varied when PCP used the device remotely compared to in-person.”
“Eureka has now enrolled more than 500,000 participants across nearly 70 studies focusing on health conditions ranging from diabetes to inflammatory bowel disease, cancer and Parkinson’s disease. Most studies are funded by the NIH or the Patient-Centered Outcomes Research Institute (PCORI).”
“We consistently observe that 0.5%–5% of invitations lead to study enrollment.”
“In the COVID-19 Citizen Science study, we requested intensive participant engagement with survey responses on a daily, weekly, and monthly basis and observed that 59% of participants completed at least one survey over a 4-week period in February 2021, nearly a year after launch of the study at the beginning of the pandemic.”
““We can leverage this group to conduct very rapid studies,” Olgin said. “As an example, we are doing a trial of a smoking cessation tool. We needed 300 participants. We contacted our then 100,000 participants and had our 300 slots filled in 20 minutes.””
“After the pilots ended, more than 90% of employees said they'd recommend the game and play it again, and more than 70% of the people who started playing stayed engaged through the entire 12-week period.”
“Ongoing engagement increased roughly 30-fold and sustained engagement increased roughly by two orders of magnitude.”
“Most companies saw one-third of employees lose an average of a half a pound per week, with one-sixth losing more than 1 pound per week.”
“Fewer Visits in Virtual Group: Patients in the virtual visit group had 0.8 fewer primary care visits than those in the usual care group (95% CI, −1.2 to −0.3).”
“Subgroup Analysis for Unsatisfactory SBP Control: For patients with unsatisfactory SBP control, there were 1.4 fewer primary care visits in the virtual visit group compared to the usual care group (95% CI, 0.3 to 2.5).”
“Physicians were on average extremely satisfied with the clinical decision-making capabilities of the virtual visit tool. Physician satisfaction with receiving sufficient data to make a diagnosis, order tests, change medications, and make referrals ranged from 4.32 to 4.53/5 (SE= 0.06–0.09). Satisfaction with the diagnostic capability was 4.60/5 (SE= 0.07). Overall satisfaction was 4.47/5 (SE= 0.08).”
“Patient Engagement Performance: • 2.9X More Adherent • 40% Click Through Rate • 50% Two Hour Response Rate • 99% Retention”
“79 percent of Avella’s HIV patients using mscripts mobile app achieved ≥ 90 percent adherence rate.”
“[mscripts] adds 3 or more fills per year, [with a] 99% retention rate on platform, and 50% of pts refill within 2 hours of text.”
“HIV patients using a mobile app which provides refill reminders, dosage reminders and other prescription management functionality are 2.9 times more likely to be adherent.”
“Avella’s HIV patients that used the mscripts mobile app were almost three times less likely to discontinue their mscripts medication refills.”
“Those using the PrEPmate intervention were 15% more likely to attend the monthly visits (86% vs 71%). They were also 15% more likely to have appropriate TVF-DP levels (72% vs 57%).”
“88% of participants in the PrEPmate arm reported the intervention was very/somewhat helpful; 83% wanted to continue using PrEPmate after the study, and 92% would recommend it to others at week 36.”
“A significantly larger proportion of visits were completed by PrEPmate participants compared to Standard of Care participants (86% vs. 71% among all visits; odds ratio (OR) = 2.62; 95% CI: 1.24 - 5.54; p = 0.01).”
“Attitudes toward PrEPmate were favorable, with 88% of those using the intervention reporting that it was very or somewhat helpful, 83% wanting to continue using the intervention after the study, and 92% saying they would recommend it to others.”
“There was a significant reduction in COPD-related healthcare utilization compared with the year prior to enrollment, with values changing from 3.4 (3.2) to 2.2 (2.3), p¼ 0.01.”
“In the subgroup that received AV reminders, all-cause healthcare utilization significantly decreased from 5.4 (4.3) to 3.1 (2.6), p¼ 0.005.”
“Patient retention was 90.5% (172/190) at 3 months, 81.0% (154/190) at 6 months, and 63.1% (120/190) at 12 months.”
“Using a mobile-app facilitated remote HTN management protocol, 84% of patients met their JNC8 BP goal, versus an estimated 53% nationally. Compared to prior reports that cite an average of 3.2 months to reach BP goal in real-world practice-based settings, patients in our study achieved JNC8 BP goal within 17 days from enrollment. A mobile app-facilitated remote HTN management platform, coupled with team-based care, can be used to increase achievement of BP control and decrease the time required to achieve BP control.”
“Seniorlink demonstrated its largest improvements over the MCBS group among medically complex consumers over age 65. This population experienced 40% fewer hospitalizations, 32% fewer ER visits, and 75% fewer falls than those on Medicare (all P<0.05).”
“Seniorlink reported nearly one-third fewer hospitalizations and 23 percent fewer trips to the emergency department for the population studied.”
“In the cases where the individuals receiving care had the highest levels of complexity and were 65 years and older, the reductions in hospitalizations and emergency room visits were even greater: reductions of 32% for ER visits and 40% for hospitalizations.”
“Most customers (73-79%) felt that they completely understood the terms premium, deductible and co-payments.”
“For the types of information that were most sought on the website, roughly 70 to 77% of information seekers obtained the information they were looking for.”
“We enrolled 92 participants (CF: n=52 and IBD: n=40), and 73% (67/92) completed the study. Average PRO completion was 61%, and average weekly platform use was 80%. Participants reported improvement in self-efficacy from baseline to 6 months (7.90 to 8.44; P=.006). At 6 months, most participants reported that the platform was useful (36/40, 90%) and had a positive impact on their care, including improved visit quality (33/40, 83%), visit collaboration (35/40, 88%), and visit preparation (31/40, 78%).”
“The 30-day avoidable rehospitalization rate was reduced by 88% for June 2019.”
“96% of patients expressed satisfaction with Wanda Health, and 95% would recommend it to family and friends.”
“Care Plan Adherence increased from 83% at the beginning of the study in December 2018 to 91% by June 2019, indicating a relative improvement of 10%.”
Other Outcomes
“It's the fastest growing company I've ever seen—and I'm including Uber in that. It has been growing at a rate that I'll just say is faster than Uber and Robin Hood.”
“In four years, [they] bootstrapped Athena to about 2,000 employees and $50 million ARR.”
“Irrational Labs found that the new sharing prompt led users to share misleading videos 24 percent less often.”
“The intervention reduced shares of flagged content by 24%. The intervention was more likely to work on older users, decreasing shares by 34% for users 35+, compared to a 21% reduction for users 18-24 and 25% reduction for users 25-34. In fact, when we looked at the data by looking at ages 18+, the intervention cut in half the number of shares relative to the control, with the video prompt being the main driver of this additional effect.”