Saltar al contenido principal

Make Your A/B Testing More Effective and Efficient

Anjali Mehra, Senior Director of Product Analytics at DocuSign, discusses the role of A/B testing in data experimentation and how it can impact an organization. They cover DocuSign's analytics goals, A/B testing mech

feb 2023

Photo of Anjali Mehra
Anjali Mehra

Anjali Mehra is the Senior Director of Product Analytics, Data Science, Experimentation and Instrumentation at DocuSign. Throughout her career, she has also worked in marketing analytics and customer analytics, giving her significant insight into the impact of A/B testing and experimentation across all multiple parts of an organization.

Photo of Richie Cotton
Richie Cotton

Richie helps individuals and organizations get better at using data and AI. He's been a data scientist since before it was called data science, and has written two books and created many DataCamp courses on the subject. He is a host of the DataFramed podcast, and runs DataCamp's webinar program.

Key quotes

Organizations should keep very close eyes on A/B tests while they are running. There are a lot of things that can go wrong, and teams make mistakes. It’s possible that you did something that's now introduced an error in your product or feature, but also it's possible that you influence a user behavior in a way that's really, really hurting you. For example, let’s say your test period is four weeks. Within just the first few days, if you aren’t monitoring your test, you may not notice that your order volume is going down significantly, and it will stay that way for the full duration of the test unless you catch it early. So, it's critical to monitor any test while it's running.

Whenever someone has a new idea, there is always a lot of excitement and urgency to jump into testing it right away, so spending some time in the planning phase may seem a little bit like a slowing you down, but it really does pay back dividends in the end. It’s important to invest time into  building a strong hypothesis because not having one just makes everything take longer and risks more since you can’t fully trust the integrity of the results. Both rushing the process and trying to measure against too many KPIs can derail any testing process and hurt overall business value.

Key takeaways


You need a strong hypothesis and a strong primary KPI to make A/B testing an effective and efficient use of time and resources.


A/B Testing is a team effort that requires thorough planning, active monitoring, and constant tweaking to get good results.


A cost-benefit analysis is a great way to identify if what you want to test whether or not the results you are aiming for are going to drive more value than the resources it would require to test and implement anything new.


10 Top Data Analytics Conferences for 2024

Discover the most popular analytics conferences and events scheduled for 2024.

Javier Canales Luna

7 min

A Complete Guide to Alteryx Certifications

Advance your career with our Alteryx certification guide. Learn key strategies, tips, and resources to excel in data science.
Matt Crabtree's photo

Matt Crabtree

9 min

Scaling Enterprise Analytics with Libby Duane Adams, Chief Advocacy Officer and Co-Founder of Alteryx

RIchie and Libby explore the differences between analytics and business intelligence, generative AI and its implications in analytics, the role of data quality and governance, Alteryx’s AI platform, data skills as a workplace necessity, and more. 
Richie Cotton's photo

Richie Cotton

43 min

[Radar Recap] Building a Learning Culture for Analytics Functions, with Russell Johnson, Denisse Groenendaal-Lopez and Mark Stern

In the session, Russell Johnson, Chief Data Scientist at Marks & Spencer, Denisse Groenendaal-Lopez, Learning & Development Business Partner at Booking Group, and Mark Stern, VP of Business Intelligence & Analytics at BetMGM will address the importance of fostering a learning environment for driving success with analytics.
Adel Nehme's photo

Adel Nehme

41 min

[Radar Recap] From Data Governance to Data Discoverability: Building Trust in Data Within Your Organization with Esther Munyi, Amy Grace, Stefaan Verhulst and Malarvizhi Veerappan

Esther Munyi, Amy Grace, Stefaan Verhulst and Malarvizhi Veerappan focus on strategies for improving data quality, fostering a culture of trust around data, and balancing robust governance with the need for accessible, high-quality data.
Richie Cotton's photo

Richie Cotton

39 min

[Radar Recap] Scaling Data ROI: Driving Analytics Adoption Within Your Organization with Laura Gent Felker, Omar Khawaja and Tiffany Perkins-Munn

Laura, Omar and Tiffany explore best practices when it comes to scaling analytics adoption within the wider organization
Richie Cotton's photo

Richie Cotton

40 min

See MoreSee More