Directly, a customer service platform that leverages power-users to answer customer questions, wanted to further improve the speed and cost-savings of their services with AI Chatbot answers. Unfortunately, the initial chatbot pilot program was not received well by the power-user community or the end-customers. Directly hired me to understand what was going wrong with chatbots and how to make them helpful and approachable enough to launch to their entire client base.
We attacked the problem from multiple angles, surveying the power-user community, conducting in-depth interviews with specific power users and iteratively prototyping and testing new designs. After 3 rounds of usability testing with both power-users and end-customers, we came up with a new design and product strategy that made chatbots more acceptable for everyone.
We discovered that power-users saw chatbots as competition instead of a helpful tool. They were also confused about the difference between live responses from other power-users and repeatable responses from chatbots. We learned that we had to involve power-users more directly in the process of creating these chatbot responses and motivate them with a clear rewards system to keep them engaged and satisfied.
Based on iterative testing and research findings, we created a new chatbot system that allowed power-users to author their own chatbot responses and redesigned the rewards system to incentivize these new functions. Chatbot engagement among power-users improved dramatically, from 5% to over 90% in 2 months. Client adoption of chatbots also grew from 3% to over 25% due to the improvements we made.
I was the sole researcher and strategy lead to design, execute, and synthesize the chatbot research program. I worked closely with the design lead, PM, and director of product to turn research insights into new design ideas, many of which have been successfully launched in the final product.