With an AI assistant, giving an answer is easy. Giving a good answer is much harder. An answer can be technically accurate. It can provide the right information and be on topic. But it can also be in the wrong context, overwhelm someone with unnecessary details, or leave them without a clear next step.

Key Challenges

At Mailchimp, our AI assistant combined curated conversation flows with generative AI. Our AI was built with a retrieval-augmented generation (RAG) system. This let customers ask questions about Mailchimp and get answers informed by our extensive knowledge base.

But generative AI introduced a new challenge. Unlike a traditional help article, the experience wasn't entirely predictable.

We needed a way to identify where responses fell short and continuously improve the quality of each turn.

The question became: How do you design an AI experience that gets better with every conversation?

Role and Scope

During this time, I led the content strategy and experience quality for Mailchimp Assistant, working with conversational designers, engineers, product teams, and customer support specialists.

My focus became creating a feedback loop that connected what customers experienced with how we evaluated and improved the assistant.

Defining what makes an answer good

The main thing I needed to establish was a consistent way to evaluate something as “good”.

Our evaluation process examined four dimensions of quality:

Accuracy. Was the information correct, or did the assistant provide misleading details, outdated instructions, or fabricated information?

Relevance. Did the response address what the customer actually wanted, accounting for their context and specific circumstances?

Coherence. Was the conversation clear and efficient, or did it introduce unnecessary complexity, repeat questions, or lose track of information?

Completeness. Did the assistant provide enough information to move the customer forward, or did it leave them at a dead end?

These dimensions gave us a shared language for describing failures that might otherwise be dismissed as simply a bad response.

More importantly, they helped us distinguish between different kinds of problems, each of which required a different solution.

Closing the loop

Evaluation is only useful if it leads to action.

I connected the insights from our AI evaluations to the team's existing content design and launch processes. This created a continuous cycle of analysis, then prioritization, then improvement.

We had three primary ways to respond when the assistant wasn't performing as expected.

Improve the source. Sometimes the problem wasn't the AI's behavior but the information it had access to. Updating a help article could improve the knowledge retrieved by the assistant and, in turn, the quality of its responses.

Improve the instructions. Other times, the information was available, but the assistant wasn't communicating it effectively. Prompt adjustments allowed us to address how the AI interpreted questions and formulated answers.

Design the experience ourselves. Some customer needs were better served by curated conversation flows. When generative responses consistently fell short, our conversational designers could develop a more intentional experience with clearer pathways to resolution.

Those decisions fed into our team's backlog, where work was prioritized, designed, reviewed, and published.

Once the improvements were live, we continued monitoring conversation performance, customer engagement, handoff rates, and resolution rates. What we learned became the starting point for the next iteration.

The process became a cycle which was predictable and easy to repeat.

Designing the system behind the experience

The assistant depended on the quality of what we fed it. In this case, it was our knowledge base.

Our knowledge base depended on understanding what customers needed. And our ability to improve either depended on recognizing when the experience wasn't working.

Creating the feedback loop meant connecting these previously siloed activities into a more deliberate system for managing conversational quality.

It also changed how we thought about content design.

Rather than treating the published response as the final deliverable, we treated it as something we could continue to observe, evaluate, and refine.

A better answer is a moving target

In the end, Designers need to consider the full system. How it understands a request, where its information comes from, when it should follow a prescribed path, and how we determine whether the experience was successful.

For me, the project reinforced a simple idea - influenced by The Bear:

After the kitchen opens, we keep improving until the kitchen closes.