What did you say to me? Instrumenting User Insights for your AI Copilot

AI copilots are the new app UX. 

Whether you’re building a chat bot or automated voice systems, your entire application is now only a prompt away from your users.

But how are your users interacting with these new model-powered experiences? And how can you take the ephemeral context from a chat window and use it to personalize the customer experience going forward?

For developers shipping LLM-powered copilots, there’s a new challenge to understand customers’ interactions with these automated systems. 

In this post, we’re going to show you how to deploy an AI Copilot experience with advanced instrumentation that allows you to analyze your Copilot performance and persist user context to drive the customer experience across sales, support, and marketing channels.

Instrumenting your AI Copilot with Segment

In this example, we’ll use the Vercel AI SDK and Chatbot template as a starting point. The Vercel AI SDK offers a twist on the traditional Copilot interface: rather than the model responding exclusively with text, the model can respond with text OR interactive components from your application to provide a richer user experience. An introduction to React Server Components (RSC) can be found in the Next.js docs.

We instrument tracking with the Twilio Segment node.js SDK to collect user and copilot behavior. Twilio Segment has integration capabilities for a wide range of other languages, so your analytics instrumentation can span different languages and implementations. We’re utilizing OpenAI’s GPT-3.5 for this showcase, but you can bring your own (customized) LLMs including Anthropic, Hugging Face, and more.

Note: While we’re skipping sign-ups and logins for brevity, it’s best practice to identify your copilot users with first-party data as soon as possible. 

Twilio Segment Setup

If you don’t have a Twilio Segment account yet, you can start by creating a new free account here. Once you’ve created your account, create a node.js source in Segment and note the write key, you will need this later to set up your tracking in the app. 

Step 1: Copilot Template Setup

To get started with your copilot setup, follow these steps:

• Fork the starter template repository

• Create a Vercel account

• Set up Vercel postgres

• Set up Vercel KV

• Populate your .env file with your keys to Segment, Vercel and OpenAI

The following commands get your app ready and deployed on localhost:

pnpm / npm install
pnpm / npm seed
pnpm / npm dev

A number of components come pre-configured with Segment tracking, with the library initiated in the analyticsInstance.ts singleton file. This makes it easy to integrate Segment on any component or route you want to use it on by simply importing.

import analytics from '@/app/analyticsInstance'

Step 2: Define your tracking plan

A clean copilot instrumentation starts by identifying your copilot’s event lifecycle. The Segment AI Copilot spec gives you an overview of relevant data points to track to understand your users’ interactions with your application. This spec is extensible to your specific use cases, a number of them will be covered in further detail in the implementation guide in this post. 

You can find the full Copilot spec here. 

Step 3: Identify your user

You can track your users in both anonymous and known contexts. While we forgo this in the template, a quick way to do so is to identify unknown users using Segment’s anonymousId property in an identify call. Generate a unique ID through a method of your choice (e.g. uuid). You can then identify your users with the following Segment identify call. You can also enrich your user profile with properties if you’d like.

analytics.identify({
  anonymousId: YOUR GENERATED ID HERE,
  traits:{
    key: value
    ...
  }
})

Once your user has given you additional information (e.g. email, user id etc), create another identify call with a userId. This will likely happen on sign-up, and log-in will enable you to track behavior across sessions and devices.

analytics.identify({
  userId: USER ID HERE,
  traits:{
    key: value
    ...
  }
})

Ensure to pass either anonymousId or userId in all following track/page calls so your copilot interactions get attributed to the correct user.

Step 4: Tracking conversation metadata

Now that your users are identified, you can start instrumenting your copilot interactions. The first key event to track are conversation starts, allowing you to track stats like conversation duration and per conversation message count. The chat.tsx file in the starter template you have forked earlier generates a unique conversation ID for each new conversation, which is attached to all following tracked interactions that belong to the same conversation. The attached Vercel project stores conversation IDs in local storage as newChatId so you can easily grab them across your application.

analytics.track({
  userId: "123",
  event: "Conversation Started",
  properties: {
    conversationId: id
  }
});

There are a number of extra events and properties you might want to track here, such as Conversation Ended events or specific models used if you have a multi-model setup. The Copilot AI spec has a number of extra events that might be worth exploring.

Step 5: Tracking standard prompts and responses

Tracking user prompts

The next important step is to understand how your customers interact with your copilot. To do so, it’s crucial to know what prompts they are using - Segment tracks this as a simple track event with Segment’s library, and passes in the message content as well as message and conversation IDs for full observability. The starter template collects your user’s prompt directly from the client in prompt-form.tsx, but you might wish to collect these from the server side as well using the same mechanism. Please ensure to pass in the relevant userId or anonymousId. The starter template uses hard-coded values for all of them.

analytics.track({
  userId: "123",
  event: "Message Sent",
  properties:{
    message_body:value,
    conversationId: window.localStorage.getItem('newChatId')
  }
})

Tracking copilot responses

Of course, you’re not just interested in your users’ prompts, but also the responses they receive from your models. Your app comes with tracking for your copilot’s response in the actions.tsx file of the starter template. The bot response is picked up from the server and sent into Segment, while simultaneously rendering to your user in their interface. You can of course enter additional properties to this event if you choose.

analytics.track({
  userId: '123',
  event: 'Message Received',
  properties: {
    message_body,
    conversationId: aiState.get().chatId,
  }
})

Step 6: Tracking custom components and interactions

As copilots evolve from being text-only, Segment helps you track your customers' engagement with custom components that you’ve built or dynamically generated. Robust and data-agnostic tracking allows you to collect data on these interactions just like you would any other data point. The folks over at Vercel have done a great job implementing a number of custom components in the starter pack that are instrumented through Segment. The template ships with three pre-instrumented custom components, one for displaying multiple stocks, one displaying a stock and its chart price over the day, and one for purchasing stock directly from your copilot.

The first important event to track is how often and which of your non-standard (i.e. text/voice) components are called. This helps you understand which of your customizations are most popular across your user base and helps you focus on iterating them. The starter pack collects (in actions.tsx) the type of component loaded as well as additional information (stock symbols for a daily price chart, an array of symbols for multi-symbol display and purchase levels for purchase components) and sends this data into Segment.

// send custom component load to Segment
analytics.track({
  userId: "123",
  event: "Component Loaded",
  properties: {
    type: 'Stock Price Chart',
    stock_symbol: symbol,
    conversationId: aiState.get().chatId
  }
});

Twilio Segment can not only help you understand the custom components of your copilot, but more importantly, it also enables you to translate them into business relevant events, such as tracking stock purchases directly from custom components. Your app takes the user interaction on the custom purchase component and automatically translates it into a Stock Purchased event in Segment. 

// send purchase event to Segment
analytics.track({
  userId: "123",
  event: "Stock Purchased",
  properties: {
    stock_symbol: symbol,
    amount: amount,
    total: amount*price
    conversationId: aiState.get().chatId
  }
});

Step 7: 360 Degree Copilot Observability

Twilio Segment’s event and data collection abilities don’t just stop there, but enable full business-centric observability for your copilots. This can also include understanding media your customers generate in your copilots, custom actions that are invoked, and data your copilots get back from your data warehouse. The newly launched AI Copilot spec gives you a baseline of mission critical events you can track with the help of Twilio Segment.

Turning Data into Action

Once collected, Twilio Segment gives you a breadth of tools to help further investigate your copilot’s user interaction by further enriching this data. You can create individual enrichments in computed traits, such as aggregating the number of times individual users have conversations with your copilot, or understanding which components they use most frequently, e.g. which stocks they interact with. 

By sending this data to one of Twilio Segment’s downstream partners, you can then activate users based on their interaction or send them to an analytics tool to gain further insight. 

One example of this can be pushing your copilot’s user interactions into an analytics tool like Mixpanel to further analyze stats like message frequency per conversation or how many messages your users exchange with your system until they complete a specific action (e.g. a stock purchase.

Your AI Copilot in the driver’s seat

By utilizing the outlined instrumentation strategies, not only are you finally understanding the black box of your customers’ conversations with your AI copilots, you are also able to leverage those insights to drive better, more personalized experiences at scale.

Importantly, Twilio Segment’s integration capabilities are not just single-sided. Our Profile API and Functions interfaces also allow you to send consistent, clean, and enriched customer data to your AI Copilot to drive more unique and tailored AI customer journeys.