Ep 289: Investigating AI Voice Agents as "User Researchers"

We'll read my new article about ​how I tested an AI voice agent. https://rbefored.com/ai-user-research-conversation-cc06367c4efb  And since I opened up this AI agent to my community, we'll see what else it captured.

Holy cats, it's here! You can now buy my new book in most formats. https://AtomicPMF.com 

Check out my upcoming free webinar about better product-market fit and using AI thoughtfully in UX qual research! https://atomicpmf.com/lightning

Sorry this posted out of order!

Transcript

SPEAKER_00 0:53

Welcome, low ego action heroes and phoenixes. I'm Debbie Levitt from Delta CX, and welcome to episode 289. We're going to be talking about AI voice agents that claim to be your user researcher or interview moderator. I've been trying one and I want to give you some of my thoughts on that, what it might be good for, what it might not be good for. As always, hello to anybody who is watching this live or later on, especially all of you catching it on the podcast version, the audio-only podcast version, which are now all available. And you can find links to everything at dcx.to. I hope that everybody will join our free Discord. We're close to 700 members now. It's a pretty active place and super friendly and supportive and completely free. We still have the Patreon, which is totally free. I kind of see that as my mailing list. And um, though it's interactive also. And we've got lots of events coming up. I'm teaching a course in, I've got one in April and one in May, and it has to do both with atomic product market fit, some content from my new book, but spending more than half the time on teaching my Claude and AI techniques on how I do my work. So for those of you looking to expand what you've been doing in the AI universe and do it thoughtfully and as ethically as possible, uh, my course is there for you. So you'll find all the links on dcx.to. So we're gonna start today with an article that I wrote which references some uh a user research tool that I tried out. And then I'm gonna dive into some of the other sample uh responses we got to that uh AI voice agent, and we're gonna talk about what is this good at and what is this not good at here in March 2026, since obviously these tools are are quickly uh evolving. So let me share my screen and we're gonna start with my article that I wrote about this, and then we'll take a closer look. So um my article from February 2026 says, in February 2026, Fireflies, hashtag not sponsored, asked if I wanted to create a voice agent. They said it could be used for user research. Sure, I'd love to try that. I used its default questions, of which there were only a few. Can you describe how you currently do this task or workflow? What part of this experience, uh what parts of this experience are most frustrating? What parts work well for you today? How often do you run into issues here? These are not my favorite questions, but let's see how the default questions go. One might imagine they would produce excellent results, especially if the bot was trained on using these questions. And I have a screenshot from it, which we'll go into it in a moment. Here's how that spoken conversation went, transcribed by Fireflies. I spoke into my computer headset. Fred, the Fireflies AI, spoke with a male American voice, though I think there was a choice of voices in the settings, my commentary afterward. So here's how that conversation went. Uh, Fred. Hi Debbie Lovett. Thanks for joining. How are you doing today? Well, I found it really weird for an AI to ask me how I was doing, but okay. Uh, and I said, okay, how about you? Because I wasn't too sure if I'm supposed to be polite to an AI. Agent, I'm doing well. Thanks for asking. I'll be asking a few questions to understand your experience and perspective. There are no right or wrong answers. Feel free to take your time. Sound okay? Debbie Love It. Sure. Agent, great, thanks. Can you describe how you currently do this task or workflow? So when I built this, I hadn't specified it and I let whoever answered it talk about whatever task or workflow they wanted to just to test this agent out. In a new test I'm running, it's more specific. Um, yeah, so I said yeah, it started when I was trying to buy a rack case for my DJ mixer, and I had a lot of problems trying to buy the right thing that would fit my mixer. Agent. Could you walk me through a bit more about what that process looked like? Me. Yeah. I was using the website to try to find what would fit my mixer, but there were a lot of confusing terms like things that were measured in U, the letter U, and RA. These were terms I wasn't familiar with. I could measure my mixer, but I didn't know what it was in terms of U or RA, and it was hard to find the right thing on the website. Agent. Got it. What parts of this experience are most frustrating? So it's moved to the next question. Me. The parts I didn't even tell you about yet. There's an entire task in which I was trying to buy the right thing and ended up having to deal with customer support and they were trying to guess what would fit. It was really quite a long task and process that we haven't really covered. Agent, sure, take your time. Can you walk me through what happened with that? Me. Yeah, after I did some searching on the site, I ended up contacting customer support and I tried to ask them what would fit my mixer, and even they weren't particularly sure. They ended up writing me back with some suggestions and went back and forth a number of times until I selected an item which they confirmed would fit. And so I bought it. But when I got it, let's just say it was not as described or expected, and I ended up having to return it. Even customer support admitted that it didn't end up working out for me, and they gave me a refund. Agent. Thanks. So just to make sure I understand, what were the most frustrating parts of that whole experience for you? So again, it's cycling back to the next question it's supposed to ask. Me. Really? Everything. I'm not sure any part of that went particularly well. There were problems with all of it. Agent. Okay, what parts work well for you today? So you can see it accepted that as an answer without getting details and moved to the next question. Me. I guess the only thing that might have worked well, if I had to pick something, was that this website did have a good variety of equipment. I had many choices, but of course I couldn't tell which choices were right for me. Agent, I got it. How often do you run into issues here? Me. Well, how often do you shop for a rack case for your DJ mixer? So you know this is the first time I tried it and it didn't go well. I'm not sure frequency matters here. Agent, I had a brief connectivity issue. I'm back. Let me continue. Me. Sure. Where were you? Agent, we just finished going through all the questions I had asked. If there was anything else you'd like to share before we wrap up, me. I don't think so. I guess I'm wondering what your key takeaway is from this. Agent, I appreciate you asking, but my role here is really just to listen and gather your perspective. I'm not here to interpret or draw conclusions. That happens later in the process. Is there anything else about your experience you'd like to add before we finish up? No thanks. Agent, thank you for sharing your thoughts today. I really appreciate your time. Take care, me. Bye-bye. Here was the AI summary of that conversation from Fireflies. So I didn't write this, I didn't ask Claude for it. This came with Fireflies. Um Lucia says, This is the worst interviews I've ever seen. One of the worst, yeah. Um, well, we'll see some more before we're done today. Um, so the AI summary was product fit confusion. Customers struggle with unclear sizing terms leading to poor product fit and multiple support queries. Support inefficiency, customer support lacks tools for accurate product recommendations causing delays and increasing operational costs, frustrating experience, a complicated purchase process, risks negative reviews and churns, especially for infrequent buyers like Debbie. Now here's the thing: I buy often from that site, but I don't often buy a case for my mixer. That's a buy one and you're done kind of thing. Need for clarity. Technical terms must be simplified to enhance user experience and reduce return rates. Improved guidance, better fit tools and clearer specs could empower customers and minimize support loads, enhancing profitability. Now, if you're anything like me, and many of you are, and that's why you join my community and watch my videos, you've probably noticed that we've gone from conversation summary to suggested solutions. So, just a heads up that this is no longer an AI summary of the conversation. These are solution suggestions. Trust building, smooth first experiences are crucial for infrequent purchases to foster customer trust and loyalty. Continuing with my article, would this be news to your company? Let's imagine that you want to research customers who have had issues that led to item returns. Which of the above do you not already know? Is any of this news to you? More importantly, after reading that conversation, what do you understand about the customer's experiences or journey? Do you understand enough hows and whys to detail the root causes, create a strategy, and consider solutions? Does the user really feel listened to, understood? Did we meet or exceed research goals? What would a human researcher do differently? A well-qualified human researcher would have understood that there are multiple tasks here. Tell me about a time, or walk me through your task wouldn't be granular enough to understand what went well or badly for me during shopping, selection and decision making, communication with customer support, receiving the item, realizing it wasn't right, dealing with customer support again, and returning it. My journey had multiple tasks or subtasks and multiple experiences. The devil is in the details. The problem started with me not understanding how rack cases are measured, but my problems got worse when customer support recommended the wrong items. There was a whole panic-filled task when the rack case came, and I thought there was something wrong with me because the case didn't fit as I expected. This AI conversation is one centimeter better than nothing and does not replace or augment humans. I would say it shines a spotlight on how much we still need human researchers. As a professional researcher, I wouldn't add a tool like this to my research studies. That was a very limited conversation. Perhaps it's just fireflies and it being a new feature. But I've tried a bunch of these now, and these are still not tools that I would add to my process. If you are not speaking to customers at all, this might seem amazing. This might feel like a customer connection that your company has never made before. And that's a great first step, though that connection was made between your customer and a robot that didn't really care about them, so there's plenty of room for improvement. Now allow qualified researchers to have decent conversations with real humans. They'll appreciate it. They'll appreciate someone really listening and responding to what they're saying, hearing what they say and what they don't say, digging for the hows and whys that our company can understand so that our company can understand what's going well or badly. AI can ask questions, but it's not really a moderator or researcher. We should understand the difference and set our expectations accordingly. You can try these AI tools, but you'll need to understand ahead of time that the technology in early 2026 is still very much like a badly spoken survey. We're asking people fixed questions. We won't be able to improvise too many follow-up questions. We won't be able to understand a complex task or set of tasks. We might not learn whys and hows, but we can get some feedback. Is there a scenario in which I would use this? Not at this time. To me, this is still fundamentally flawed and the tech isn't there yet. I feel like if I wanted to survey someone or ask short questions, I could run a survey. I wouldn't use an AI voice agent to learn, to really learn tasks, behaviors, journeys, or anything with complexity. It could be good if you have a survey or something close to it, and you want to open this up to people who would rather speak answers due to time, context, or accessibility needs. Hey Deb, did you prefer talking to the AI, like some AI companies claim? I felt like it wasn't really listening or understanding. I felt really awkward when it asks how I am. Um, I guess I'm okay? Do you ask an AI how it's doing today? That seems polite, but it's a robot. It's not going to say, oh, it's humid and my knee hurts. I felt like it was dying to barrel through the four questions I knew it had to ask. It asked me a broad tell me a story question, but then didn't let me tell that story. As soon as I took a breath, it went to the next question. So it got pieces of the story. It only let me tell more of the story when I pointed out that I hadn't finished. And still, it didn't get enough of the story. It got bits that I rushed through. I felt rushed. Thinking about research sessions with human researchers, I didn't smile, I wasn't friendly, I didn't feel like someone really got my story. I didn't get follow-up questions that dug into my experience. We tested Claude on the same story. In January 2026, we tested Claude as a chat-based UX researcher and had it ask me about the same story buying and returning my mixer case. I scored Claude a 3 out of 10 for moderating the session. You excuse me, you can watch that video here. And I need to quickly find uh what number video that was so I can reference it for people because obviously the link on the screen is ugly. Uh let's see, can Claude be a CX or UX researcher? January 2026 edition, episode 282. So if you go to uh YouTube and you type Delta CX 282 or 282 Claude, it should probably come up. So there you go. Uh you can find that one if you want to see how uh Claude did. Continuing the article, Claude got a three for writing its own questions and improvising questions that were okay, not the best questions. You could tell it was trained on material that probably came more from non-researchers excited about research than professional and experienced qualitative researchers. If Claude got a three, Fireflies in February 2026 got a score of one. This was barely a conversation. This was a survey disguised as a customer conversation. Fireflies didn't have context and didn't ask about it. It had four questions and it couldn't seem to improvise other questions that would help it gather more data. Um, okay, so that's my uh article. We're gonna look at more of those Fireflies responses in a moment, since we put it out to my online community and offered it to the others if they wanted to uh respond to that agent. And some people did, and we're gonna take a look at their answers in just a moment. I also want to read out loud the uh what Anna Lucia said, which was if I are the one, if I were the one being interviewed and my interviewer was one of these bots, I would issue a complaint. It feels a lot like, oh, look, we didn't care, just that. We actually don't. Also, the AI decided to suggest a bunch of platitudes, stuff that anyone could have said and doesn't actually help. We don't know anything other than you had a bad experience. Yeah, that was kind of my point, also. Um, so let's go back to my shared screen and I'm gonna go into Fireflies voice agents. And um, I did try to look up some other tools, but all of the tools, and I'm a paid Fireflies customers right now anyway. Hashtag not sponsored, and so they made this available, so I figured I would try it. I looked at, I think, seven other AI voice agent websites in the last few days. Um, and all of them were either contact a salesperson for a demo. So that means taking time out of my day to have a salesperson pitch me and going into someone's sales funnel where they're gonna follow up with me, or uh, and I think nearly all of them, if not all of them, wouldn't tell you the pricing. The pricing was you were gonna have to talk to a salesperson. And whenever I see a heavy reliance on salespeople, I tend to assume, though I could be assuming incorrectly, that this is just not a great product that's going to sell itself. Um, if this product were that good and would sell itself, and if I saw the demo, I would go, holy crap, I need this. Then you don't need uh a salesperson to sit with me in a demo um or to uh talk to me about pricing. Just tell me what this costs. You know what it costs, just freaking tell me what it costs. Don't make that something that I can't easily find out. So just to show you how Fireflies works in in March 2026, you can start by building a voice agent. The voice agent currently has knowledge, so you can put in company knowledge so that your AI can be smarter about the context, which I generally did not do. So I don't know if I would have gotten different results if I had put an extensive knowledge base or spreadsheet or database or information uh into this. I'm not too sure when and why I would do that, but maybe if this is a support bot, it would make sense to upload my knowledge base. But if this is going to be a user research or discovery research uh AI voice agent bot, I don't think we need a knowledge base. But again, it's something you're welcome to experiment with. Then there were instructions. How should the voice agent behave during the conversation? I didn't put anything there, I just wanted to see what it would do. Default, out of the box, nothing changed. Anna Lucia says, not to mention what will they do with that information. I think Fireflies claims to not train on your stuff, but always good, as Anna Lucia is pointing out, to read the terms and conditions or privacy policy or both and make sure you understand what any company is doing with your data and these recordings. Uh, then there are some, I'll go up to the top in a moment, but there are also some settings about what language you want the bot to speak. There are uh looks like about 20 or 30 languages here. You get to pick from a few different voices. There are only six voices. Um, tone, professional, casual, or talkative. Um, these were a little weird to me. I put I just left the default professional. It sounds like if you change it, you can then preview it and hear what the voice sounds like. And if I open up uh the more menu, I can speed up the the or slow down the speed of the bots talking, and I can say what is the maximum length of the recording that I want, uh, because that would help kind of time box the recording as well as help you control how many AI tokens you're you're using. So I set it on a 10-minute conversation, which is short, um, but I wanted to test it out without burning too many tokens. Up here are the questions. These are the things that the bot has to ask in the session duration. The languages don't include regional differences, Latin American Spanish. No, I don't see regional differences, I just see general languages. So I don't know which Spanish you're getting. I don't know if you're getting Spain, Mexico, Peru, Latin America, or anything else. Uh, and same with any of the other languages here. I don't know if the German is going to be more Germany German or Austrian German or Swiss, probably not Swiss German, but there you go. And Chinese, there's a z many Chinese languages, maybe not a zillion, but which one we're getting, I don't know. Uh, you can rename this, you can rename your company. So I just put in, I left it with most mostly the default questions, but I did find in testing this the other day. Day in a separate voice agent, not this one, that if you didn't put a question mark at the end of the question, it skipped it. Yeah, that was weird. Um, I had a sentence that was something like, Tell me about the last time you bought a laptop. Period. It skipped it. When I changed it to a question mark, how did I know to do that? I had to guess. There was nothing there that said, your question won't be asked if it doesn't have a question mark at the end. That was me guessing, and I guessed correctly. When I put a question mark at the end, then it asks that question. And that was really, really weird. So um these are the questions that the voice agent will ask. This is how it comes through to me. So it comes through as a transcript uh on the right side of my screen, and the AI notes on the left side of the screen. I I thought the AI notes were longer than the transcript, so you also have to ask yourself the value of that. Um, so here I already read these two from the article, but then there were more notes here. Debbie's purchase of a rack case failed due to unclear fit information and confusing terms, and it's got all these bullet points. The website offered a variety of equipment, which was the only positive noted support and service challenges. But again, we went so shallow on these. We didn't go deep on these, and I would worry that a bad tech worker of any title, not pointing at any title in particular, might look at this and go, Oh, yeah, now we understand our problems. Yeah. Um, customer support told Debbie that something would fit, didn't fit. Don't you need more of that story to understand what happened so you can fix that? Um, I feel like there was a lot of my story that I never told because the AI rushed me through those questions. Ultimately, the AI was allowed 10 minutes to talk to me, and our conversation ended four minutes and 42 seconds in. So you also have to ask yourself, is the agent maximizing the time you've allotted? If I told it don't spend more than five minutes, then it did a good job time boxing that at five minutes. If I say give people 10 minutes, this conversation is a little short. Let's look at some of the people from my community having the same con having a different conversation with the same agent. Remember, everybody was uh, we didn't tell them what they had to talk about returning or what their process or workflow was. So everybody talked about something different because the agent just said, What's your process or workflow? Some people didn't even talk about a return. So, yes, it was badly written and it was all over the place, but I just wanted to see how these conversations would look. So this one was with Mags, one of our great community members, been a member for many years. Um, and what I've noticed is um the agent starts with, How are you doing today? And I find that people, when I read these transcripts later, people generally seem to be responding positively to that. I didn't see people being like, Why the hell are you asking me that, you robot? Um, it looked like people mostly took that question well, which I think is interesting. Um, so how do you do the task? So, how do you currently do this task or workflow? Mag says, which task? And the AI says, Oh, sorry the confusion, shopping and returns. So that was something that was here in the knowledge. We're asking people about their shopping service shopping and returns process to understand what goes wrong. But the AI never said that to me. What I found in the second tests I've been running with a different agent was that if you type instructions into it and you say, please say this to the user, and you put it in quotes, it will say it. So you could say, uh, hi, I'm an AI voice agent, and today we're going to talk about your laptop shopping experience or your musical equipment shopping and returns process. So you can put something in the instructions in quotes that it will say. Yes, Anna Lucia is making a great point. Let's go back to the transcript. Um, that was a different uh yes. So uh Anna Lucia said, Why did that bot say great thanks? Yes, because I've noticed that that's something that a lot of these bots say. I tried one last year. I don't remember the company name. It was some startup that wanted me to check out their user research bot and uh did not expect the reckoning they got from me. And when I tried it, it asked me a question and I inhaled to answer it. You know me, I breathe loudly. I inhaled to answer it, and before I could talk, it went, great, thanks, and went to the next question. So I feel like that's something, and it's a totally different bot from a totally different company. I feel like that's something that somebody programmed in as a great way that a research moderator thanks the person for whatever they've said and transitions to the next question. So uh agent asks Mags how they're doing. I'm doing okay. Great, thanks. So yeah. So let's take a look at Mags. Mags shops online for heavy items and locally for urgent needs, balancing convenience and community support, frustration from unclear return instructions, inconvenient drop-off locations, uh, openness to AI support. I don't remember seeing that in here. Um so openness to AI support. Where did Mag say that? Mag's values, clear instructions, and quick resolutions, showcasing the importance of user-friendly AI interactions, but I don't think Mag said that. Mags talked about heavy, bulky items, what worked well for me, uh, easy to identify, I can take care of the rest. How often do you run into issues? Not often. Wrapping up. So, uh oh, here it is. This is the AI part. If it's a digital item or item I bought online, I sometimes talk to whatever AI chat or chat they have, sometimes a sales wrap and get that going as quickly as possible. If it's in person, I would carve out the time to go, but I return items that I buy online and it's a quality. Did Mags say they're open to AI support? Yes. I guess Mags said that they're open to use AI because they're just going to use whatever chat the company has. But look at the assumption that the AI made here. It showcases the importance of user-friendly AI interactions. I don't think we got deep enough into Mags's experience with AI chat to understand what's going well or poorly with AI interactions, or even if Mags values an AI and AI chat being friendly, or what does user-friendly look like? So I tend to worry about some of these uh these things. This was Mira who uh also jumped in and didn't understand what uh that this was going to be about returning an item. And who can blame Mira? Because that's not clear in the way that the AI starts. So take a look. Hi, Mira, how are you doing today? I guess I'm okay. I'm a bit tired, but fine otherwise. Again, I notice it's interesting that people have a kind of an interesting moment of honesty or openness when the AI asks them how they are. They don't go, you're a robot, why do you care? And then the agent says, Thanks for being here. I'll be asking a few questions to understand your experience and perspective. No right or wrong answers. Sound okay, yeah, sure. Great thanks. So again, now they're using great thanks, which I found is a common thing baked into these. I don't know why. How do you currently do this task or workflow? And again, that's the way the question is written, so that's what the agent asks. Um, Mag said she would use the AI chat sometimes, and I can read between the lines. She never asked if she liked the solution. That's right. Mira ended up talking about some student debt and didn't realize we were talking about shopping and returns, so got it. You know, this went off the rails a little bit, but it's a good reminder that if you are going to write these, your questions have to be specific, or your opening text has to be um introducing people to the topic at hand or a better recruited group. I didn't recruit this group very carefully, I just asked my online community who wanted to try an AI uh voice agent bot. Here's Dominic from our community. And again, notice how open he is with the agent when it asks how he's doing. I'm good, thanks. How are you? Which is polite. It's a Friday, and I'm looking forward to the weekend. AI agent says, I'm doing well. Thanks for asking, which again is an emulation of humanity. The agent isn't doing well or poorly, it's a robot. But then look at what the agent says next. Yeah, Fridays have that nice feeling to them, don't they? So again, it's interesting that the agent has obviously been modeled on something that is supposed to emulate some sort of rapport. So um I found that interesting. And um, you know, to you know, feel however you want about it. Um, can you describe how you do your workflow? Again, Dominic went in in his own direction and you know, uh ended up talking about some annoyance I think he was having with job interviews or something like that. So then you have the AI summary. Enhancing stakeholder education can elevate research impact and team collaboration quality. So I think not knowing and it not being clear that this was an agent uh designed to talk about shopping for and returning a product. Dominic talked about some uh instances he had with uh user research and stakeholders. The bot is kind of jarring. It says something and then thanks. Yeah, um, it's a style that I've noticed in some researchers, which has probably crept into the training, which is a lot of thanks for sharing and got it, thanks. And I do a little bit of that, but uh I think you're right that this does it a lot more. Let's look at who else did the bots. Here's Shalen from our uh community. Uh bot asks, how are you doing today? Shalen says, I'm good. It's Monday. Generally, Mondays are a little bit slow, you know, blue Mondays. So again, interesting that people are uh opening up a little bit to the bot when they could have just said, Hey, fine, thanks, keep going. Hey, Paula, Paula, Paula, I think you are. Quite uncanny behavior. I'm fine, thanks. And then the next sentence. Um, and look at the agent trying to do fake rapport. I hear you. Mondays can definitely feel that way. I'll be asking a few questions. So there must be something built into this agent that says, uh, hey, these are rapport-building things. Say something about what people say. And if you don't know what else to say, say great thanks. Um, so shopping returns workflow. So here's Shalen talking about shopping and returns workflow, lockers, uh, Amazon, blah, blah, blah. What's frustrating? Thanks for sharing that. So, again, I think these are techniques that the bot was trained on. I do say that, but sometimes what works well, what works well for you? In what scenario? Interesting. So, Shalen, we we blew through her transcript, but she must have talked about some different things. And again, we do have the mixed message here that we're talking about shopping and returns. And so when you accidentally or on purpose ask a double question, tell me about a time you shopped for something and returned it, or tell me about a time you did more than one task, you do run the risk that you're gonna get a mixed or confused or incomplete answer from your participant because what are you referring to? And so I wrote some of this on purpose just to see how the bot would handle it and how the user would handle it. And so uh Shalen notices that she's been asked two different things, shopping and returns, and she says, in which one? And the agent says, in the shopping and returns process. So the agent isn't smart enough to realize these are two separate tasks and just lumps them together. Shalen gives some sort of answer. A agent says, typical research moderator trained things, that makes sense. That's a common neutral thing that we say. I train people to say that. So there you go. How often do you run into issues? Not often. Could you tell me more about that? When do issues come up? What does that look like? When do issues come up? So it's weird that the agent probed here because you would think that you would maybe probe when someone says they do see issues come up, but Shalen is saying the issues don't come up very often. I don't know why the agent probed with Shalen, but didn't probe with other people who tried this out. And now it's saying, tell me more about that. When do issues come up? What does that look like? Now again, I think they've over asked the question. I think could you tell me a bit more about that? Is enough. Um, and so this is a little weird. Um, not great research stuff. Um, oh, and then uh before we wrap up, is there anything else you'd like to share? And then Charlene says no, but then she goes into a story. I did an IKEA return. That was quite different, difficult because the package came in different boxes. Um, and the agent says, got it. That sounds like a hassle for sure. And that's tough because usually I train people to not say some of those things because you don't want to uh judge the participant's situation or uh put words into their mouth about their situation or give your assessment of their situation. Um so you know that that's usually not something that I recommend. That doesn't mean I've never done it, but it means it's usually not something I recommend. But again, I can see that this agent was trained with a couple of rapport-building types of things. Now, again, notice that we've allowed the agent to talk to people for 10 minutes, and this one comes in at six. So that's four minutes of content, information, probing, follow-ups, context that we could have gotten. I would have burned the AI tokens, and we didn't get it. So it sounds like this agent is not yet good at managing that time where it's so worried about finishing the questions under the allotted time that it's not uh probing and trying to get more answers. And I think we had one more person try it, Cherish from our community. Uh thanks for joining. How are you doing? I'm good. How are you? So it's another person asking the AI how they are. Shalen was the only one so far past the five-minute mark. Let's see how Cherish did. Yep, Cherish is under four minutes. So again, I don't know why the AI is biased toward a shorter call, even when I've said I will pay for the credits. AI says, I'm doing well. Thanks for asking. Um, how do you do this task or workflow? Charish says, which task? Agent says, good question. I'm asking about your shopping and returns process. Cherish says, could you be more specific? Now we know Cherish is a professional researcher. She's great at what she does. She's probably testing DAI a bit, and I I love her for it. So thank you, Cherish. Agent says, sure, let's stop with short with shopping. When you buy something, whether it's online or in a store, can you describe how that process usually goes for you? And then if you've ever had to return an item, what does that look like? Take your time, no rush. Okay, so obviously, those of us who are even unprofessional researchers or still learning on Elusias is I'm laughing. Um, we know that this is a badly formed question. You may also refer to chapter eight, an atomic product market fit, as a reminder as to which of these questions are bad and why. Um, so we have over-asked. We have asked too many things for Cherish to think of at the same time. Cherish was great. Question the bot. Yeah. And if I ever post these and you want to challenge them, please do. Feel free to mess with the bots. I'm paying for the same AI tokens no matter what you do. So challenge it. So notice the AI now has to improvise because the AI has given Cherish all the information I fed it, and now it has to improvise. And when it improvised, it went all over the place. So when you buy something, whether it's online or in a store, describe how that process usually goes. And then if you've had to return, what does it look like for you take your time? So it sounds like it's probably been trained on a take your time, no rush way to be friendly, open, um, you know, uh, but the funny thing is, I always find when you say to someone, take your time, no rush, sometimes they feel more rushed. But um, Cherish is obviously trying to answer the question. Cherish's answer, based on the timestamps here, runs 35 seconds, if I'm seeing that correctly. 45 seconds? 45 seconds. Um, and she's kind of blowing through everything. And then the agent says, Got it, thanks. What parts of this experience are most frustrating? So again, not probing, not going deeper, not finding out the whys and hows. Now, just because Fireflies is doing this, it doesn't mean that every other AI voice agent is doing this. There might be other ones out there that are better at probing or improvising, but I wouldn't know because I couldn't try any of these for free. All of them required me to either start paying an unknown amount of money because I would have to talk to a salesperson, or they would require me to have a live demo with a salesperson. I couldn't just go into the bot and have a conversation with it, or set up a test bot and test it. So uh Cherish says, I guess it's frustrating when it's difficult to figure out how to return something. The bot probes, which is decent. Could you tell me more about that? And again, this time it uses a compact single question, which is better, excuse me. Uh, what works well for you? How do you run how often do you run into issues? That covers a question. Anything else you want to share? I just think I would have more specific information for you if we talked about specific purchases. Thank you, Cherish. Cherish recognizes that the bot was only going to do so well if we didn't zero in on one particular experience and dive more deeply into that. And the bot says, that makes sense. Thanks for sharing. Anything else before we close? No. Ending the call now, which I I think is bad training, and we should probably it should just say thanks for your time, take care, and and end the call. The ending the call now is uh really awkward. So yeah, that's what uh that's what the Fireflies voice agent looks like here in March 2026. Um I tried it with uh a little bit something more specific where I tried to write more specific questions, but I found that it still really did not probe very well. Um, Anna Lucia says, What's the Ask Fred tab? So Fred is the name of the bot uh over at Fireflies. So I can ask this is like ask the AI for something in the the transcript. Um so for example, I'm Fred the assistant. I can ask any question about your meeting. So remember, Fireflies is mostly built for recording meetings, transcribing meetings, summarizing meetings. I'm using it for the transcription. That's it. I bring it into Claude for other things, but you can um you can have it summarize stuff, and then you can ask the AI uh questions about stuff. Uh, what did Cherish find most frustrating? And so Fred is checking it out, and now Fred says, Cherish found it most frustrating when it's difficult to figure out how to return something, gives me a timestamp. The frustration arises when return instructions are unclear in purchase confirmation emails, timestamp when companies don't include a clear workflow in the purchase confirmation email, she has to hunt for the policy. So Ask Fred is just you asking about uh the particular recording or conversation you're looking at. I'm gonna ask it now what should the agent have asked to get better information from Cherish? Something more actionable for our client. Just inventing this just to see what the agent says. Based on the meeting, oh, here we go. Lots of stuff. So Fred says, here's some questions the agent could have asked to get more actionable information. One, specific examples. Can you think of a recent purchase where you had trouble returning it? What company was it, and what made the process difficult? Okay, we know that's a lot of questions all thrown together, but I recognize the agent realizes we did not get people focused on one thing. Return policy details. When you have to search for the return policy, what information is typically missing or unclear? Are there specific details you're looking for that companies don't provide? Again, that's too many questions in one, but I see what it's aiming for here. Email content specifics. What would be the ideal way for a company to present return instructions and a purchase confirmation email? What information should be included? Again, it's a bit over-asked. So uh even Fred recognizes that the agent didn't do great and really also and also notice that Cherish herself notice that the agent didn't do a good job getting her focused. So that is the ask Fred feature that you'll see around Fireflies. You'll see this in other tools as well. When I briefly used Fathom, which I no longer use, there was a whole ask the AI section where you could ask it questions about the uh the conversation that was had, and it would make sure to find the answers only in that conversation, not from any other training or common knowledge or or anything else. Um, so winding down our um uh webinar for today, our Take Action Tuesday, episode 289, um, I wanted to just mention that ultimately I think that we should test these out. We should try these out, we should know about them, we should be ready to speak to them. We might be working at places that say, ah, Debbie, you don't have time to do all the research that we need. We're gonna put some of these bots out there. And I think that the bots can probably do an okay job if you have a well-written survey type of question. If you have a very narrow, fixed, limited, clear uh survey style question, then I think like any well-written survey questions, if you have an audience that would rather speak that answer than type that answer, then a bot like this could collect that answer and could help you gather that data. And then you could probably bring it into Claude to analyze it. I'm not sure if Fireflies analyzes things across conversations or only within one conversation. I tend to export things and have Claude work with them. Um, but and again, Fireflies hashtag not sponsored. Um, I'm I'm a paying customer, they're not paying me. But I think that um I won't feel great about these until they're better at probes and follow-ups. But even when we tested Claude in the typed chat version of this just two months ago, it didn't do great with probes and follow-ups. So I think this is something that is still a weakness and certainly doesn't replace us. But I would be curious about trying this type of interface for a survey style um uh short questionnaire where uh the user is more likely to want to speak the answers than to type the answers. However, if you have someone who is unlikely to be able to speak or speak the language well or understand the language well, then you probably have to give them a uh typed or translated option. I'm not sure how you would do, you'd have to figure out a way to do that, or make this bot available in different languages, or make a typed survey version also available. So just remember accessibility, hashtag accessibility. Not everybody is going to want to speak out loud, not everybody's going to be in a quiet place, not everybody's going to understand what the agent says, and not everybody's going to speak in a way that the agent will easily understand. Just listen to my Italian. So uh just some thoughts there. Anyone, anybody joining live have any other additional thoughts on these agents? If not, we can certainly wrap up. Um just want to check if anyone had any other thoughts or comments. Um, cool. All right. Oh, Anna Lucia says, I'm a bit confused about how we could even use them. We usually apply a screener and then get people for interviews. Would this work the same? I'm not sure if I would use this for a screener survey. I usually like to make sure I get that as structured data in a survey that I can then export into a spreadsheet and decide who I'm going to include in the survey and who I'm not. I've also seen people sometimes be very biased in um some of these AI tool uh types of things where if the person has an accent or a heavy accent, people will say, Oh, don't include them in the study or screen them out of the screener survey because they have an accent or a heavy accent, and we want articulate people, which is being super racist and super crappy. This person might be very articulate and just not great with the language, or very articulate and have a heavy accent, or you know, like these are separate issues. So I'm always nervous about people using this type of stuff for screeners because I worry about AI bias and human bias, especially if someone happens to have an accent. Anna Lucia says, would this be something like someone makes a purchase and then get gets asked to answer this, which would be terrible. Yeah, again, I think this could replace short surveys if we think people want to speak answers. That's a bottom line. If people would rather click choices and type answers, if that's better or faster or easier, then this is not an upgrade. If people would rather speak answers, um, or there's some reason why we want our answers spoken, then I could see this being used to replace some short surveys. But there was somebody else on LinkedIn who claimed they they had some AI uh interviewing tool, and they claimed that people were speaking to it for two or three hours. And my thought was, then your AI is badly trained. Um, because they claimed like our AI is so great that when we set it up for a half-hour interview, it's talking to people for two or three hours. And my thought was, well, this is a mess because who are these people who have two or three hours to talk to this bot? Like, are they are they and are you giving them an incentive based on how much time? Like, there's so much here that just sets off red flags. And so I don't think that someone speaking to uh a bot for longer is necessarily a good thing. I would ask, why can't the bot get the answers it needs in the time allotted? A half hour is a lot of time. I gave this bot 10 minutes and it didn't even use it. Are they using the bot as therapy? Yeah, it was supposedly user research. But, you know, it was a LinkedIn post by the founder of the AI startup, who of course needed to make the AI look and sound very good. Take that as you will. So I'll I'll keep experimenting with these, and I think they could be interesting for um a variation of a survey, or have, or uh giving people the opportunity to take a survey with their keyboard or mouse or phone, or take a survey by speaking. I think that could be interesting for accessibility and and other things, so that people who don't want to type or can't type or find surveys too difficult to do when typing or using their hands might prefer voice. Cool, that's that makes something more accessible and that's good. But I still don't think that this uh would replace um real qualitative research that gets into discovery or explorator uh exploratory research or generative research. I think it just could uh probably make an interesting survey. Um, any other thoughts or questions before we wrap up for today? I'll start playing the wrap-up music just in case. I don't know how this can help. It's shallow. I think it really depends on the tool and how we use it. And I've only tried one tool today. That's all we can say. I didn't have access to any other tools because tools wouldn't give me access to them without talking to a salesperson, which I don't want to do, even with my love for all of you. So if I can find a tool that will let me demo it for free, if you find it, send it over to me. I'll do it, I'll screen record it, we'll talk about it. Um, but it's the only one I can find. So we'll check in on this topic, I'm sure, from time to time. And uh thanks everybody for joining. I'll see you tomorrow for uh Ask Me Anything. And uh don't forget to join our community, bcx.to, and definitely check out my upcoming um uh courses, especially if you want to learn using Claude in uh research strategy and other stuff from me as I do it. Thanks. Bye everybody, have a good rest of your day or night.