Module Description:
Ubiquity Extended Team member and Tonal's former SVP of Product & Design Sumner Paine explores how founders and product managers can establish a continuous user feedback loop to make smarter product decisions.
Full Transcript:
Welcome everyone to our Ubiquity University session on understanding user sentiment. Ubiquity Ventures is a pre-seed seed-stage venture capital firm investing in software beyond the screen. That means we back early-stage entrepreneurs who are moving software off a computer into the real physical world, often using smart hardware and machine learning to solve really meaty problems. Today, we're very lucky to have Sumner Paine, formerly head of product at Tonal and part of our Ubiquity Extended Team, and he'll be diving in with a presentation on understanding user sentiment, both high level concepts and some very tactical advice, so Sumner, thank you for being here and please take it away.
Thanks, Sunil. Happy to be here. Today, I'm gonna talk about user sentiment and product decisions. So, as everybody here probably knows, in the course of building products, prioritization decisions present themselves constantly. Every sprint at least. Sometimes even more frequently than that. And to make those decisions efficiently and optimally, what you really need is to understand the sentiment of your users. So, my thesis here is that optimal decision making for product prioritization depends on a continuous understanding of user sentiment. Obviously, many things go into making prioritization decisions. There's many dimensions of value, cost, schedule, et cetera. Most of those things come from work that you do internally. What's different about user sentiment data is that it comes from people outside the company who don't work for you. In fact, you work for them. So, it's a little bit different, it's a little bit more challenging.
I am going to suggest an approach here that is really lightweight and that can work for anybody, even a five-person startup, and that can scale up as needed. So, a little background on me is as Sunil mentioned, I ran product and design at Tonal, where I was for nearly seven years, and we introduced a product that had a negative NPS score in the very beginning of a private in-home beta. If you're familiar with NPS, net promoter score, negative is not good. You want a positive score because it means more people are saying good things about your product than bad things about your product, and that's gonna lead to positive word of mouth and organic growth. We started with a negative score, and over a couple of years, by listening very carefully and consistently to our users and understanding their sentiment, we improved that to a score above 80, where we held it for many years. We did it with an approach very similar to the one I'm going to talk about today, so you too can do this. The key here is establishing a feedback loop.
Now, you already know what decisions you're making, what prioritization decisions you're making. You have that data. That's easy to come by. What you need is that data on user sentiment. Now, it can be tempting to think that data you already have is all you need, is suitable for this purpose, and you probably have a lot of data on your users and their feedback on the product already. In conversations you have with your users, I'm sure you hear a lot about what they think. You're probably developing an intuition from that. Depending on how far along you are, you might have things like app store reviews or customer support cases. Maybe you've done user research projects. So, you've got a lot of data, but you may not have the right kind of data to establish the feedback loop that we need here. All of the examples I've mentioned here have fatal flaws, one or more of these fatal flaws. Either they're not representative in the way that they need to be of your users, they're not structured in the right way, they're not continuous in the way that we need it. So, watch out here. So, I'm saying there's a bunch of things that are important about the data that we collect, and there's a bunch of ways not to do this, so now, let me tell you what is important here.
So, I'll propose what the requirements are for a good source of user sentiment data and a good system to go about getting that. All right, so what is important about data that we collect on user sentiment? So, it's important to remember that what we're striving for here is that when there are changes in your product, if you ship an update for example, or changes outside of your product in the real world, that when those things happen, you immediately understand the sentiment of your users because it could be changing. So, the requirements here are that you have a consistent way to sample your users, because you're only going to be asking some of them for their feedback at any given time. It's that you are continuously, or if not continuously, very, very frequently asking users for their feedback, so that you have this real-time sense of how things are going. Consistency is key here. So, a consistent core set of questions with both numerical and qualitative data, where results are easy to review, meaning it's not a lot of effort for you, because you're so busy and there aren't a lot of other people who can take this on. So, the way that the data comes in, it has to make the data very easy to review. This system can't cost a lot to operate in terms of dollars or people, and it really should be something that can scale up. You can start minimally, start scrappy, but scale up as your needs evolve and as you see fit.
So, in the next three slides, I'm gonna describe this lightweight system that is something you could build, and the first part of this is the part where you reach out to the users you want to understand the sentiment from. So, what's important here are a couple of things. As we talked about, representative samples are key here. So, don't ask people when they're in the middle of using your product. Now, I've seen this a lot out there in other products. While you're in the middle of the product, something pops up and asks you for your feedback. Don't do that, because that's going to result in a skewed sample, probably of more active users, and probably not in the right mindset to be thoughtful and reflective about the product experience and the impact that that's had on them. So instead, contact people when they're not using the product and let them decide when they have a moment that is conducive to giving thoughtful feedback. Make sure you include users who you haven't seen in the past week. You know, of course you have to use some judgment about how far back to go, but you definitely wanna understand from people who are not the most active in using the product. So, I recommend email. Email is perhaps an obvious solution, but it's a great solution to meet these requirements. Automate an email. A simple approach is to reach out to every user at a certain point, or certain points in their tenure with the product. After so many weeks or months in using the product, they get an email. And make sure to avoid any temptations to make a very pretty and polished email. One thing that I've learned is simple text goes a long way. If it looks polished, it has a lot of nice imagery and formatting, it can be perceived as a marketing email and you get far fewer people responding to it.
The second part here is collecting feedback from users once you've got their attention. The important thing here is to get in and get out. Ask them what you need and let them move on. Now, I recommend a core set of questions that you will keep consistent over time. I recommend net promoter score and the two questions that are core to that. How likely are you to recommend, and why did you give that score? There are plenty of other options that are equally viable here. I think net promoter score is a great, straightforward option. A tool I've used to do this is Delighted. It's very easy to ask these questions, to add additional questions as needed to understand with more specificity or more detail what you are learning about from your users. They're happy about a certain thing? Pop in a follow-up question that gets more details about that thing, so you can really understand what is driving their sentiment. And you can change those additional questions anytime and as frequently as you need to. You're maintaining the core, and that's what'll allow you to track changes and understand trends over time.
The third part is making sense of the data once you've collected it. Now, the important thing here is that you have a pulse, a finger on the pulse of your user sentiment. What's great about the approach that we're talking about here is that it can flex and scale with you and the needs of your company. You can always dial up or down the level of analysis and the sophistication of the analysis, whether that's about adding people to the team or responsibilities to the team to understand this more deeply, or it's about ad hoc analysis because you need to do a deep dive because something has come up. This approach can flex. One thing I strongly recommend is that you as the founder read individual users' responses. It's very easy to do this with this system. I have found there is no substitute for hearing from users in their own words what it is that they're experiencing with your product. What I would do is I would read these responses in the morning during my workouts, but it's something I would do just to make sure I was in touch with the sentiment. I had my finger on the pulse from a qualitative perspective. If you use a tool like Delighted or any of the other tools, it's also very easy to watch the metrics here, because remember, you're collecting that numerical data as well. It's very easy to see if sentiment is improving or if something's come up and it's taken a dip. You don't need to do any work in order to see these changes. The tools have that all built in. And as I mentioned, you can always do deeper dives, and it can be really interesting to join the data you get here with user data or engagement data, to understand much more precisely or with more granularity exactly who is saying what about the product, and there are some exciting new tools out there now that use LLM technology to help you make sense of qualitative data. So, it can be a lot of qualitative data. The more you collect, the more users you have, the more success you have with users, and at some point, it does make sense to employ additional tools to make sense of that qualitative data, but you won't need to do that in the beginning, and you certainly don't need to do that in order to keep your finger on the pulse. Lastly, the sentiment data that you get here is a great early signal of places you may want to deep dive. You'll get signals about potential areas of interest that are opportunities or risks to the business, and you can always go do more formal user research around any of that. This is not in any way meant to be a substitute for that, but this can be a great way to understand where you should be doing that.
And it's important to remember that another reason to be doing something like this is that you could be surprised by what you learn. Sometimes very small-seeming issues can turn out to be huge drivers of dissatisfaction, and even having small cohorts of your users who are very unhappy can poison word of mouth out there, and that's not something that you want to have happen. This can be a canary in the coal mine that you can use to your advantage. Conversely, you may be hearing a lot about certain feature requests or additional functionality from your users, and it could be that giving them that, it takes a lot of time and resources but doesn't move a needle from a business perspective that matters to you. Sentiment data can put all of this in perspective and give you the context you need in order to make effective prioritization decisions. Of course, what you want is up and to the right. You want happier users and users who are happier more of the time. With the approach that we've talked about here, you can get the data that you need to inform those decisions and do all of that with very low operational costs and very low effort. If you have any questions about the approaches that I've used, I'd be happy to help, and that's it.
Perfect, Sumner. Well, thank you for that. Thank you for your help on the Extended Team. You may have some Ubiquity CEOs reaching out to you from our portfolio. More broadly, this has been our session on understanding user sentiment. At Ubiquity Ventures, we do love to hear from you. If you wanna set up a meeting, we're at pitch.ubiquity.vc. If you have questions for Sumner or wanna follow up with me, I'm at sunil at ubiquity.vc. So, thank you again, Sumner. We really appreciate it.