Win-Loss Analysis Sampling: Selecting the Right Deals

“How many interviews do we need to do?”  That’s a very common and valid question clients ask me at the outset of each win-loss program engagement. There is no hard and fast answer. Each situation is different. In answering that question, I’ve provided some best practices to consider, along with some cautions about what NOT to do.

Mistake #1: Trying to do too many interviews

Don’t try to come out of the gate with a huge number of interviews. You’ll find that it’s too difficult to get the contacts and participation that quickly.

I always recommend starting small, with very high-quality deliverables. Then grow your program iteratively – because you can’t go from 0 to 100 interviews overnight. You’ll find that you need stakeholder support, and you’ll need to gain trust.  Rich, insightful, and well-summarized interviews, and a very diligent selection, distribution, and analysis approach, will help you gain that trust.

Some win-loss providers will tell you that you need to do a very large number of interviews to get to some level of statistical significance before you can take action.  But I’ve found that even doing a very small number of interviews has been helpful to my clients.  One interview alone can be valuable because, with win-loss analysis, it’s not just purely a numbers game.  The qualitative data can be just as important as the scores for the vendors across the criteria.

Mistake #2: Not properly sampling based on your demographics

You have to know which deals to interview (and which ones to avoid).  An intelligent sampling approach starts by identifying your key demographics (that is, the slices to want to look at) beforehand.  Then sample in a manner that allows you get some patterns and/or statistical significance in each demographic. You’re going to want to look at how the reasons you win and lose differs by demographic.  You do this by looking at subsets of deals. For instance, generate a win-loss chart using just the deals in which you competed against competitor-A. Then you’ll generate the same chart, but using only the deals where you competed against competitor-B.  Then you’ll compare the charts by putting them into a dashboard (as an example, see the dashboard of our win-loss portal that supports this type of analysis).  The differences that you observe will provide you with more insight.

What do I recommend?  To get started, focus your sampling on a few top demographics.  Sample only your top three competitors, selecting hard fought deals that were recently closed (won and lost).  Sample each of your regions.  Choose another demographic, perhaps industry or deal size.  Aim to do between 16 to 32 interviews in your first round.  Focus on delivering very high-quality, insightful interviews that wow your management. Repeat in a regular cadence, which is most often quarterly, but for some companies, this will be yearly.  Look also at the data across all of your demographics and do an analysis.  If you lack some level of statistical significance in one or more of those demographics, increase the number in the next round for that demographic. Now that’s a recipe for success.

Mistake #3: Picking the wrong deals to sample

Interview only those deals that are aligned with your strategy.  For example, if your company wants to go in the direction of selling bigger deals, don’t interview the small ones.  And avoid sole-source deals where there are no other competitors.  Avoid deals that are upsells of seats to existing accounts.  Focus instead on the key competitors.  You don’t need to interview deals for every competitor – and if you believe some of the less frequent competitors are very similar in their offerings or sales approaches, you can group them together in a bucket for sampling and apply the same competitive strategy against both.  Finally, shoot to interview an equal number of wins and losses. You’ll learn from both of them.

Mistake #4:  Waiting to complete your Interviewing before putting findings into action

Once interviewing starts, you’ll see patterns right away.  For instance, you’ve only done 5 deals and you notice competitor X switches its pricing to an all-you-can-eat, enterprise license at the end of every sales cycle. The pattern is clear – we don’t need wait to do an analysis in order to put this into use. Forward the information on to the sales team before they encounter the competitor again.

Mistake #5: Not focusing on interview quality

The quality of the interviews is hugely dependent upon the people involved—the interviewer and respondent. When selecting the respondent, ask your sales person to recommend the most knowledgeable and open person to interview. Qualify that respondent on the phone as part of the scheduling process to make sure he or she will be forthcoming with the details.

Equally if not more important is the interviewer.  The key to delivering insight is having an interviewer who can draw real depth and participation from the respondent, while understanding the context of your competitive arena.  Only use highly knowledgeable, skilled interviewers who have deep and varied backgrounds, and who employ a rigorous approach.  See an example win-loss interview on predictive analytics.

You don’t want a junior level person buzzing through a script; you need someone to engage the respondent in a conversation so that the respondent opens up. Consider this… What will happen if the interviews you produce offer no more insightful than a sales-debrief? Your program will get shot down.

So when a company asks me about how many interviews they need, I like to step back and ask a few questions in return.  Ultimately, your goal should be to execute your program flawlessly with high quality deliverables.  Then you can use your that research to recommend the changes that will increase your competitiveness.

More information about statistics in win-loss and market research: