This article first appeared on Insights Association Website.

I regularly speak with prospective and current clients about their decision-making process for choosing a vendor for their projects. Often, they make decisions based on numbers that look good on paper, only to find that these numbers do not accurately predict the success of their projects. I’ve come to understand that for many clients, selecting a vendor can sometimes feel like rolling the dice.

One of the key numbers that vendor companies like my own will include in proposals to potential agency clients is match rate. Stated simply, this refers to the number of physicians from our panel that “match” with the physicians on our agency’s list of respondents they want to reach. At first glance, it may seem that a vendor that boasts a large match rate will be better positioned to efficiently complete the client’s project than a company stating a much more modest match rate. But the truth is that when it comes to match rate, bigger doesn’t always mean better. The far more important number to know is not match rate, but rather response rate.

To give an example, in a hypothetical scenario, an agency client may come to two data collection companies about a project. The client has a list of 5,000 physicians and needs at least 30 oncologists from that list to respond to a survey about a given pharmaceutical treatment. In this scenario, Vendor A may come back to the client and state that they have a match rate of 1,500 oncologists from the client’s list to their own panel. By contrast, Vendor B may only have a 400-doctor match rate. On the surface, Vendor A may seem far more likely to be able to successfully complete the agency’s project. But without knowing the response rate of each vendor’s panel, we actually know less than it seems. If Vendor A has a 2% response rate to their panel, then they will only be able to reach 30 respondents. By contrast, if Vendor B has a 10% response rate, then despite a far smaller match rate, they will be able to reach 40 respondents on the project—significantly more than Vendor A.

But the truth is that when it comes to match rate, bigger doesn’t always mean better.

When a high match rate is included in a vendor’s proposal for a project, it tends to look promising. However, as the example above demonstrates, making a decision to go with one vendor over another based solely on match rate can be a mistake. In the example given above, Vendor A may have had a larger panel to work from, which was reflected in their much larger match rate, but the quality of the panel is just as important as its size. When vendors provide their match rate, this number tells only part of the story. I’d like to recommend some follow-up questions that any client should feel comfortable asking potential vendors to get a fuller picture. These can help give clients confidence that their projects can be completed successfully.

The first and most important question agency clients can ask a vendor is, “What is the response rate to your panel?” Because the response rate on panels can vary dramatically, knowing this number will better equip you to determine whether a potential vendor can truly deliver the number of completes you need, regardless of the size of their match rate. As a general rule, I would say that a “good” response rate will be 5% or greater, although there can always be exceptions. However, if a company has a match rate of 800 but says they’re likely to reach only 8 respondents (a 1% response rate), this is a key indicator that the panel may not be able to fully recruit the project.

What is the response rate to your panel?

The issue of panel quality brings me to the second question I would recommend asking potential vendors: “What is your process for building the panel?” Not all panels were created equal. On one end of the spectrum, a data collection company may compile a large panel in part by purchasing email lists or collecting healthcare providers’ emails en masse in other ways. Such a panel may not be what we call “active,” and may actually consist of a large number of people who do not even know that they are considered panelists and who do not participate in market research studies. At the other end of the spectrum, some boutique data collection companies working on very small qualitative studies may have tiny panels, but they may have a custom recruiting process and know every physician on their panels by name. These panels are likely quite active and engaged. Other companies will fall somewhere in the middle. Learning more from a company about their process for building their panel may give you a better idea of how engaged their panelists are likely to be.

Finally, I would recommend a third question that can help get to the heart of a panel’s quality. That is, “How often do you review or clean your panel?” Our clients often work under the assumption that vendors are regularly going through their lists and cleaning them out, for example to eliminate physicians who are retired or who no longer participate in research studies. But the reality is that it takes time and money to do these thorough reviews, only to result in a lower number of panelists, so they sometimes become a low priority and happen less frequently than would be ideal. In order to find out how active a vendor’s panel is, you may want to ask whether they have checked their list in the past six months for retired doctors and those who no longer participate in surveys. This will help ensure that their panel includes “active responders” only. (One way to get at this would be to ask what percentage of their panelists have participated in market research in the past six months.) You may also want to specifically ask whether all the email addresses on the panel have been verified to ensure that survey invites actually reach respondents’ mailboxes. There are software programs that companies can use to check “deliverability,” so there is no reason for emails not to be verified.

How often do you review or clean your panel?

I believe that the questions recommended here are a good starting point for clients evaluating project proposals. Of course, when agencies hire a data collection company, they shouldn’t need to know all the minute details of the process and the statistical analysis—it is our job to dig in and collect data and crunch numbers. At the same time, at the level of the bidding process, there needs to be enough transparency that when a vendor makes a bid, there is no unnecessary confusion over which numbers mean what in terms of our ability to successfully complete the project. In my view, the confusion that can result over match rate and the quality of a panel highlights a continuing need for greater transparency in our industry. Agency clients should feel comfortable asking questions like those I’ve suggested above; often the more questions they ask, the more confident they can be. Ideally, I hope that we can soon reach a point where our clients consistently get all the answers they need to make an informed decision without needing to ask at all.