Steering leading indicators

User Interviews was a roughly thirty-person product-led-growth company when I joined as a PM in late 2020. The product was a two-sided SaaS marketplace for user research. Researchers recruited participants, participants got paid for completed sessions, and the unit of value was a successfully-run study. For the company's size, the analytics program was unusually mature: data scientists were embedded in product pods rather than centralized, and most meaningful product decisions were scoped as experiments, launched first to roughly 50% of users, and held or shipped based on the data.

I joined the Core pod (researcher-side flows—email theming, multi-team org accounts, etc.) and later moved over to Matching, the platform pod that owned the core marketplace. My data background unlocked the move. Matching owned two pieces that mattered for everything that follows: the targeting capabilities researchers used to define their audience, and the email-batching algorithm that fed participants into studies on a cadence to fill them.

The instinctive metrics for a research marketplace (study completion, time-to-fill) are lagging—by the time they move, the participant or researcher is already either delighted or gone. The data science team had identified an upstream signal that predicted both: Early Qualification Rate, or EQR—the percentage of new participants who qualified for their first study within two weeks of signup. Higher EQR meant participants were materially more likely to complete a session, which meant researchers were materially more likely to complete an entire study on their timeline, and both parties were materially more likely to stick around for future studies.

One metric, both sides of the marketplace, leading rather than lagging. EQR was the metric to move.

The diagnostic was straightforward and uncomfortable. Too many new participants were entering screener funnels for studies they weren't a fit for, being screened out, and disengaging. Researchers had only basic demographic targeting available—location, age, gender—but their actual use cases (often B2B research) needed to filter on occupation and skills. So researchers had been doing it indirectly: writing screener questions like “Are you a product manager?” and relying on the post-invitation screener to filter. The problem is that screener filtering happens after the invitation lands. By the time a participant is told they don't qualify, they're already hopeful if not expectant that they do.

User Interviews ran continuous discovery as a standing practice, and the same gap surfaced from both sides of the marketplace at once.

On the quant side, my embedded data scientist had already done the work to establish EQR as the leading indicator; we worked together to scope the experiments that followed, and I built my own SQL dashboards in Mode to watch what we shipped. The picture from the data became clearer with each successive experiment: qualification was concentrated among participants whose profiles contained occupational data; researcher and participant retention were increasing.

The bet was that moving qualification upstream of the invitation would compound through the funnel. If researchers could target on occupation and skills, and participants could surface those attributes on their profiles, two things should follow. First, invitations would land on participants who were more likely to qualify, which would lift EQR directly. Second, participants would feel selected for rather than filtered out—improving retention even in the cases where they didn't ultimately fit a specific study.

That second mechanism is the part that most marketplace PMs underrate. A screener rejection is a big piece of bad news delivered after a small piece of good news. A targeting attribute mismatch, by contrast, is more or less invisible to the participant entirely. The first costs you retention almost immediately. The second costs you retention over time if you don't implement other solutions (which is why we tended to iterative changes to batch size, cadence, and composition).

The pods' shipping cadence shaped the project as much as the data did. We scoped each effort to a two-to-three week build and a two-to-three week live experiment. Experiments launched to roughly 50% of users, and we decided go or no-go from the data. We then full-shipped the previous experiment while the next one was going live. That rhythm is what let a build that involved two third-party integrations and an algorithm rewrite land in two quarters, from ideation to full scale. Each new data attribute went through three phases.

The primary outcome was on the metric we set out to move: EQR rose 15%, with downstream lifts on the lagging indicators (study completion and participant retention).

An unrelated solution was also contributing to the rise in EQR during this time. While I was still on Core, we had a fragile hypothesis: if we could increase usage of the “invite past participants” feature, researchers would see better study outcomes. The data showed that researchers who used it had measurably better study outcomes; usage was surprisingly low. We had to figure out why and fix it.

I performed a heuristic analysis that suggested comprehension was the issue, rather than discoverability. A few discovery sessions confirmed it over a couple of weeks. We clarified the help text language for the feature, explaining what it was and how it worked, and tweaked the icon and its color as a secondary fix. The whole thing—diagnosis, discovery, build, ship—took about a month (fast for the pre-Claude-Code era). Re-recruitment usage rose 135%. When we began working on occupation and skills targeting, re-recruitment was already influencing EQR in a positive direction; the new releases compounded the effect of that prior work.

The biggest leverage move at User Interviews wasn't a feature in the traditional sense—it was changing where general qualification happened in the supply-side funnel. This holds true across all early-stage marketplaces.

In the case of user research marketplaces, it is about moving general qualification out of screeners and upstream of study invitations. Screener-based filtering should be niche and project-specific. If general qualification is done via the screener, the marketplace pays its cost in churn, because every screener rejection is a participant who had unrealistic expectations set and is now disappointed in the entire platform, not just the single outcome.

Pre-qualifying via attribute targeting costs more upfront—data sourcing, integration work, participant-side adoption—but the benefit compounds in the marketplace's favor over time. Every marketplace has its own version of this tradeoff. The lesson generalizes: be skeptical of downstream supply-side qualification, especially when the price is paid by your users.

The other lesson is about metric selection. The reason this project worked is that the org had picked the right thing to measure before anyone wrote a line of feature code. EQR predicted both sides of the marketplace and was upstream of every product decision I made for two quarters. A weaker org would have anchored on whichever side was loudest in a given week. Picking the right metric is product work, not analytics work, and an embedded data-science function is the easiest way to make that judgment accurately. At People Inc., the same instinct meant stewarding a multi-year identity roadmap one program launch at a time, each launch a proof point for the bet and a vehicle for the next increment of executive buy-in. The re-recruitment side story is the same lesson at smaller scale—a fragile hypothesis, a few discovery sessions, and a month of work outperformed several quarters of people not prioritizing investigation of a solution on the assumption that it would be complex.

Two next steps, if this is the kind of PM work you're hiring for: review my resume →.

Or, if you're ready to talk, book a 30-min product chat ↗ (opens in new tab).