This is an admittedly nerdy exercise: I've been playing with ways to visualize how I evaluate scientific evidence. This mental thread owes no small part to the flurry of AI-related controversies in peer review (see here and here).
AI shenanigans aside, anyone who's been through peer review knows firsthand the divergent feedback (pay special notice, reviewer 3 - you so casually delight in torpedoing our speedy accept with minor revision). Your first reviewer wants elegant genetics; your second wants deep biochemistry, and so on. In neuroscience, there's a camp that will involuntarily eject their last sip of coffee if you fail to show behavior data.
Regardless, we all bring our own priors to peer review. And while griped about by academics worldwide, there is little formal recognition of this. Rather, they're passively received as implied scientific wisdom that grant good fortune or coolly imprison you in resubmission limbo.
My approach
I began toying with different tactics to cast into clearer relief my own reviewer tendencies. After some fiddling and false starts, I landed on a React app that ingests a paper PDF and renders it as a hierarchical graph with claims and supporting experiments parsed as nodes and leaves, respectively. The user can then modify edge weights to reflect their judged strength of evidence, add notes, and export a completed review. As a case study, I chose molecular biology research because it lends itself particularly well to this type of approach since evidence can be parsed biochemically, cellularly, genetically, pharmacologically, etc. This effort culminated in the React app below:
A self-eval
With this all in hand, I set out to review a paper I know well investigating a potential cancer therapy by targeting treatment resistant tumor cells. The evidence includes mechanistic rationale, in vitro efficacy, selectivity over normal cells, and in vivo survival benefit in xenograft models.
Using this tool, I noticed I’d weighted the selectivity data equal to the xenograft survival (I’m sure readers are astonished). But this surprised me since achieving in vivo efficacy is notoriously hard. But for me, the selectivity is what makes me appreciate this study as a potential targeted therapy.
My priors
This all made me think about my habit of focusing on mechanism over therapeutic effect. This is likely a byproduct of my research rather than clinician mindset. I want to know how things work at the physical level. Without this, complex systems feel disorienting and arbitrary. I need an anchor point and find it hard to form hypotheses without a palpable, even if provisional, mechanistic mental model. Regardless, it’s useful to recognize this as a prior for me. It could lead me to be overly influenced by mechanistically compelling studies that lack clinical or real-world assessments, or alternately, to under-appreciate impactful translational work with yet unclear MOA.
I thought about developing this further as a tool, and still might, but refining it to handle edge cases and ensure accuracy gave way to a sense that this was better as a thought exercise. There are plenty of portfolio repos out there already.
In the least, it’s given me an oblique insight into ways I may be someone’s ill-gotten “R3.” If so, as for self aware review, know that I’m trying :)