VDP-53 The Longevity Decentralized Review (TLDR) - Uber for peer review service

[timrpeterson]

TL;DR

The Longevity Decentralized Review (TLDR) is an on-demand service that funds peer review using funds from those who submit their research to be reviewed. TLDR will also accept donations. The focus of the content will be longevity research posted to preprint servers, Biorxiv, Medrxiv, and Arxiv. TLDR will benefit VitaDAO in three ways: 1) VitaDAO will grow its treasury by taking in revenue from TLDR. 2) VitaDAO will grow longevity knowledge because TLDR is focused on longevity research. 3) It will augment VitaDAO’s longevity dealflow by creating a larger more decentralized community evaluating longevity research.

Problem

Publishing in journals especially in the biomedical sciences has significant issues. Journals historically provided an important function in physically printing research manuscripts. However, now with the internet anyone can make their work accessible by posting it online. Researchers are now doing this at several websites that are known as “Rxiv” (pronounced “archive”) preprint servers. Biorxiv, Medrxiv, and Arxiv are examples.

The remaining major distinction between a journal and any website on the internet is that journals have peer review. Peer review is the process of scientists reviewing each other’s work to determine if the work is worthy of publishing. In most cases, scientists recommend other scientists with similar domain expertise to a journal to review their work. Scientists then often spend many hours assessing the quality of results and suggesting changes. Often, there can be several rounds of revision.

This is a system that benefits the world that scientists take pride in, with one major issue – The journals do not pay scientists for any of this work. Rather, the journals charge large and growing fees (many $thousands) to scientists for the privilege of publishing in their journals. The journals have little costs because scientists do most of the work for free. This results in journals having massive profit margins. Some estimates have journals outdoing the profit margins even of some of the most successful companies on earth such as Apple.

To recap, scientists pay journals from their increasingly inadequate non-profit funding budgets to have other scientists do free labor reviewing for the journal. It is widely acknowledged by many that this is an unacceptable exploitative system that needs to change.

Solution

An on-demand peer review service where reviews are paid by those who submit their research as well as by donations. The business model involves taking in funds that will cover what is paid out. To grow VitaDAO’s treasury, ideally the funds taken in would be in fiat and those paid out would be in $VITA. VitaDAO also benefits in that TLDR is focused on longevity research. Thus, TLDR will help scale and decentralize its core mission of evaluating longevity research.

As the name implies, a major goal of TLDR is to enable researchers to have proof of their peer reviews such that they don’t need to rely on journals for this proof.

Benefits

• Reviewers should have the option of being compensated for their efforts.
• Researchers will obtain peer reviews of their work in a timely and fair fashion.
• The general public won’t have to wait (often for years) for research to be published in a journal to know whether it’s credible.
• Journals will profit less from the free labor of researchers.
• TLDR will not interfere with the status quo. It is an opt-in mechanism for researchers to get extra eyes on their work.
• TLDR will have a simple web interface with a familiar UX. No new skills will be needed. Web3 aspects will be abstracted away, which is important because most biomedical scientists won’t be familiar with Web3.
• This is intended to augment longevity dealflow evaluation in general to benefit more longevity organizations than just VitaDAO. The goal is to follow the spirit of how @vincent initiated the The Longevity Prize by pulling in the Foresight Institute, the Methuselah foundation, and others.
• The initial focus is longevity, but similar to Stackoverflow and StackExchange or Reddit and its subreddits, this service will hopefully evolve to handle other research domains.

Specification

There are two types of submissions. Those that will be auto-submitted and those that will be manually submitted by researchers. TLDR will be regularly auto-updated with the longevity research from Biorxiv, Medrxiv, and Arxiv. Researchers will also be able to pay to directly post their work. Thus, TLDR has two tiers, free and paid. Reviewers are incentivized to review the freely posted Rxiv submissions because they will receive a portion of the overall donations. Reviewers are more incentivized to review the paid submissions because they will get a portion of the overall donations plus a portion of the overall paid submissions.

Paid submissions will be handled as a subscription model like how software as a service (SAAS) often operates. Potential pricing tiers were determined based on polling. This polling showed $250/manuscript would be acceptable to ~50% of those polled. ~50% of people wouldn’t pay thus, the free tier accommodates them. The subscription model is tentatively $249/year per manuscript with lower per manuscript amounts the more manuscripts one submits. One also pays less if they pay yearly vs. monthly. To us, it appears more beneficial to manuscript submitters to allow reviews to accumulate over time rather than to restrict a manuscript to a certain number of reviews. Said another way, a time-based subscription payment model rather than a pay-per-review model seems better for researchers. A subscription model also seems better as a business model for TLDR/VitaDAO because it has lower friction. With subscriptions one only pays one time whereas with pay-per-review one would have to decide to pay each time they want another round of reviews.

Pay-in: Fiat. Most researchers who will pay for this service will not be Web3 natives. Thus, to achieve adoption it will be critical to facilitate fiat on-ramp. Donors will be more likely Web3 native, so BTC, ETH, and USDC are the initial coins that will be accepted. A range of fiat options (especially credit cards), provided by the well-known payment provider, Stripe, will be used for both paid submissions and donations.

Pay-out: VITA tokens. The specific amounts will be TBD depending on market conditions and what will make business sense to VitaDAO. Tentative amounts that are coded into the website are:

• 2 $VITA per upvote of a review. -2 $VITA per downvote of a review
• 1 $VITA per upvote of a reply to a review. -1 $VITA per downvote of a reply
• 1 $VITA per upvote of a manuscript that someone pays to have reviewed. There is no downvoting manuscripts, as we don’t want to discourage people paying to have their work reviewed.

The Web3 aspect of this project will grow over time. Besides receiving donations in BTC, ETH, or USDC and paying out in VITA, one can imagine formalizing the “proof of peer review” concept by making the reviews and voting on-chain. One can also imagine more tokens besides VITA to be used for payouts. For example, LabDAO’s $LAB token could be used. In that case, the content that could be submitted would have to be expanded beyond longevity.

Team: Tim Peterson, Rhys Anderson

Initial ask from VitaDAO: 20,000 $VITA tokens to have funds available to pay out to reviewers. This should increase our tokenholder base. We also plan to fundraise via Gitcoin’s GR15 happening this upcoming September. GR15 has a dedicated DeSci round, so this seems a good fit for creating awareness.

• Agree
• Disagree
• Agree with revision

[bowtiedshrike]

Strength of this proposal is this is a huge issue in peer review, and paying reviewers hopefully fixes some of this. I have many questions about the challenges associated with feasibility and implementation.

Feasibility questions
What is the success rate for preprints that get public comments vs ones that never do? Or put another way, what are the metrics that support the premise that posting comments on a bioRxiv manuscript will help its acceptance?

Would payments to reviewers run afoul of their employment restrictions? You might be able to classify it as an honoraria paid to reviewers, but med schools and especially intramural NIH can get cranky about people getting paid.

How would the ‘time based review’ model work? If it’s much longer than 2 months, it’s slower than a normal journal. If it’s less than 2 weeks, you’ll have a harder time finding quality reviewers.

How would this be billed to authors? I.e. will it be coded in such a way they can bill it to their grants as a publishing charge?

As proposed, this adds to an author’s cost to publish a manuscript without impacting existing revenue streams (ie all the money made by the journals). Why not launch a VitaDAO journal and fund reviewers via journal profits? Could even add a ‘perspectives’ section and ask VitaDAO funded projects to contribute a perspective/update/etc 1x per year. If it’s an invited review, it pads the author’s CV some, and gives a public, one stop update shop for the community.

How are you going to define longevity? Would organizing a bioRxiv channel on longevity be an important first step to drive reviewers to the papers VitaDAO wants reviewed?

How will expertise be determined? Will this be by name, or an NFT w/key expertise in the metadata that an anon can hold?

Who would maintain this program-- defining/ID longevity articles, dealing with reviewers, etc? Would this be a new working group?

Have you talked to anyone at Review Commons to see how that initiative is going? They have vague plans for a pricing model if the project works, but it’s not clear if that model is working or not. They might have stronger data than a twitter poll for pricing models (or might not), and they’re connected to some decent journals.

Questions about integration with existing efforts
Would this be done via the ‘public comments’ on bioRxiv?

bioRxiv also has some ‘solicit reviews’/‘publish in journals’ options, IIRC. Would the plan be to integrate with that?

Could the Ants-review smart contracts be forked and/or deployed to Arbitrum (for the love of gas, plz not mainnet… Synapse* has a new chain to help transfer tokens like Vita to Arb) for this effort? May avoid reduplicating effort on the web3 end of things.

Would partnering with EMBO (who handles the Review Commons editorial stuff) to compensate reviewers doing peer review on ‘longevity research’ (defined via biorXiv channel) via Review Commons be a better approach?

Along with Review Commons, ASAPbio is piloting 'crowd preprint review’. Another alternative to your approach might be to have VitaDAO collectively do a crowd peer review for papers it wants reviewed/service for authors which are then posted on bioRxiv. This might be a better way to pilot your idea, and it is easier to engage grad students/postdocs in peer review/reward contributors/encourage contributions to VitaDAO.

web3 challenges
A $Vita payout faces 2 challenges. First is that this won’t appeal to most peer reviewers because they’d need a MM wallet-- think you’d have to have a payout in fiat option to avoid limiting your peer review pool. A ‘social contract’ to lock tokens will decrease the pool even more.

Second challenge is the ‘growing the treasury’ concern. Even without reviewers dumping tokens, preferring fiat to $Vita means $VitaDAO is taking a put option on $Vita. On one hand, you can’t drive value to the token (it’s not a security!), so team shoudn’t focus on token price. On the other hand, the team betting against their own token is concerning. Or is the plan to pay out less in $Vita for expenses than is taken in for fiat? For researchers, that disclosure would be important since we don’t know how to value our time.

*full disclosure for Synapse: I have the bowtied bias towards Synapse vs other bridges, and previously helped them with biz dev work.

[timrpeterson]

Thanks @bowtiedshrike. How meta to receive such a thorough review of this proposal!

@bowtiedshrike

what are the metrics that support the premise that posting comments on a bioRxiv manuscript will help its acceptance?

Many are citing that getting feedback on their preprint adds to its appeal with journals. See this thread.

@bowtiedshrike

Would payments to reviewers run afoul of their employment restrictions?

Many academics are allowed to do paid consulting work. If an employer tries to restrict this highly aligned activity, one should question their policies.

@bowtiedshrike

How would the ‘time based review’ model work?

Manuscript submitters can stop payment at any time, therefore if they get helpful reviews within days then they don’t need to pay any longer. On the other hand, people puruse the literature at different rates. By keeping a manuscript paid longer, one can benefit from reviews for as long as one wants.

@bowtiedshrike

How would this be billed to authors?

We are using Stripe for payments. Their “Checkout” functionality allows multiple different payment methods used around the world.

@bowtiedshrike

Why not launch a VitaDAO journal and fund reviewers via journal profits?

This is the most controversial topic. A big part is having the work indexed in PubMed such that it counts for funding agencies. With our own journal, this wouldn’t happen for a long time. Don’t want to be dependent on an incumbent or 3rd party in order to succeed.

With COVID, NCBI has started to allow preprints to be posted to PubMed. It seems conceivable to think in some point in the future that if we and others (like Review Commons referenced below) provided the peer reviews for these preprints, then they could be listed on PubMed too.

@bowtiedshrike

How are you going to define longevity?

Keywords in the abstract and full-text. We can get more sophisticated over time. The paid submissions we’re thinking of restricting to Rxiv submissions to ensure quality.

@bowtiedshrike

How will expertise be determined?

We could track “.edu” addresses but this biases against the many people outside academia that will do a great job. Reviews will having voting, such that one can earn reputation as is done on many sites (Hacker news, Stackoverflow, etc.). NFT/crypto solutions can come in time, but seems overkill for a MVP.

@bowtiedshrike

Who would maintain this program-- defining/ID longevity articles, dealing with reviewers, etc? Would this be a new working group?

It definitely could. We’d love as much help as possible.

@bowtiedshrike

might have stronger data than a twitter poll for pricing models

It doesn’t appear Review Commons have implemented payments. Like with biomedical research, sometime there’s only so much you can hypothesize what will happen and you just have to do the experiment.

@bowtiedshrike

Would this be done via the ‘public comments’ on bioRxiv?

We can’t track who’s doing what and we won’t have quantitive data in order to make payouts unless we have our own site. Also, one doesn’t want to be entirely reliant on a 3rd party for one’s existence.

@bowtiedshrike

Would partnering with EMBO

Partnering with incumbents is not typically how paradigms shift.

@bowtiedshrike

A $Vita payout faces 2 challenges.

Payouts are largely TBD, but we are aware of the issues you raise. We can consider paying out in fiat if that’s what it will take to make this work. We can implement a social contract or even hopefully eventually a smart contract such that reviewers aren’t able to dump the VITA they receive.

Thanks again!

[timrpeterson]

@bowtiedshrike definitely happy to consider Synapse for inexpensive VITA payouts. Let’s talk!

@bowtiedshrike

Or is the plan to pay out less in $Vita for expenses than is taken in for fiat?

Yes, the plan is to be sustainable by balancing what is taken in with what is paid out. A main goal for POPR is to help fund research.

[bowtiedshrike]

Many academics are allowed to do paid consulting work. If an employer tries to restrict this highly aligned activity, one should question their policies.

I think NIH is the worst one, but medical centers sometimes have very restrictive policies due to the ‘pharma bribing physicians’ concerns. Some don’t officially let you participate in ‘buy 2, get 1 free’ promotions. Adding an opt-in or opt-out for payment would be the easiest way to include people stuck with these policies. Some of the grant agencies that pay for reviews do this.

We are using Stripe for payments.

I meant more along the lines of how the business would be classified from a Purchasing/accounting perspective. That may make or break if the grant can pay for it, and some universities have weird rules for using their credit cards.

Reviews will having voting, such that one can earn reputation as is done on many sites (Hacker news, Stackoverflow, etc.).

I think the crowd review idea would be stronger than this. Peer review should not be a popularity contest. If there’s low engagement, very easy to Sybil if you tell your friends to all like a review. If there’s high engagement, is that from grad students, the community, or people with the relevant expertise?

Part of what makes lower ranked journals (I mean MDPI-level here, not JBC) lower ranked is the quality of the articles and the quality of the peer reviewers.

Another option might be to leverage a scientific society or two. The program would benefit from the status of partnering with the society and get experts vetted by the society, while the society could benefit by added exposure to revolutionary programs, web3 and public good.

one doesn’t want to be entirely reliant on a 3rd party for one’s existence.

How do the reviews get shared with the authors? Is this intended to be an informal ‘get an extra pair or two of eyes on your work’ review, or will the reviews ultimately be posted to biorxiv? If they do go to biorxiv, would it be via the ‘public comments’ or would the longer term plan be to integrate into one of the other reviewer tabs?

It doesn’t appear Review Commons have implemented payments.

Right. Their roadmap is to pilot their service using their grant funding for the first bit, decide if the program is worth it, and if so, how much to charge. If they’re willing to talk about how it’s going, that would be useful information for this initiative.

We can consider paying out in fiat if that’s what it will take to make this work. We can implement a social contract or even hopefully eventually a smart contract

While over ambitious for version 1 of this program, building an easy-to-use non-custodial wallet into the system might help on-board participants into web3 and reduce fiat payouts if the system makes it easier for them to hold $Vita and/or swap for Eth.

[timrpeterson]

@bowtiedshrike

Adding an opt-in or opt-out for payment would be the easiest way

It’s a great idea. We can implement this.

@bowtiedshrike

how the business would be classified from a Purchasing/accounting perspective

Haven’t decided whether this a non-profit or for-profit venture. There’s pros and cons to both. Open to hear your thoughts…

@bowtiedshrike

crowd review idea

How do you envision “crowd review” voting? The way it’s set up now can’t be a popularity contest because no names are displayed on any of the content. We specifically wanted to avoid people associating content with usernames.

@bowtiedshrike

How do the reviews get shared with the authors?

Each manuscript will have all reviews posted to a dedicated manuscript page. It will be similar to most internet post/commenting sites. We can’t force BioRxiv to accept them.

@bowtiedshrike

they’re willing to talk about how it’s going, that would be useful information for this initiative.

Agreed. This is a decentralized effort so if you would like to help in this outreach it would be much appreciated.

@bowtiedshrike

building an easy-to-use non-custodial wallet into the system

Another good idea! We certainly need a dashboard for reviewers on how much they are making, so enabling a custody solution is a great idea for non-crypto people.

[bowtiedshrike]

Haven’t decided whether this a non-profit or for-profit venture. There’s pros and cons to both. Open to hear your thoughts…

If you’re willing to go through the IRS exemption stuff, non-profit makes everything donated tax deductible. If the mission is advancing longevity research, you could likely justify investing any excess beyond review costs back into research projects.

Closer to launch, it would be good to articulate how it should be coded --as a publication charge, as a service, etc and check NIH allowable costs to ensure it’s easy to spend grant funds on the service.

How do you envision “crowd review” voting?

From what I understand of ASAP bio’s crowd review proposal, the manuscript is reviewed by groups instead of individual peer reviewers. So 2-4 people would provide comments (internally) over a period of time, and then the comments are discussed/synthesized into a final review that is shared with the authors. Could either involve a non-scientist to try to summarize/synthesize the reviews, or have a lead reviewer do it. Push for consensus, or have a ranking system. The report would have ‘crowd review 1’ ‘crowd review 2’, but within each crowd, reviewers would know who they were discussing with.

This would be a way to get trainees and some of the lay longevity people involved, while ensuring there is expertise within the review group. There are also some training options for peer review.

Each manuscript will have all reviews posted to a dedicated manuscript page. It will be similar to most internet post/commenting sites. We can’t force BioRxiv to accept them.

But they can be posted there as ‘comments’ at the least, right? That would help visibility. And once established, you might be able to auto-transfer using the “community review” tab or the TRiP tab. There’s also a metric for ‘blog/media’ links.

[ry0kawasak1]

Great proposal which has a potential to revamp the peer-review system once again. I totally agree to introduce incentive to the reviewers, but flip-side of this is how to secure the “peer” is qualified to review. I think this is now universal issue given the number of journals are growing.
Ideally, some sort of selection or designation of qualified reviewers would be a key to make this system successful. Like Peer-review of the reviewers?? Otherwise, low quality review by peers who just want to earn could be happen…

[timrpeterson]

@ry0kawasak1

Like Peer-review of the reviewers??

Exactly! The solution i’m thinking is to enabling voting on the reviews.

[timrpeterson]

several good suggestions. Thanks again!

[longevion]

Any ideas on sybil resistance? Gitcoin I think is pretty good at it.

For both payments and donations we can easily allow people to choose between fiat and crypto (including VITA).

[longevion]

I suggest we continue the conversation here unless there are objections (even though the proposal itself went on chain).

Especially around the developments with the exodus of the entire editorial team at Aging Cell.

We have a narrow window to make a splash with a new version with the same editorial team and get a jump on the chicken-and-egg problem of launching a new journal.

See the discussion in the discord #new-journal channel.

[timrpeterson]

IMO we need to focus on adoption before focusing on Sybil resistance.

but yes, once we get some traction we should implement an already successful approach.

[VitaDAO]

This proposal has successfully passed!