In this guest post, Michael Casp and Anna Jester look back at the ALPSP 2022 Annual Conference and Awards. 

 

https://www.wiley.com

Peer review in the Digital Age relies

heavily on email communications and web-based peer review platforms to contact reviewers and organize

recommendations. In many ways, peer review today is just a recreation of the pre-internet

mail, fax, and file cabinet template, but pasted online.  With the current advancements in preprints,

social media, and communication platforms, it is possible – even likely – that

the model of peer reviewing, and the technology that supports it, have more

evolutions to come.

 

As communication begins to move beyond

traditional text formats, so does content. Getting “beyond the PDF” has been a

staple at scholarly publishing conferences for years, and as it becomes more

commonplace, we are navigating its demands on the peer review process.

 

But it’s not just about technology. We must

also focus on developing and empowering the next generation of reviewers in

order to maintain a robust and sustainable reviewer pool. This happens through

teaching and developing early academics, which can offer a technology-versed,

diverse pool of reviewers that can continue to utilize the technologies we

develop.

 

At the ALPSP 2022 Conference we were treated to many innovative ideas that will

have a direct impact on peer review, potentially changing it and hopefully

improving it, for the researchers and publishers and ultimately providing

better value for society.

 

Beyond Email

 

Peer review’s reliance on email is a given.

That is, unless you’re in China, where the app WeChat has in many ways

supplanted email as the default communication system. Western publishers that

seek to engage with Chinese researchers might struggle if they only use email.

However, Charlesworth presented the ALPSP audience with a possible solution

with their new product, Gateway. Gateway

uses an API to allow journal-related communications

to be sent to authors, reviewers, and editors via WeChat. This solution allows journals

to meet Chinese researchers where they are, rather than trying to pull them

into an (let’s face it, antiquated) email system.

Given this shift, eLife presented the ALPSP

audience with Sciety, a new

preprint evaluation platform that allows academics to review, curate, and share

preprints and reviews with their peers. Preprint servers have also started to

become social hubs for researchers to connect in more of a real time

environment than traditional publishing tools. This system holds the promise of

opening up peer review and publishing to a wider user base, allowing more

people to curate and review research than ever before. The challenge presented

by the scale of preprints is immense, and Sciety has the potential to

reorganize how we deal with all this research in a social-focused way.

 

One more data point to mention: with the pervasive

use of social media throughout the world, it is no surprise that academics

would have their own version. Despite controversies,

ResearchGate has maintained its

position as the largest academic social network, connecting about 17 million

users. With this many scholars connected, it’s possible we could see something

like peer review networks emerge, though that doesn’t seem to be ResearchGate’s

focus at the moment.

 

Beyond the PDF

 

A decade or so ago, getting “beyond the

PDF” was still a new idea being speculated about at conferences. It is now a

reality, with authors providing data sets, code, and detailed methods notebooks

alongside their traditional manuscripts. As a partner to those authors, we’ve

come up with ways to publish this content, but it can present special problems

during peer review.

 

For starters, journals that employ an

anonymized review model can find it quite difficult to extract identifiable

information from complex content like code or reproducible methods. Sometimes

an author’s institution might be inextricably linked to this content, making

anonymization impossible – or at least impractical.

 

Other forms of content, like ultra-high-resolution

images, can present logistical problems. Industry standard peer review

management systems have hard limits on the size and format of files they can

manage. For example, fields like pathology can rely on extremely detailed

images of microscope slides, and these multi-gigabyte files are hard to move from author, to

editor, to reviewer. Paleontology research can also require larger-than-usual

images, as sharing highly detailed photos of artifacts is crucial to the field.

Dealing with these kinds of challenges at the peer review stage can require a

lot of creativity and patience for all involved without a more flexible

solution.

Massive datasets can also present review

challenges. Beyond the logistics of moving large

files, there are also often more basic concerns, like is this data organized

and labeled in a useful (and reusable) way? Is it actually possible to do a

real review of a large dataset in the time that reviewers have to give to a

paper? FAIR data principles

are targeted at answering some of these questions, and services like Dryad and Figshare

seek to help by curating and quality controlling

datasets, ensuring they meet basic standards for organization, labeling, and

metadata. But these services come with an additional cost that not everyone can

bear. And a data review still depends on a reviewer willing to go the extra

mile to actually review it.

 

Moving peer review beyond the PDF is still

a work in progress, but many of these are solvable problems as our technology

and internet infrastructure improve. Our J&J Editorial

staff regularly handle content like videos, podcasts, and datasets. At EJournalPress’, our platform is integrated

with third parties including Dyrad, Code Ocean, and Vimeo. These integrations

are an added convenience, as most journals and societies have to have direct

agreements with third parties for the integrations to be fully utilized. But we

often have to work around the peer review system, rather than with it, relying

on basic cloud file servers (e.g., Dropbox or OneDrive) instead of more

purpose-built technology.

 

Open/Pre-submission Review

 

Another decade-old conference trope was the

constant talk about new open peer review models. You might recall that people

were split on the wisdom of this approach, but the rise of preprints has done a

lot to push open peer review and pre-submission review into the limelight.

 

Organizations like Review Commons are working with EJournalPress to make pre-submission and

peer review a viable choice for authors by building review transfer

relationships with more traditional journals. The Review Commons model is to

take preprint submissions and have them peer reviewed. These reviews can then

be shared with participating journals if authors choose to submit. Journal editors

can use the existing Review Commons reviews to evaluate whether or not to

publish the work. In data presented at ALPSP 2022, manuscripts that came into

journals with reviews rarely needed to be sent out for further review. This has

many benefits, saving the editor time in soliciting reviews, and giving a

journal’s (probably over-taxed) reviewer pool a little break.

 

Review Commons is currently free for

authors, being supported by a philanthropic grant. It will be fascinating to

see if they are able to pivot towards a sustainable financial model in the

future.

 

We won’t exhaust you with the long list of

other open peer review initiatives, but suffice it to say, a lot of smart people

are working hard on making this a standard part of research communication.

 

Developing the Next Generation of

Reviewers

 

None of what we’ve written so far will

matter one iota if there aren’t enough people in place to do the actual content

reviews. One of the interesting revelations we had while managing journal peer

review was the incredible range that exists in review quality. From the

in-depth, multi-page discussions of every point and nuance of an author’s

manuscript, to the dreaded “Looks good!” review, anyone in peer review can tell

you that we can (and must!) do a better job training our reviewers. We can

offer guideline documents and example reviews, but some people need a more

engaging approach to understand and deliver what editors expect.

 

It would be lovely if reviewing was a

required part of the science curriculum. It currently seems to happen in a

piece-meal, ad hoc fashion, many times driven by people who are willing to just

figure it out themselves. A more standardized approach is called for,

especially as reviewable content becomes more complex and varied.

 

One of the best examples we’ve seen of

reviewer training was actually a writer’s workshop for researchers wishing to

submit to a medical journal. The EIC of the journal

led this workshop, asking authors to submit their manuscripts ahead of time to

serve as examples for the workshop. During the workshop, the EIC talked through

several of these manuscripts, giving the authors invaluable feedback and what

amounted to a free round of review prior to the official journal submission.

 

Though this workshop was ostensibly for

authors, it was equally valuable for reviewers. Participants got to watch the

EIC go through a paper in real time, ask questions, pose solutions, and talk

through the subject matter with someone who had written hundreds and reviewed

thousands of manuscripts. This program has always stuck out as a great way to

train authors and reviewers, while also building the relationship between the

journal and its community. Win-win-win!

 

Formal peer review training benefits all

parties, such as the Genetics Society of America’s program which also includes direct feedback from

journal editors. If you’re thinking of implementing something like this, your

organization may wish to conduct a pilot prior to a full rollout. Another great

model for peer review training is to pair mentors and mentees to simultaneously

provide training and increase the number of high quality trained peer reviewers in the field broadly. If

your team is willing to study the results of your reviewer training efforts, be

sure to submit them to the next International Congress on Peer Review and Scientific Publication so that we can all

benefit from your findings.

 

Demographics and Diversity

 

Many of the journals and publishers we work

with are prioritizing diversity within their community by making efforts to

extend their reach to people who might have been historically left out of the

conversation. These organizations are also looking inward to seeing what their

current level of diversity looks like in order to improve it.

 

Many organizations have begun collecting

demographic information regarding their authors, reviewers, and editors. We

recommend a thoughtful approach when embarking on this project, as it can be

fraught with pitfalls and unexpected consequences if you don’t get it right.

Before your organization embarks on this endeavor, please consider best practices regarding data collection and clearly define the initiative’s

goals. Wondering where to start? Do a landscape scan of what other organizations aim to do with this data and please use standardized questions for self-reported diversity data collection.

 

Fortunately, many people are working on

demographics data initiatives, and there is lots of support and ideas available

from our community.

 

Summary

 

To put it mildly, there is a lot going on

right now. The technology we use has the potential to upend the traditional

research communication process, and in some cases (like preprints) it already

has. With a host of new data, content, and equity concerns, people involved in

the peer review process have more to deal with than ever before. And it’s unlikely

that we’re doing enough to equip them with the knowledge and training they need

to succeed. But we can do better, and I’m heartened to see the many people in

and around our industry who are trying to improve the situation. From our end, eJournalPress is supporting societies

and journals as they work to collect and evaluate demographic information and

metadata, and J&J Editorial staff

are always investigating ways to support journal innovations through a

combination of technology and experienced staff.

 

I often think about peer review in the

context of that old Churchill quote about democracy: “It has been said that

democracy is the worst form of government, except all of those other forms that

have been tried from time to time.” Peer review might not be the best method of

scientific evaluation, but it’s the best we’ve got, and who knows, maybe we’ll

make something even better. But until then, we’ve got work to do.

Anna Jester, VP of Marketing and Sales, eJournalPress, Wiley Partner Solutions

Michael Casp, Director of Business Development, J&J Editorial, Wiley Partner Solutions

Wiley Partner Solutions was gold sponsor of the ALPSP Conference and Awards held in Manchester UK in September 2022.  The 2023 ALPSP Conference will be held in Manchester from 13-15 September 2023.

Wiley is one of the world’s largest publishers and a global leader

in scientific research and career-connected education. Founded in 1807, Wiley

enables discovery, powers education, and shapes workforces. Through its

industry-leading content, digital platforms, and knowledge networks, the

company delivers on its timeless mission to unlock human potential. Visit us at

Wiley.com. Follow us

on Facebook, Twitter,

LinkedIn

and Instagram.