Compare transparent peer-review practices before planning a pilot

1. Compare separate transparency decisions

Start with a diagram of the proposed publication record. Identify the manuscript version, editor’s letter, review report, author response, attachments and identity information as separate objects. Decide which remain private and which might become public under an adopted policy. A report can be public without a reviewer’s name; publishing a name does not necessarily make the complete decision history available.

For each object, record the person responsible for permission, the stage at which the choice is made and the correction route after publication. Specify whether the choice applies to all articles or only a defined category. Without that detail, a comparison of two “transparent” models may combine very different recipient experiences and consent arrangements.

2. Read two current examples within their scope

PLOS One offers authors the option to publish decision letters, reviews and author replies with an accepted article. A reviewer’s name appears if the reviewer signed. The policy also allows exceptional withholding for ethical considerations. This describes an available publication practice, not a measured effect on review quality.

eLife selects submissions for review first. Articles it reviews become Reviewed Preprints with Public Reviews, an assessment and an author response when available; there is no post-review accept-or-reject decision in that model. Authors may later revise or choose a Version of Record. Selection for review remains a distinct stage.

The examples differ in publication pathway and decision architecture, so their outcomes cannot be compared by treating their article counts as interchangeable. A small journal can study a limited feature, such as permission handling for reports, without claiming to reproduce an entire publisher’s model. Read the destination’s current instructions again before acting; this is a dated policy snapshot.

Reference: PLOS One: editorial process, review history and appeals · eLife: peer review and publishing model

On small screens, scroll the table horizontally, or focus it and use the arrow keys.

3. Prepare permission and sensitive-content handling

ICMJE’s confidentiality guidance requires permission before publicizing reviewer comments and protects an unsigned reviewer’s identity. Use the applicable policy and permissions for the particular records. Prior participation in a confidential process should not be treated as automatic agreement to a new public display.

A proposed pilot needs an owner for checking third-party material, personal details and confidential content in attachments. Define what happens if a participant declines, if a file cannot safely be released, or if a published record needs correction. Keep the reason for withholding distinct from a missing file. A blank public slot should not be filled with an invented review or a summary presented as the original report.

Reference: ICMJE: submission and peer-review responsibilities

4. Calculate a fictional pilot without changing denominators

Suppose an imaginary journal offers a proposed review-history pilot to 12 eligible accepted articles. Eight author teams opt in. At the planned checkpoint, five of those eight histories are cleared and posted; three remain pending because attachment permissions have not been resolved. These numbers describe an invented workflow, not Lumora activity or a completed trial.

The opt-in proportion is 8/12, approximately 66.7%. The posted proportion among opt-ins is 5/8, or 62.5%. The posted proportion among all eligible articles is 5/12, approximately 41.7%. Each answers a different question. None is a review-quality score. The three pending records must remain visible in the operational record instead of being counted as posted or silently removed from the denominator.

5. Design the evaluation before collecting results

Choose an operational question the journal can actually answer. For example: can the team match every released report to the correct manuscript version and documented permission? Define a sampled inspection, responsible reviewer, correction process and the records needed to resolve a mismatch. Report both successful matches and unavailable evidence. A test of document handling should be labelled as such.

If the question concerns review quality, define a separate method and relevant outcome before examining the reports. Explain who evaluates them, what information evaluators receive and which comparison is justified. Do not use an increase in visible files as a substitute for that evaluation. Likewise, a publisher’s stated rationale for transparency is not itself evidence of a measured benefit.

6. Recognize a defensible reason to defer

A journal may need to defer a proposed feature when it cannot preserve permissions, link versions accurately, manage corrections or protect material that must remain private. Document the missing capability, its owner and what would count as readiness. Deferral should lead to a concrete preparation task, not an unsupported claim that the feature is already available.

Before any real launch, review the final participant information, file display and operational responsibilities with the authorised editorial team. Announce only the practice that has actually been adopted and implemented. Keep a dated policy page and a correction contact accessible to readers. The purpose of transparency is served by an understandable, accurate record of what occurred and what remains uncertain.

Questions for a practical final check

• Public objects and identities defined separately.
• Current policy and article scope recorded.
• Permissions tied to the actual files.
• Pilot counts retain their denominators.
• Operational and scientific outcomes distinguished.
• Unready capabilities and withholding reasons recorded.