Choose a rapid-review screening workflow before choosing its software

1. Establish the method and its intended use

Cochrane’s 2024 rapid-review recommendations address reviews of health-intervention effectiveness. Their discussion of piloting and reviewer arrangements should be read within that scope, not applied as a universal recipe for every diagnostic, qualitative or other review. A shorter deadline does not remove the need to explain the chosen method and its limitations.

Write the question, eligible designs, population, setting, outcomes and report types before importing records. Identify any proposed restrictions, their rationale and who agreed them. A decision to use one screener for part of the work differs from independent duplicate screening. Software with two user accounts does not establish that two people actually made independent decisions.

Reference: Cochrane rapid review methods guidance, 2024

2. Test understanding with difficult cases

The following miniature exercise is deliberately too small to validate a screening method. Imagine six invented title-and-abstract records assessed independently by two trainees. They agree on four decisions. On one other record, they disagree about whether a mixed adult-and-child population meets the scope; on the last, they disagree about a report that describes implementation but supplies no effectiveness outcome.

Retain the original labels, identify the wording behind each disagreement and resolve it through the agreed review procedure. If the eligibility definition changes, identify previously screened records that need rechecking. Four agreements out of six teaching cases should not be presented as sufficient calibration, a reliable agreement estimate or evidence that single screening is safe for the real review.

On small screens, scroll the table horizontally, or focus it and use the arrow keys.

3. Understand what the prioritization tool actually does

The stable ASReview documentation describes a screening process in which a person marks records relevant or irrelevant and the model reorders the remaining material using those labels. It also documents notes and changes to earlier decisions. These are software capabilities described by the provider, not results of a product test performed for this guide.

A high-ranked record is a candidate to inspect, not an included study. A low-ranked record may still be relevant. For a proposed implementation, document the software version, inputs, starting labels, screening stage and how human decisions are reviewed. Check which local label corresponds to the review’s eligibility process; a tool’s “relevant” label need not mean that full-text assessment is complete.

Reference: ASReview stable documentation: screening

4. Keep unseen records visible in the export

Consider a separate fictional dataset containing 100 distinct records. Its supplied export marks three as relevant and nine as irrelevant; 88 have not been screened. Three + nine + 88 = 100, while only twelve records have human labels. The 88 unseen records cannot be added to the nine exclusions just because the interface displays a complete ranked list.

ASReview’s export documentation distinguishes label 1, label 0 and a missing label for unseen records. Preserve that distinction when importing an export into a spreadsheet. Keep the untouched export, decision history and reasons in controlled storage. Check exported notes and user fields before wider sharing; a screening file can contain information beyond the public bibliographic record.

Reference: ASReview: progress, stopping and export

5. Decide how stopping will be justified

The ASReview progress documentation leaves the stop-or-continue decision to the user and describes monitoring consecutive irrelevant records. A run of irrelevant labels is not, by itself, proof that nothing relevant remains. Agree how the team will review its stopping decision, what evidence it needs and how any unseen records will be represented in the report.

If resources end before the planned method is complete, record the actual stopping point and limitation. Do not relabel unscreened material as excluded or describe a software recommendation as independent scientific approval. The fictional 100-record set remains unfinished; neither its order nor its twelve labels demonstrates that the review question has been answered.

Reference: ASReview: progress, stopping and export

6. Report the performed human and automated work

PRISMA 2020 asks authors to describe the selection process, reviewer arrangements and the role of automation when used. For the applicable review, explain who screened which stage, whether decisions were independent, how disagreements were handled and how prioritization affected the process. Report the actual method and deviations rather than reproducing a default software description.

Before handover, reconcile the counts and sample decision records against their sources. Separate records, reports and studies; preserve unresolved retrieval and eligibility states. A coordinator should be able to reopen the controlled dataset, identify the next record requiring work and understand why it has not been completed. That is a practical test of traceability, not certification of the review’s findings.

Reference: PRISMA 2020: expanded reporting checklist

Questions for a practical final check

• Guidance scope matches the review type.
• Eligibility and proposed restrictions are written.
• Calibration disagreements preserve their reasons.
• Machine ranking is separate from human decisions.
• Unseen labels remain distinct from exclusions.
• Stopping and automation are reported as performed.