OJS Peer Review Quality: 7 Common Mistakes Journal Editors Make (And How to Fix Them)

The quality of your peer review process is the single most consequential factor in your journal’s long-term impact — more than your submission volume, your open access policy, or even your editorial board’s name recognition. Yet most editors running OJS peer review quality processes fall into the same seven traps, almost invisibly, because peer review quality is hard to measure until it starts to fail. This guide identifies each mistake precisely and gives you a concrete fix for each one.

Why Peer Review Quality Determines Journal Impact

Peer review serves two distinct functions that are easy to conflate. The first is quality filtering — catching errors, methodological weaknesses, and unsupported claims before publication. The second is developmental — helping authors strengthen work that is fundamentally sound but not yet publication-ready.

A journal that does neither produces low-quality published articles. A journal that does the first but not the second develops a reputation for harsh, uninformative reviews that drives away good authors. A journal that does the second but not the first publishes papers that later attract post-publication criticism or correction.

The journals that build citation impact over time do both consistently — and that requires an intentional peer review management system, not just good intentions. OJS provides the infrastructure, but the process has to be designed deliberately.

For context on how peer review quality connects to indexing applications, our guide on OJS editorial quality services covers the full spectrum. See also our post on reducing bias in peer review for evidence-based approaches to reviewer selection.

Mistake 1: Not Qualifying Reviewers Before Adding Them to Your Pool

Many OJS journals accumulate reviewer databases by accepting self-nominations, importing names from reference lists, or using whoever has reviewed for the journal before. This creates a pool where reviewer expertise is inconsistently documented and hard to match to specific submissions.

The fix: Implement a reviewer registration form that captures specific research areas using controlled vocabulary (ANZSRC codes, JEL classifications, or your field’s standard taxonomy), publication record (ask for three recent DOIs as expertise evidence), and declared conflicts of interest. In OJS, this data belongs in the reviewer profile — not just in a spreadsheet that someone maintains separately.

See our detailed guide on building and managing a reviewer database for the complete setup process.

Mistake 2: Assigning Reviewers Without Checking for Conflicts of Interest

Conflict of interest checking in OJS journals is often limited to a checkbox that reviewers self-complete when they accept an invitation. This is insufficient. Reviewers frequently do not recognise conflicts that editors can identify from context — institutional collaboration, shared authorship in the past five years, documented professional disagreements, or competitive research programs.

The fix: Before assigning any reviewer, an editor should check: the reviewer’s recent publication list for co-authorship with the submitting authors, institutional affiliation overlap, and whether the reviewer has recently cited or been cited by the authors. This takes under five minutes per assignment and eliminates the most common, damaging bias sources. OJS does not automate this — it requires editorial discipline.

Mistake 3: Using Unstructured Review Forms

An open text box labelled “Comments to Author” produces reviews that range from four sentences to fourteen pages, cover entirely different aspects of the paper, and are almost impossible for editors to compare when making accept/revise/reject decisions.

The fix: Build a structured review form in OJS (Settings → Workflow → Review → Review Forms) that prompts reviewers for specific assessments: novelty and significance, methodological rigour, clarity of presentation, appropriateness of conclusions, and overall recommendation. Each section should have a rating scale and a required text field. Structured forms produce more consistent, actionable reviews — and make editors’ synthesis decisions dramatically faster.

Mistake 4: Setting Unrealistic Deadlines Without a Reminder System

The most common cause of slow peer review in OJS journals is not unwilling reviewers — it is no systematic reminder process. Editors send an invitation, set a deadline in OJS, and then wait. When the deadline passes, nothing happens automatically unless someone remembers to follow up.

The fix: OJS has a built-in reminder system (Settings → Workflow → Review → Review Reminders). Configure it to send an automated reminder three to five days before the due date, and another on the due date itself. Set a second-tier follow-up seven days after overdue — at which point an editor should personally contact the reviewer. Track your average review completion time per reviewer and retire persistent late-submitters from your active pool.

Mistake 5: Not Calibrating Your Reviewer Pool for Quality

Most OJS journals know their reviewers by name and field. Very few know their reviewers by review quality. This matters because a reviewer who consistently produces three-sentence reviews, who always recommends acceptance regardless of paper quality, or who uses review comments to advance their own research agenda is actively damaging your journal’s peer review process.

The fix: Implement a simple reviewer quality scoring system. After each published paper, rate the reviewers who assessed it on a three-point scale: Did the review identify the significant issues? Was the recommendation consistent with the paper’s actual quality? Were the comments useful to the author? Scores do not need to be elaborate — just consistent. After three or four reviews, you have enough data to preferentially invite high-quality reviewers and deprioritise those who are not adding value.

Mistake 6: Treating All Submissions with the Same Review Depth

A 2,000-word methodology note and a 10,000-word original research article with novel findings require fundamentally different reviewer assignments and timelines. Yet most OJS journals process both through identical review workflows with identical deadlines and identical invitation letters.

The fix: Segment your OJS sections by article type and configure different review settings for each. Short communications can often be reviewed by two reviewers with a two-week deadline. Full research articles warrant three reviewers and three to four weeks. Meta-analyses and systematic reviews may need specialist reviewers and a six-week window. OJS supports different review settings per section — use that flexibility.

This also connects to how you structure your publication frequency. See our post on publication frequency and workflow for how review cycle length affects your publishing cadence.

Mistake 7: Not Recognising and Retaining Your Best Reviewers

The best peer reviewers are in demand across multiple journals in their field. They do not need to review for yours — and they will quietly stop doing so if the experience is poorly organised, if their reviews are ignored, or if they never receive acknowledgement of their contribution.

The fix: Send a personal thank-you email for every completed review — not an automated OJS notification, an actual personal note from the editor. Publish an annual reviewer acknowledgement list (with permission). Offer reviewer recognition certificates, waive article processing charges for one submission per year of active reviewing, or provide early access to published articles. These cost almost nothing and dramatically improve reviewer retention.

For journals that are also considering the broader landscape of AI in scholarly publishing, note that AI-assisted reviewer matching is becoming viable — but the human relationship with your reviewer community remains irreplaceable.

Building a Better Peer Review System in OJS: The Practical Summary

Addressing all seven mistakes does not require a complete system overhaul. It requires a structured improvement plan implemented over two to three months:

1. Month 1: Rebuild your reviewer database registration process; implement structured review forms; configure OJS reminder automations
2. Month 2: Establish a COI checking protocol; segment review settings by article type; launch reviewer quality scoring
3. Month 3: Implement reviewer recognition programme; conduct a retrospective review of past three months’ review quality data

If your editorial team is small or you are launching a new journal that needs a professional-grade review process from day one, NeuCitePress offers OJS peer review management services covering reviewer recruitment, assignment, quality monitoring, and workflow optimisation.

Frequently Asked Questions

How many reviewers should each OJS submission receive?

The standard is two reviewers for most submissions, with a third added when the first two recommendations conflict significantly. For high-stakes research (clinical trials, meta-analyses, statistical methods papers), three reviewers from the start is advisable. One reviewer is rarely sufficient for anything beyond short correspondence.

What is a reasonable turnaround time for peer review in OJS?

Four to eight weeks from submission to first decision is competitive for most disciplines. Journals that exceed twelve weeks consistently lose authors to competitors. Track your average time-to-first-decision and make it visible in your journal’s author information — it is a selling point if it is good.

Should I use open, single-blind, or double-blind review?

This depends on your field’s norms and your journal’s positioning. Double-blind is standard in social sciences and humanities. Single-blind is more common in biomedical and physical sciences. Open review (where both parties know each other’s identities) is growing in some communities. Whatever you choose, document it clearly and apply it consistently — inconsistency is more damaging than any particular choice.

How do I handle a situation where a reviewer’s report is clearly biased?

If a review appears motivated by personal or competitive animus rather than scientific critique, do not use it. Invite a replacement reviewer, and if the author has already seen the review, acknowledge that you are seeking additional expert input. Document your reasoning internally. See our evidence-based guide on reducing bias in peer review for systemic approaches.

References

• Tennant, J. P., et al. (2017). The academic, economic and societal impacts of Open Access: An evidence-based review. F1000Research, 5, 632. https://doi.org/10.12688/f1000research.8460.3
• Lee, C. J., Sugimoto, C. R., Zhang, G., & Cronin, B. (2013). Bias in peer review. Journal of the American Society for Information Science and Technology, 64(1), 2–17. https://doi.org/10.1002/asi.22784
• Squazzoni, F., et al. (2021). Peer review and academic publishing in the digital era. PLOS ONE. https://doi.org/10.1371/journal.pone.0251784