The peer-review system has been around as part of the medical scene since the early 1800s. While the peer review system has its fair share of detractors it is still highly regarded in most circles as the best method we have for evaluating academic work.
So, what is a peer review system, and what should it be included to ensure you are delivering a paper or abstract that has the best chance of success?
Why are Peer Reviews Important?
A peer review system is a quality control mechanism that ensures the material presented at a conference or event will deliver material that is suitable for the intended audience. They also assess the validity of the information contained in the paper.
How Does the Peer Review System Work?
Any scholarly work that is to be presented or published must first undergo a preliminary check referred to as a desk review. The editor reviews the document and makes recommendations, which could be to reject it immediately or send it on to experts with the relevant knowledge and experience to make a more thorough examination.
These experts are then tasked with thoroughly evaluating the document and determining if it fits within the conference, event, or publication scope.
Reviewers will also determine if the research covers the main topics adequately and an appropriate approach has been taken when addressing the issues that need to be covered.
Peer reviewers may also deliver feedback on how a paper can be improved and information about the general quality of the work. However, if the reviewers consider the paper or abstract doesn’t sufficiently meet the guidelines, they will send a letter of rejection with a clear explanation outlining why they came to that conclusion.
Authors are then free to make the necessary changes and resubmit their paper or abstract.
Different Types of Peer Review Systems
Peer reviews help organizers know when to accept or reject papers or abstracts submitted for a conference or presentation.
Single or double-blind reviews were the most popular formats in use until fairly recently, where authors were kept in the dark about who the reviewers were.
Now, there are several more options available, including:
- Single-Blind Review
- Double-Blind Review
- Open Review
What is a Single-Blind Peer Review?
In a single-blind review, reviewers’ names are not revealed to the authors. However, the names of the authors and their affiliations are listed on the papers.
The single-blind peer review system is the traditional version and is still the most common in use today. The main reason behind the anonymity of reviewers is that authors are unable to contact them to influence the results. Reviewers can also feel more confident in their criticisms because they are protected by their anonymity.
What is a Double-Blind Review?
Both the author and the reviewer are anonymous during a double-blind review. The system was developed to eliminate bias on both ends. It prevents reviewers from being influenced by an author’s prestige or notoriety, and authors from contacting reviewers to influence their review.
Unfortunately, keeping author identities anonymous is incredibly difficult when they are prominent in their field. Their work is easily identified through their content and writing style. Plus, most will reference previously published works.
Open Peer Review
The open peer review system is an alternative approach that creates more transparency between reviewers and authors. Both authors and reviewers are aware of each others’ identities. One of the main advantages is that reviewers who are notorious for delivering not-so-constructive criticisms are exposed to all, which is a huge incentive for them to be a little more exhaustive with their feedback.
What Makes a Good Peer Review?
It can be challenging to know what makes a good review versus a bad review, especially for reviewers with little to no experience. Here are a few guidelines on how to create a high-quality review.
Is It Focused?
Every submission is different, which means they cannot all be reviewed in the same way. When examining a proposal, it’s always a good idea to have a set of standard questions you ask yourself to help you keep your focus. For example, keep these ideas in mind as you go through the material:
- Are the findings robust? If not, what needs modifying?
- Are the results unique when compared to other research in the field?
- Is there a better way for the authors to deliver their ideas?
Keeping these concepts at the top of your mind will help highlight strategies you can convey to the author to improve on their work or ensure the material does not deviate from the scope of the conference.
Deliver Constructive Criticism?
All good reviews will contain a certain level of criticism. This is fine provided they are delivered constructively and include ideas about where improvements can be made. Problems should be clearly identified and followed with suggestions for rectifying them. The reviewer should also state why they came to a particular conclusion.
Be Precise
Accurately identify the location of mistakes or possible improvements using page and line numbers.
Be Polite and Professional
Author’s will not always look on a negative review favorably, but the blow can be softened somewhat when you keep your comments fair, professional, and justifiable. This isn’t to say you should avoid criticisms whenever warranted, especially when you recommend a rejection of the abstract or paper. In short, avoid mean comments, rants, or personal attacks on the author.
Ask Someone Else to Review Your Review
You can ask a colleague or fellow reviewer to provide an unbiased opinion about our review. Get them to go over your work without reading the paper so they can give you feedback while remaining as neutral as possible. If getting someone else to go over your review is not possible, you could try leaving it for a day or two before re-reading it.
What Not to Include in a Peer Review
Knowing what not to include in a peer review can be just as helpful as knowing what to include, so here are a few things to leave out.
Contradictions
Some reviewers will leave glowing comments about the paper the author can read but will provide their editor with contradictory statements in the confidential section. The editors may need to pass these notes to the author when delivering feedback. Reviewers might do this to prevent hurt feelings on the author’s part, but it is not a good practice. Contradictory statements are confusing at best and can often lead to time-consuming (and justifiable) appeals.
Vague or Ambiguous Comments
Always strive for clarity in your peer review. For instance, the comments ‘needs work’ or ‘misses the mark’ are entirely unhelpful. These statements don’t provide any insight or guidance on what the author needs to do to improve their submission.
Mean or Callous Comments
Mean, or insensitive comments are unnecessary and are one of the reasons we need anonymity in peer review systems in the first place. Feel free to rant about a paper if you must, but maintain your professionalism by leaving it out of the abstract or paper. All of your comments should keep an air of professionalism and lack of bias.
Finding an Appropriate Peer Review System
When you are the recipient of a large stack of papers or abstracts, it is highly recommended to have a system that can help you quickly sort through them.
So, here are a few of the features you should look for in a peer review system that can make you and your team of reviewers more efficient.
1. Efficient Allocation of Submissions
Distributing a flood of submissions to the relevant reviews can be overwhelming if your call for papers or abstracts has been popular and created a massive influx of abstracts.
Many peer review systems neglect to include a feature that allows event organizers to efficiently distribute papers to reviewers. Always check the feature lists of any peer review system you consider, as this feature can signfiicantly reduce your workload.
A quality review system can go a step further and automatically match papers to reviewers with the relevant expertise
2. Marking Scheme
Matching your review process to your event’s requirements is critical. This means your peer review system should include a feature that allows you to configure a marking scheme that matches your process, such as giving reviewers the ability to provide feedback to chairs and authors, for example.
Ensure the system is not overly complicated for reviewers to submit their work. If your system is hard to figure out, you may find willing reviewers are harder to come by.
3. Include Multiple Stages
Research methodology is constantly evolving, and so too are peer review systems. A continually shifting landscape in research strategies means that peer review systems need to keep up, so your peer review system should include a multi-stage review feature.
A multi-stage peer review system will be necessary if your event or conference enables authors to re-submit their papers with requested corrections for the second round of reviews. Many of the peer review systems on the market have yet to implement this feature, so make sure the one you consider includes it.
4. Re-allocation of Submissions
Reviewers generally go unpaid and freely give their time to benefit the research community. They are regular people, so they may not always be able to commit to a paper review for your event.
A good peer review system should incorporate a feature that allows you to quickly swap out one unavailable reviewer for someone who is more accessible. When you combine this feature with a robust communication system, finding a new peer reviewer is no longer the roadblock it once was.
5. Efficient Communication Channels
When you are planning a conference or event, efficient communication channels are critical to its success.
Organizers have a lot on their plate with sending out a call for papers, sending reminders, and communicating with reviewers.
Relying on outside communication channels can add bottlenecks to the system as emails get drowned in a flooded inbox. An integrated communication channel delivers messages directly to reviewers so there is no chance your friendly reminders can go unmissed.
6. Ability to Deliver a Balanced Workload
Without some way to accurately track the work you are handing out, you can expect to receive complaints from reviewers who are lumbered with more work than they can possibly handle, while their colleagues seem to have free time to twiddle their thumbs.
It’s not uncommon for overworked reviewers to go silent or withdraw their offer, which leaves you in the awkward position of having to replace them.
Ensure your peer review system delivers an overview of the submissions you are dishing out, so you can make sure everyone is receiving a fair workload.
7. Progress Dashboard
A modern office is a busy place, so people won’t be submitting regular progress reports or watching the email inbox like a hawk. However, you need to stay updated with how things are progressing.
A good peer review system should include a progress dashboard that will keep you in the loop despite your emails going unanswered. This one simple feature is often overlooked, but it can be a huge timesaver that will save you from sending unwelcome progress report requests every day.
Plus, your eagle eye overview will show you who is lagging behind, so you can send a few timely reminders.
