An abstract review scoring matrix gives every reviewer the same questions, score anchors and decision rules. That consistency matters when a conference receives hundreds or thousands of submissions across different tracks. Without a shared rubric, one reviewer may reward novelty, another may focus on writing quality, and a third may treat personal interest as evidence of relevance.
A useful matrix does more than add up numbers. It separates eligibility checks from academic merit, explains what each score means, records the reason behind the score and shows program chairs when a submission needs closer attention.
This guide includes four downloadable templates, a practical 1-5 scoring system, an editable weighted matrix and a workflow for calibrating reviewers before live scoring begins.
Download the abstract review scoring matrix templates
Use the general template as a starting point, then adapt the criteria to the conference discipline, submission type and decision process. Confirm the final rubric before reviewers receive any submissions.
- Download the complete template pack
- Download the editable Excel template pack
- Download the print-ready PDF template pack
Template 1: General academic conference matrix
General-purpose abstract review scoring matrix with five scored criteria, 1-5 anchors, reviewer confidence and chair escalation checks.
This general-purpose matrix scores methodological soundness, originality, relevance, clarity and potential significance. It keeps reviewer confidence separate from the composite score and includes eligibility and escalation checks for the program chair.
When to use this template: Choose this version for multidisciplinary conferences, research symposia, association meetings and smaller review programs that want a straightforward, equally weighted rubric. It is also a practical starting point when a committee is introducing structured abstract review for the first time.
What the template fields mean:
- Eligibility checks confirm scope, completeness, conflict disclosure and any applicable ethics concern before scoring begins.
- Five scored criteria ask reviewers to assess methodological soundness, contribution, relevance, clarity and potential significance on the same 1-5 scale.
- Reviewer rationale records the evidence behind the scores and any important limitation.
- Overall recommendation lets the reviewer state the practical decision without treating the numerical average as automatic acceptance.
- Reviewer confidence and chair checks identify low-expertise assignments, major score differences and issues that need another review or chair discussion.
How to use the result: Average the applicable criterion scores, then read the rationale and overall recommendation alongside the total. Compare criterion-level scores across reviewers rather than relying only on the average. The chair should review material disagreements, conflicts and low-confidence recommendations before making a final decision.
Download the full-resolution general matrix
Template 2: Weighted multi-track conference matrix
This version assigns sample weights to five criteria. The percentages are examples, not universal rules. Test and approve the weights before review begins. If a criterion does not apply to a track, redistribute its weight instead of entering zero.
When to use this template: Use a weighted matrix when the conference has a documented reason to make some criteria more influential than others. A research-intensive track may give greater weight to methodological soundness, while an applied track may place more emphasis on relevance or practical significance. Use the same approved weights for every submission within a track.
What the weighted fields mean:
- Raw score is the reviewer’s 1-5 assessment of the criterion.
- Weight is the percentage of the final composite assigned to that criterion.
- Weighted result is the raw score multiplied by the decimal form of the weight. For example, a methodology score of 4 with a 30 percent weight contributes 1.20.
- Composite score is the sum of all weighted results. When the weights total 100 percent, the result remains on the 1-5 scale.
- Reviewer confidence, eligibility checks and safety or ethics flags remain separate because they describe the review context, not the abstract’s weighted quality score.
Why use weighted scoring: Weighting makes the conference’s priorities explicit and stops a lower-priority criterion from influencing the final result as much as a critical one. It is useful for multi-track programs, competitive research conferences and disciplines where technical or clinical rigor must carry greater influence. Do not introduce or change weights after seeing the submission scores. Test the model with sample abstracts, confirm that the weights total 100 percent and explain the calculation to reviewers before live review begins.
Download the full-resolution weighted matrix
Template 3: Medical and clinical abstract matrix
The medical template asks reviewers to judge whether the study design fits the research question, whether the methods and results are sufficiently reported, and whether the conclusion follows from the evidence. Ethics approval, consent and registration appear as applicability checks rather than automatic quality points.
When to use this template: Choose this version for medical, clinical, public-health and health-science meetings where reviewers must judge whether the design, reported evidence and conclusion align. Adapt the terminology to the study types accepted by the conference rather than assuming every abstract describes a clinical trial.
What the template fields mean:
- Question and design fit checks whether the study design can answer the stated objective.
- Methods and reporting assess whether the population, intervention or exposure, comparison and analysis are described well enough for an abstract.
- Results and evidence ask whether the reported findings support the claims without rewarding positive results simply because they are positive.
- Conclusion and clinical relevance consider whether the interpretation is proportionate and useful to the intended audience.
- Ethics, consent and registration checks record whether each item is reported when applicable. They are chair-level applicability checks, not bonus points in the quality score.
How to use the result: Review the composite with the study-design comments and applicability checks. Refer missing or unclear ethics information to the chair under the conference’s policy. Do not automatically reject an abstract or apply a numerical penalty simply because a check does not apply to that study type.
Download the full-resolution medical and clinical matrix
Template 4: Computer science abstract matrix
This technical-review matrix covers soundness, novelty, significance, clarity, reproducibility and track relevance. It also gives reviewers a separate place to flag privacy, security, safety or dual-use concerns for chair review.
When to use this template: Use this version for computer science, machine learning, data science and engineering tracks where technical validity and reproducibility require explicit attention. It works best when the conference has already stated what evidence can reasonably be expected in an abstract and whether code, data or other artifacts are required later in the process.
What the template fields mean:
- Technical soundness asks whether the method, reasoning and evaluation support the claim.
- Novelty and significance separate what is new from why the contribution matters.
- Clarity and track relevance assess whether reviewers can understand the work and whether it fits the published scope.
- Reproducibility considers the method detail, experimental setup and availability or planned availability of supporting artifacts when applicable.
- Privacy, security, safety and dual-use flags alert the chair to risks that should not be hidden inside the composite score.
- Reviewer confidence shows whether a specialist or additional review may be needed.
How to use the result: Compare the technical criteria and the written rationale before ranking submissions. Send unresolved safety, privacy or dual-use concerns to the chair even when the composite is high. If reviewers disagree because one lacks the required technical expertise, assign another qualified reviewer instead of averaging the difference away.
Download the full-resolution computer science matrix
What a useful scoring matrix actually does
A scoring matrix helps a conference make consistent, explainable decisions. It cannot compensate for a weak review process, and a precise-looking total is not automatically a fair decision.
A reliable matrix does eight practical things:
- Give reviewers the same criteria and definitions.
- Separate administrative eligibility from academic merit.
- Use score anchors that explain what low, adequate and high performance look like.
- Require a short rationale for consequential scores.
- Handle missing or non-applicable criteria explicitly.
- Keep reviewer confidence separate from the abstract’s quality score.
- Show chairs when reviewers disagree enough to require discussion.
- Preserve an audit trail for final decisions and author feedback.
Research supports using detailed, criterion-level anchors rather than a bare number scale. A study of scientific meeting abstract review found that a more detailed rubric improved inter-rater agreement and reduced leniency bias in that setting. The finding does not prove that one rubric will work for every conference, but it supports testing a defined scoring framework instead of relying on personal impressions alone. Read the scoring-rubric study.
How to adapt the template to your conference
Start with the decision the committee must make, not with a long list of possible criteria. A research track, a workshop proposal and a practice case study may all need different evidence. If the same form covers every submission type, reviewers may end up scoring questions that do not fit the work in front of them.
Use the following sequence to turn the downloadable matrix into a working review form:
- Define the decision. State whether reviewers are recommending acceptance, presentation format, track placement or a combination of these. Tell them which decisions belong to the chair rather than the individual reviewer.
- Separate eligibility from merit. Put scope, required fields, anonymity, conflicts and ethics checks before the scored criteria. Decide what happens after each flag. Some issues require reassignment, some require an author query and others require chair review.
- Choose only criteria that affect the decision. Five or six well-defined criteria are usually more useful than a longer form with overlapping questions. If originality and contribution lead reviewers to assess the same evidence, combine them. If a clinical track needs a separate results-reporting criterion, add it only to that track.
- Write criterion-specific anchors. Describe what a score of 1, 3 and 5 looks like for each criterion. Reviewers can infer the positions between those anchors, but they should not have to guess what “strong” means. Use evidence that can reasonably appear in an abstract.
- Decide how N/A, confidence and flags work. Make these rules visible on the form. State whether an N/A response needs a comment, how the composite will be recalculated and when low confidence triggers reassignment or an additional review.
- Set comment requirements. A practical rule is to require a brief rationale for scores of 1 or 5, all borderline recommendations and any recommendation that conflicts with the calculated result. Give reviewers a separate confidential field for concerns that should not go to the author.
- Test the complete form. Run two or three sample abstracts through the matrix. Check the calculation, required fields, reviewer permissions, chair alerts and exported results. Revise confusing wording before real submissions enter the workflow.
Finish with a one-page reviewer instruction sheet. It should name the scoring scale, explain the difference between merit and eligibility, define the comment standard, list the escalation route and state the confidentiality and conflict rules. Reviewers are more likely to follow a short document they can consult during scoring than guidance scattered across invitation emails.
Start with eligibility gates, not scores
A few questions need to stop or redirect the review before merit scoring begins. Treat them as gates or flags:
- Does the submission fit the conference or track scope?
- Is the required information present?
- Does the reviewer have a conflict of interest?
- Has the submission followed the applicable anonymity policy?
- Is there an ethics, safety, privacy or research-integrity concern?
Do not bury these issues inside the composite score. A conflict of interest calls for disclosure or recusal, not a lower score. A missing ethics statement may need an author query or chair review, depending on the study and conference policy. It does not automatically justify a mathematical penalty applied to every submission.
The COPE Ethical Guidelines for Peer Reviewers ask reviewers to disclose conflicts, protect confidentiality, assess work objectively and provide constructive feedback. Turn those principles into visible checks at the start of the form.
For a fuller workflow, see Dryfta’s guide to handling conflicts of interest in abstract review.
Five scored criteria and two separate signals
Most conferences can begin with five scored criteria. The definitions can change with the field or submission type while the basic structure stays useful.
1. Methodological soundness
Ask whether the approach fits the question and whether the abstract provides enough information to judge its credibility. Do not reward one study design simply because it sits higher in a familiar evidence hierarchy. Judge a qualitative study, case study, simulation, survey or trial against the question it addresses and the standards of its field.
Useful reviewer questions include:
- Is the objective or research question clear?
- Does the design address that question?
- Are the sample, data source and analysis described well enough for an abstract?
- Are the claims proportionate to the evidence?
- Are material limitations visible?
2. Originality and contribution
Originality does not always mean a completely new topic. A submission may make a valuable contribution by testing an idea in a new context, improving a method, providing a useful replication, presenting important negative findings or connecting evidence that has not been considered together.
Ask reviewers to identify the contribution rather than guess whether the work feels novel. This produces more useful comments and makes the score easier to defend.
3. Relevance to the conference or track
Relevance measures fit with the published call for abstracts. It does not measure whether the reviewer personally likes the topic.
Before review opens, give each track a short scope statement and examples of borderline submissions. If a submission fits another track better, allow the reviewer to recommend reassignment instead of lowering its academic score.
4. Clarity and completeness
A good abstract lets the reviewer understand the problem, approach, main findings and conclusion within the stated word limit. Score the clarity of the ideas, not the author’s accent, institution or command of idiomatic English.
Minor language errors do not outweigh clear and important work. When the meaning is difficult to follow, identify the specific missing or ambiguous information.
5. Potential significance or impact
Significance asks what the work may add to research, practice, education, policy or the conference discussion. Look for a credible connection between the findings and the claimed value.
Avoid vague prompts such as “Is this interesting?” They invite personal preference. Use a defined question such as “If the findings are sound, how useful are they likely to be for this track’s audience?”
Overall recommendation
Keep the overall recommendation separate from the calculated score. A composite helps reviewers compare submissions, but it cannot capture every program decision. Track balance, duplicate topics, presentation format, capacity and ethics flags may still require committee judgement.
Use a small set of recommendation options, such as Accept, Borderline and Do not accept. Require a reason when the recommendation conflicts with the numerical score.
Reviewer confidence
Confidence describes the reviewer’s familiarity with the subject and their ability to judge the submission. It does not describe submission quality.
Do not automatically multiply the abstract score by reviewer confidence. Reviewers interpret confidence scales differently, and a mathematical multiplier can give the most self-assured reviewer disproportionate influence. Use confidence to identify a need for reassignment, an additional specialist review or chair discussion. The ACM SIGPLAN Conference Submission Review Policy similarly treats expertise and review quality as matters that require careful interpretation rather than a simple substitute for judgement.
Make every score mean something
A number without a definition invites inconsistency. A practical 1-5 scale can work well when every point has a clear meaning.
| Score | Anchor | Reviewer interpretation |
|---|---|---|
| 1 | Major concerns | The criterion is seriously deficient, unsupported or not assessable from the submission. |
| 2 | Substantial concerns | Important weaknesses reduce confidence, although some relevant information is present. |
| 3 | Adequate | The abstract meets the expected conference standard with manageable limitations. |
| 4 | Strong | The abstract performs clearly above the expected standard for this criterion. |
| 5 | Exceptional | The abstract is outstanding for this criterion and the reviewer can explain why. |
Place these general anchors beside criterion-specific prompts. “Strong methodology” and “strong relevance” mean different things. Reviewers need to see what evidence supports each score without consulting a separate manual.
Avoid anchors such as “perfect,” “zero flaws” or “completely out of scope.” No abstract is perfect, and an out-of-scope submission belongs in the eligibility workflow rather than at the bottom of a quality scale.
N/A is not the same as missing information
Allow N/A only when a criterion genuinely does not apply, and require an explanation. A blank score signals an incomplete review. N/A records an intentional decision.
For an unweighted matrix, calculate the average from the applicable numeric criteria only:
Composite score = sum of applicable criterion scores / number of applicable criteria
For a weighted matrix, do not leave an inapplicable criterion’s weight active. Redistribute the weight before review starts so that the applicable weights still total 100 percent.
Missing information is different from N/A. If a required result or method is absent, the relevant criterion may deserve a lower score. If the form never requested that information, do not penalize authors for omitting it.
Weighted scores: simple maths, explicit choices
Weights can help when a conference intentionally values some criteria more than others. They can also hide value judgements if organizers choose them casually.
The sample weighted template uses:
- Methodological soundness: 30 percent
- Originality and contribution: 25 percent
- Relevance to the conference or track: 20 percent
- Clarity and completeness: 15 percent
- Potential significance or impact: 10 percent
Calculate each weighted score as:
Criterion weight x criterion score
Then add the five weighted scores. When the weights total 100 percent and every score uses the 1-5 scale, the final composite also remains on a 1-5 scale. Round only the final result, normally to two decimal places.
Treat these percentages as an example. Test the model with past or sample abstracts and check whether it produces reasonable results across tracks.
Worked example: calculate and interpret a score
Suppose a reviewer gives one submission the following scores in the sample weighted matrix:
| Criterion | Score | Weight | Weighted result |
|---|---|---|---|
| Methodological soundness | 4 | 30% | 1.20 |
| Originality and contribution | 4 | 25% | 1.00 |
| Relevance to the track | 5 | 20% | 1.00 |
| Clarity and completeness | 3 | 15% | 0.45 |
| Potential significance | 3 | 10% | 0.30 |
| Composite | 100% | 3.95 |
The 3.95 composite summarizes the criterion scores, but it does not finish the review. The reviewer still needs to explain the strongest evidence, identify the main limitation and give an overall recommendation. If the reviewer selects “Do not accept,” the form should require an explanation because that recommendation appears inconsistent with the numerical result.
Now suppose a second reviewer gives the same abstract 2.85 and questions whether the methods support the conclusion. The chair needs to inspect the criterion-level scores and both comments. The difference may come from a genuine methodological concern, a subject-expertise gap or inconsistent use of the anchors. Assigning a third reviewer can help when the disagreement concerns technical evidence. A short chair discussion may be enough when the reviewers simply interpreted an anchor differently.
Avoid setting an automatic acceptance line such as “every abstract above 3.5 is accepted” before you understand the submission pool and available program capacity. A threshold can support triage, but it should not override conflicts, ethics flags, reviewer disagreement or a clearly documented problem. If the committee uses a cut-off, record how it was chosen, apply it consistently within the relevant track and define who may approve an exception.
Do not compare raw scores across tracks unless the criteria, weights and calibration are equivalent. A 3.8 in a clinical research track and a 3.8 in a practice workshop track may reflect different evidence and expectations. Make final comparisons within the decision structure the conference has actually published.
Averages can hide reviewer disagreement
The average of several reviews can be useful, but do not let it hide disagreement.
Suppose Reviewer A gives an abstract 4.2 and Reviewer B gives it 2.8. The mean is 3.5, but that number conceals a 1.4-point gap. The reviewers may have interpreted a criterion differently, missed information, brought different expertise or applied different standards.
Set an escalation rule before review begins. For example, a conference might ask the chair to inspect:
- A difference greater than one point in composite scores.
- Opposing overall recommendations.
- A score of 1 on methodological soundness.
- A conflict, ethics or safety flag.
- A low-confidence review paired with a decisive recommendation.
These are examples, not universal thresholds. Choose a rule that suits the scale, number of reviewers and volume of submissions. The important point is to define it in advance.
When disagreement occurs, the chair can request clarification, assign an additional reviewer or hold a short adjudication discussion. Do not simply average away a material difference.
Calibrate before real abstracts arrive
Calibration is one of the most useful steps in conference peer review, yet many committees skip it.
Use this sequence:
- Select two or three anonymized sample abstracts that represent clear, borderline and difficult cases.
- Ask reviewers to score them independently with the proposed matrix.
- Compare criterion-level scores, not only the final totals.
- Discuss why reviewers interpreted the anchors differently.
- Refine unclear wording and document the agreed interpretation.
- Repeat one sample if the first round exposes large differences.
- Open live review only after the form and guidance are stable.
Calibration does not force everyone to give identical scores. It helps reviewers apply the same questions and recognize where professional judgement legitimately differs.
Prepare backup reviewers before assignments begin. Dryfta’s guide to building a reserve reviewer pool explains how that reduces delays when a conflict, subject mismatch or withdrawal appears.
One matrix cannot fit every discipline
One rubric cannot cover every conference without adjustment.
Medical and clinical conferences
Match criteria to the study design. Check whether the question, methods, results and conclusion align. Do not automatically reward a randomized design or penalize negative and inconclusive findings.
Ethics approval, consent, trial registration and disclosures apply differently across study types and jurisdictions. Ask whether each item is reported when applicable. The EQUATOR Network’s conference-abstract resources point organizers to design-specific guidance, including CONSORT for randomized trial abstracts and PRISMA for systematic-review abstracts.
Computer science and engineering conferences
Technical soundness, novelty and significance often need separate scores. Reproducibility may include sufficient method detail, access to code or data when expected, or a clear explanation when artifacts cannot be shared.
Keep privacy, security, safety and dual-use concerns outside the composite score. A submission can be technically strong and still require chair review.
Humanities and social-science conferences
Replace laboratory-style prompts with criteria appropriate to the field, such as argument quality, use of sources, theoretical contribution, interpretive rigor and relevance to the conference theme.
Do not retain a methodology criterion whose wording assumes experimental or quantitative work.
Practice, education and case-study tracks
Consider practical relevance, transferability, evidence of outcomes, reflection on limitations and value to the intended audience. Explain whether preliminary initiatives are eligible and how reviewers will judge them.
Comments need to explain the number
Scores alone do not help chairs resolve disagreement or help authors understand a decision. Require short comments that point to evidence in the abstract.
A useful comment usually contains:
- The main strength.
- The most important concern or missing information.
- The reason for any very low or very high score.
- A clear explanation when the overall recommendation differs from the composite.
- A confidential chair note for conflicts, ethics or integrity concerns.
Avoid comments about an author’s identity, institution, nationality or presumed experience. Avoid asking for information that the call for abstracts did not require.
Confidentiality still comes first
Treat submissions and review records as confidential. Never paste unpublished abstracts into unapproved external services or share them outside the review team. Any permitted support tool must follow the conference’s confidentiality policy, and the reviewer remains responsible for every score and comment.
The ICMJE recommendations on peer review emphasize confidentiality, conflict disclosure and constructive critique. State those expectations in the reviewer invitation and repeat them in the review form.
Before a reviewer clicks Submit
Before submitting a review, confirm that you have:
- Disclosed any conflict of interest or recused yourself.
- Reviewed only submissions that match your expertise.
- Used the published criteria and anchors.
- Kept eligibility flags separate from merit scores.
- Explained low, high, borderline and N/A ratings.
- Assessed clarity without penalizing non-native language unnecessarily.
- Kept reviewer confidence separate from the composite score.
- Flagged ethics, safety or integrity concerns for the chair.
- Protected the submission’s confidentiality.
- Checked that the overall recommendation matches your written rationale.
What chairs need to watch
Program and track chairs need to review the process as well as the scores.
Check for:
- Incomplete reviews and unexplained N/A selections.
- Reviewers who consistently score much higher or lower than the group.
- Large score gaps or opposing recommendations.
- Conflicts and low-expertise assignments.
- Criteria that reviewers interpret inconsistently.
- Comments that do not support the numerical scores.
- Potential language, prestige or affiliation bias.
- Ethics, safety, privacy or integrity flags.
- Track-level differences caused by uneven reviewer calibration.
Do not change scores silently. Record the reason for reassignment, additional review or chair adjudication.
Putting the matrix into practice with Dryfta
A well-designed matrix still needs a dependable operational workflow. Dryfta gives conference teams a central place to collect abstracts, assign reviewers, manage review activity and move submissions through decisions.
Before configuring the form, finalize the criteria, anchor text, weights, recommendation options and escalation rules. Then test the complete workflow with sample submissions and reviewer accounts. This catches ambiguous questions and permission problems before live review begins.
For the wider setup sequence, read the step-by-step guide to peer review for abstract submissions. You can also review Dryfta’s abstract management software or request a demo to discuss your conference workflow.
Frequently asked questions
What is the best scoring scale for conference abstracts?
There is no universal best scale. A 1-5 scale is usually easy to explain and gives reviewers enough distinction without suggesting unrealistic precision. The anchor definitions and reviewer calibration matter more than the number of points.
Should reviewer confidence affect the abstract’s score?
Usually no. Keep confidence as a separate signal for assignment, additional review or chair escalation. Do not use it as a multiplier unless the conference has deliberately tested and documented that policy.
How many reviewers should score each abstract?
Use more than one independent reviewer when resources allow. The appropriate number depends on submission volume, specialist coverage, risk and conference policy. Document any exceptions and define how the chair will handle disagreement.
Should a conference use weighted criteria?
Use weights only when the differences reflect an intentional program priority. Test the model with sample abstracts, publish the rules to reviewers and keep the weights consistent within a track.
What should happen when a criterion is not applicable?
Select N/A and explain why. An unweighted average excludes that criterion. In a weighted matrix, redistribute the weight before review begins rather than treating N/A as zero.
Can a high composite score guarantee acceptance?
No. The score informs the decision, but chairs may also need to consider reviewer disagreement, ethics flags, track fit, duplicate topics, presentation format and available program capacity. Document any departure from the published scoring process.
A good matrix makes judgement visible
Start with the simplest matrix that captures the conference’s real decision. Define each criterion, publish the anchors, test the form with sample abstracts and keep confidence and eligibility checks outside the composite. A transparent scoring process will not eliminate professional judgement, but it will make that judgement more consistent, reviewable and useful.
