Community Moderation Process

All reviews the Community Reviewers provide will be subject to an automated review process using Catalyst Flags. Catalyst Flags* are pieces of code known as scripts that are run to analyze all reviews by the Community Reviewers. Catalyst scripts will raise flags in case of;

• Similarity ~ reviews which are identical or almost identical to a previously submitted review
• Profanity ~ reviews that contain profanities or abusive content or language
• Use of AI ~ reviews that do not reflect the opinion of the Community Reviewer or which are provided by others or an AI-agent acting on their behalf.

Length requirements have already been applied for automatically (reviews containing less than 150 characters in impact, feasibility, and value for money criteria)

All reviews that have been flagged will go through a moderation process involving Community Moderators (LV2).

• Community Moderators will be randomly allocated flagged reviews for further assessment. In addition, between 1-5% percent of reviews not flagged by scripts will be allocated to LV2 Community Reviewers for a spot check (depending on the total reviews received).

• Community Moderators will be provided a secure login for their moderation function in the Moderation Module.

• Community Moderators can see which flags have been raised by the scripts about the review/reviewer.

• After reading the review and flags, Community Moderators will submit their moderations, marking reviews they believe were legitimately flagged and providing a reason for removing the review. Eg. Where the flag is raised because of potential ChatGPT AI involvement or plagiarism, the Moderator must judge whether they believe the review is not a reviewer’s opinion. If the Moderator agrees with the flag, they mark it as REMOVE and give a rationale.

• The rationale should be a short written statement; a selection must be made from a menu of reasons. For example, the moderator would select from options such as:

  1. There is no evident issue with review
  2. I believe this is not the person’s own opinion
  3. There is profanity
  4. There is too much similarity with another review
  5. Rationale doesn’t correspond with the score given
  6. Rationale contains a false statement

• If Moderators disagree with the flag and believe the review should be included for voters to consider, they mark it as VALID and give a rationale.

• Moderators should only discard reviews if there is a valid reason why voters shouldn’t see that review. Unless the review has an apparent reason, such as the flags below or another cause (spamming, for instance), the review should be included.