How to run a paper matching

Running Paper Matching

In order to run the matching for Senior Area Chairs, Area Chairs, and Reviewers, you must first run Setup Paper Matching from the venue request form.

Once that is done, a link for 'Paper Assignments' should appear on your Program Chair console.

Clicking on one of the assignment links will bring you to the assignment page, where you can create a new matching configuration. If members of your Senior Area Chair, Area Chair or Reviewer group have profiles without publications, you will need to select ‘Yes’ for ‘Allow Zero Score Assignments’ in order to obtain a solution. Please note that all members of a group must have OpenReview profiles in order for the automatic assignment algorithm to run. Any members without profiles must be removed from the group before this step.

You can learn more about our automatic paper matching algorithm from its GitHub repository. To create a new matching, click the 'New Assignment Configuration'. This will pull up a form with some default values pertaining to your matching settings:

User Demand

• The number of users that should be assigned to each paper

Max Papers

• The maximum number of papers that can be assigned to each reviewer

Min papers

• The minimum number of papers that can be assigned to each reviewer

Alternates

• How many alternate reviewers should be saved per paper

Paper Invitation

• Invitation of the submissions that will be assigned in this matching

• Defaults to venue_id/-/Submission for single blind and venue_id/-/Blind_Submission for double blind venues

Match Group

• The group ID of users to be assigned to submissions

Scores Specification

• JSON providing further details and customization to scores.

• Each key represents an edge invitation (affinity score, bid, etc.). Weight can be added to a given score value with the numerical field 'Weight'. 'Default' is a numerical value that is used when there is not an edge for a specific reviewer-paper pair. Finally, 'translate_map' is a map function that translates an edge label value into a number.

  In the example below, the aggregate score being used by the optimizer is: weight * (affinity score) + weight * (translate_map(bid))

• {
      "Example_Venue/2022/Conference/Reviewers/-/Affinity_Score": {
          "weight": 1,
          "default": 0
      },
      "Example_Venue/2022/Conference/Reviewers/-/Bid": {
          "weight": 1,
          "default": 0,
          "translate_map": {
              "Very High": 1,
              "High": 0.75,
              "Neutral": 0,
              "Low": -0.5,
              "Very Low": -1
          }
      }
  }

Conflicts Invitation

• Invitation for storing conflicts between users and papers

• Defaults to venue_id/user_group/-/Conflict

Custom User Demand Invitation

• If certain papers require a custom number of assigned users, Program Chairs can create edges determining these requests and provide the invitation for used for those edges here.

• Defaults to venue_id/user_group/-/Custom_User_Demands

Custom Max Papers Invitation

• Some reviewers may submit requests to only have a certain number of assigned papers. The matcher will convert those requests into edges. This determines the invitation that will be used for those edges.

• Defaults to venue_id/user_group/-/Custom_Max_Papers

Solver

• Which algorithm (MinMax, Fairflow, or Randomized) will be used to generate automatic assignments.

  • MinMax: Optimizes the scores while respecting the min and max quotas for each paper and reviewer. You can read more about MinMax here.
  • Fairflow: Tries to make every match have at least some minimum affinity. You can read more about Fairflow here.
  • Randomized: Generates randomized assignments and selects the assignment that maximizes expected total affinity without breaking the probability limits. You can read more about the Randomized solver here.
• You can read more about all solver options here.
• Defaults to MinMax

Allow Zero Score Assignments

• Whether or not assignments with scores of 0 should be allowed. If a reviewer does not have any publications listed on their profile, they will have an affinity score of 0 with all submissions. Therefore, if you have users without publications, you will need to select "yes" in order to get a solution.

Randomized Probability Limits

• If you select "Randomized" for the solver, it will select a random assignment that maximizes expected total affinity, subject to the probability limit provided here. What this means is that for each reviewer-paper assignment, the probability of that match being generated in a random assignment is limited to this value. This should make the outcome of the matching more difficult to predict.

After filling out the matching configuration form and hitting submit, you should see the following:

You can view, edit or copy the values you filled out in the matching form. When you are happy with your configuration, you should hit 'Run Matcher' and wait until its status is 'Complete'.

Modifying Proposed Assignments

Once the Matching algorithm completes, the proposed assignments will be generated with options to browse them, view statistics or deploy proposed assignments. If you click ‘Browse Assignments’ you will be brought to the edge browser, where you can browse, edit, and create proposed assignments. You can read more about modifying proposed assignments here.

Be careful before deploying proposed assignments! Do not deploy them unless you are sure that you are satisfied with them. Undoing a deployment is difficult and in most cases it requires the OpenReview staff intervention which can only be done during office hours.

Troubleshooting Matcher

If you get "No Solution" after running the matcher, you can view the configuration to see the entire error message. If the message is something like the following:

• Error Message: Total demand (150) is out of range when min review supply is (34) and max review supply is (100)

that means that your constraints require more reviewers or area chairs than you currently have. The total demand is equal to (number of submissions * user demand) + (number of submissions * alternates). The max review supply is the number of reviewers available * max papers and the review supply is the number of reviewers available * min papers. Your total demand must fall within this range in order to obtain a solution.

Note that completion of this step does not make assignments, it only creates a proposed assignment configuration. Those assignments will need to be deployed before Reviewers or Area Chairs will see them.