Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
If you selected to use affinity scores, conflicts, bids, or reviewer recommendation scores, you will not be able to use the following options. Instead, follow the guide on how to do automatic assignments.
If you did not specify you wanted to use the OpenReview matcher to assign reviewers to papers, you will be able to manually assign them using the PC console.
Make sure your submission deadline has passed. Unless your venue is single blind and public, assignments cannot be made until after the submission deadline.
Run the review stage by clicking on the Review Stage button on the request form for your venue.
Under the 'Paper Status' tab in the PC console, click on 'Show Reviewers' next to the paper you want to assign reviewers to.
To assign reviewers from the reviewer pool, you can choose a reviewer from the dropdown. Here, you can also search reviewers in the reviewer pool by name or email. After finding the reviewer you want to assign, click on the 'Assign' button.
To assign reviewers from outside the reviewer pool, you should type the reviewer's email or OpenReview profile ID (e.g., ~Alan_Turing1) in the text box and then click on the 'Assign' button. A reviewer does not need to have an OpenReview profile in order to be assigned to a paper.
Note that assigning a reviewer to a paper through the PC console automatically adds that reviewer to the reviewers pool and sends them an email notifying them about their new assignment.
Area Chairs: Unfortunately, assigning ACs is not available through the PC console, but manual AC assignments can be done through the Python library: (You can check out the docs for our Python API here)
You will need to use your own OpenReview credentials to initialize the Client object.
request_form_id (string) refers to the forum id of the venue request for your venue, (e.g., https://openreview.net/forum?id=r1lf10zpw4)
paper_number (int) is the number of the paper you want to assign an area chair to (you can find this in the 'Paper Status' tab of the PC console)
user_id (string) is the email address or OpenReview profile ID (e.g., ~Alan_Turing1) of the user you want to assign
Note that assigning an area chair using python does not send an email to that user. For information on how to contact Area Chairs through the UI, click here. For information about how to contact Area Chairs using python, click here.
If you chose to manually assign reviewers from the PC console, that means that assignments are based solely on group membership. For example, anyone in the Paper1/Reviewers group will have reviewer permissions for Paper 1. If you removed them from that group, they would be effectively unassigned from Paper 1 and would lose access to those permissions. Note that the following steps should only be used for venues that used manual assignment through the PC console. Do not do this if you used the edge browser.
Locate the reviewer group for a particular paper. You can build the url to this group like so: https://openreview.net/group/info?id= + your venue ID + /Paper + the paper's number + /Reviewers
For example, if you wanted to assign a reviewer to Fake Conference paper 1, you could go to https://openreview.net/group/info?id=Fake.cc/2022/Conference/Paper1/Reviewers.
Click "Edit group".
Add a profile ID or email address to the group to assign a new reviewer. Or, remove a group member to un-assign them.
Venues that selected a value for 'Paper Matching' in the venue request form will have the option to use automatic assignment by doing the following:
Make sure your submission deadline has passed. Unless your venue is single blind and public, assignments cannot be made until after the submission deadline.
Run the Post Submission stage to reveal submissions to reviewers.
Run the Review Stage.
Before calculating affinity scores and conflicts, you should make sure that your submission deadline has passed and that you have run either the ‘Post Submission Stage’ or the ‘Review Stage’.
You can calculate affinity scores and conflicts for your venue using OpenReview's 'Paper Matching Setup' feature. Paper Matching Setup is enabled for any venue that selected an option for the 'Paper Matching' question on the venue request form. This feature allows Program Chairs to compute or upload affinity scores and/or compute conflicts.
You can find the 'Paper Matching Setup' button on your venue request form next to 'Remind Recruitment'.
Clicking it should bring up the following form. The 'Matching Group' is a dropdown menu of the groups you can use in the matcher (Reviewers, Area Chairs, Senior Area Chairs), depending on whichever you selected for your venue. You can select if you would like affinity scores and/or conflicts computed. Alternatively, you can compute and upload your own affinity scores using the OpenReview expertise API: https://github.com/openreview/openreview-expertise
Conflict detection uses the information of the users' coauthors from publications in OpenReview as long as they are publicly visible and the users' Profile. Therefore, the more complete and accurate the information in the Profile is, the better the conflict detection.
The sections of the Profile used for conflict detection are the Emails section, the Education & Career History section, and the Advisors, Relations & Conflicts section.
Another parameter that can be controlled is the amount of years you want to consider when looking at conflicts. For example, there might be two users who worked at company A at some point. However, User A worked at Company C ten years ago and User B just started working at Company C. If the amount of years is set to 5, for example, a conflict won't be detected between User A and User B because only the history, relations and publications from the past 5 years will be taken into consideration. By default, all relations, history, and publications are taken into consideration for conflict detection.
Since a lot of users use email services such as gmail.com, a list of common domains is used to filter them out before conflicts are computed.
There are two policies when computing conflicts: default and neurips.
Uses the domains and computes subdomains from the Education & Career History section.
Uses the domains and computes subdomains from the emails listed in the Advisors, Relations & Conflicts section.
Uses the domains and computes subdomains from the Emails listed in the Emails section.
Uses the publication ids in OpenReview that the user authored.
Note that emails do not have a range of dates for when they were valid in the user's Profile. The Neurips policy addresses this issue.
Uses the domains and computes subdomains from the Education & Career History section.
Uses the domains and computes subdomains from the emails listed in the Advisors, Relations & Conflicts section.
Uses the domains and computes subdomains from the Emails listed in the Emails section, if and only if, no domains were available in the Education & Career History and Advisors, Relations & Conflicts sections.
Uses the publication ids in OpenReview that the user authored.
Once all the information is extracted from the users' Profiles, the following rules apply to find a conflict between User A and User B:
If any of the domains/subdomains from the Education & Career History section of user A matches at least one of the domains/subdomains of the same section of User B, then a conflict is detected.
If any of the domains/subdomains from the Advisors, Relations & Conflicts section or the Emails section of user A matches at least one of the domains/subdomains of the same sections of User B, then a conflict is detected.
If any of the publications of User A is the same as one of the publications of User B. In other words, if User A and User B are coauthors, then a conflict is detected.
Running the paper matching setup should output a comment on your venue request page. If there were members missing profiles or publications, the message will identify them and say 'Affinity scores and/or conflicts could not be computed for these users. Please ask these users to sign up in OpenReview and upload their papers. Alternatively, you can remove these users from the Reviewers group.' This message does not mean that the process failed, but that those members were excluded from the calculations. You have two options:
Remove reviewers without profiles from the reviewers group.
Remind the reviewers that they need OpenReview profiles and wait for them to create them. You can run the Paper Matching Setup as many times as you want or until all users have completed profiles.
Note that when a reviewer creates a profile, their email address will not be automatically updated to their profile ID in the reviewers' group. The matcher will still detect email addresses as users without profiles, so any email addresses will either need to be removed or replaced with tilde IDs. This can be done automatically by re-running Paper Matching Setup.
You can confirm that the affinity scores were computed by checking if an invitation for the scores was created: https://api.openreview.net/edges?invitation=your/venue/id/role/-/Affinity_Score. Next, you should be able to run a paper matching from the ‘Paper Assignments’ link in your PC console.
In order to automatically assign Reviewers and Area Chairs, you must:
Enable the 'Review' or 'Post Submission' stage from your venue request form. This can only be done AFTER the submission deadline has passed.
After you complete these steps, a link for 'Paper Assignments' should appear on your Program Chair console.
Clicking on one of the assignment links will bring you to the assignment page, where you can create a new matching configuration. If members of your reviewer or area chairs group have profiles without publications, you will need to select ‘Yes’ for ‘Allow Zero Score Assignments’ in order to obtain a solution. Please note that all members of a group must have OpenReview profiles in order for the automatic assignment algorithm to run. Any members without profiles must be removed from the group before this step.
You can learn more about our automatic paper matching algorithm from its github repo: https://github.com/openreview/openreview-matcher. To create a new matching, click the 'New Assignment Configuration'. This will pull up a form with some default values pertaining to your matching settings:
After filling out the matching configuration form and hitting submit, you should see the following:
You can view, edit or copy the values you filled out in the matching form. When you are happy with your configuration, you should hit 'Run Matcher' and wait until its status is 'Complete'. This generates proposed assignments, with options to browse assignments, view statistics or deploy matching. If you click ‘Browse Assignments’ you will be brought to the edge browser, where you can browse, edit, and create proposed assignments.
If you get "No Solution" after running the matcher, you can view the configuration to see the entire error message. If the message is something like the following:
Error Message: Total demand (150) is out of range when min review supply is (34) and max review supply is (100)
that means that your constraints require more reviewers or area chairs than you currently have. The total demand is equal to (number of submissions * user demand) + (number of submissions * alternates). The max review supply is the number of reviewers available * max papers and the review supply is the number of reviewers available * min papers. Your total demand must fall within this range in order to obtain a solution.
Note that completion of this step does not make assignments, it only creates a proposed assignment configuration. Those assignments will need to be deployed before Reviewers or Area Chairs will see them.
The sets the readership of reviews.
The stage sets readership of submissions.
Use the 'Paper Matching Setup' button on your venue request form to
MinMax: Optimizes the scores while respecting the min and max quotas for each paper and reviewer. You can read more about MinMax .
Fairflow: Tries to make every match have at least some minimum affinity. You can read more about Fairflow .
Randomized: Generates randomized assignments and selects the assignment that maximizes expected total affinity without breaking the probability limits. You can read more about the Randomized solver .
You can read more about all solver options .
From the assignment page, click ‘Deploy assignments’ next to the matching configuration of your choice. Note that this will not notify group members of their assignments. You can contact different roles either through their group consoles or through the python client.
The edge browser is a tool for visualizing edges, or assignments, created by OpenReview’s automatic paper matching algorithm. You can use it to browse, sort, search, and create proposed assignments between reviewers and papers before deploying them.
When you first open the edge browser, all papers will appear in a column on the left. You can click on a certain paper to see a second column of reviewers pop up to the right. Similarly, if you click on a reviewer, all of their assigned papers will pop up in another column to the right, and so on.
The color of each item represents the relationship between that item and the one selected at left:
Light green means that the item is assigned to the item selected at left.
Light red means that the item has conflict with the item selected at left.
Light orange means that the item both has conflict and is assigned to the item selected at left.
White means that the item is not assigned to and has no conflict with the item selected at left.
Each item will display various edges calculated by the matcher and used to make assignments, such as the Bid, Affinity, Aggregate scores, and Conflicts. The trashcan button can be used to remove an edge. You can create new assignments using the ‘Assign’ button.
'Assignments' tells you how many papers are assigned to a given reviewer. You may also see 'Custom Max Papers' here if certain reviewers requested a specific max number of papers. You can filter out reviewers who have met their quota with the checkbox 'Only show reviewers with fewer than max assigned papers.' Once a reviewer has hit their quota, the 'Assign' button will be disabled and you will only be able to assign them additional papers using the 'Invite Assignments' button after deployment.
You can search for specific papers by paper title or number at the top of the first column. At the top of the subsequent columns you can also search for specific reviewers by profileID, name, or email. You can sort subsequent columns on the right by whatever edges are displayed, such as Assignment, Aggregate Score, Bid, Affinity Score, and/or Conflict, using the 'Order By' dropdown.
You can copy, edit, and create matching configurations as many times as you want until deployment. You can also use the ‘View Statistics’ button on the assignment page to view a breakdown of paper assignments. When you are happy with your assignments, you can deploy them.
The edge browser is a tool for visualizing edges, or matches, created by OpenReview's automatic paper matching algorithm. You can use it to browse, sort, search, and create new assignments between reviewers and papers until you are happy with the assignments generated.
Navigating the Edge Browser
If the option to modify assignments is available for Area Chairs in your venue, you should find a link to do so in your Area Chair console that will bring you directly to the edge browser.
All of your assigned papers will appear in a single column on the left. Clicking on a paper in the list will pull up a second column containing all reviewers, colored by their relationship to the selected paper:
Light green means that the reviewer is assigned to the paper selected at left.
White means that the reviewer is not assigned to and has no conflict with the paper selected at left.
You can search for specific papers by paper title or number at the top of the first column. At the top of the subsequent column you can also search for specific reviewers by their name, email or profile id. You can sort the column on the right by whatever edge types are shown, such as Assignment, Aggregate Score, Bid, or Affinity Score, using the 'Order By' dropdown. The second column will show the total number of assignments for the selected reviewer.
Creating and Removing Assignments Using the Edge Browser
You can delete an assignment using the trash can button on a certain Reviewer.
There are two ways to create assignments:
Using the 'Assign' button. This assigns a reviewer to a paper and notifies them by email. The assignment becomes automatically available in the Reviewer and AC consoles.
Using the 'Invite Assignment' button. This is only availble if it has been enabled by the Program Chairs. This sends an invitation to the reviewer with an accept/decline link. The reviewer can then respond to the invitation. If you want to invite a reviewer from outside the reviewer pool, including another Area Chair, you can do so by searching for their email address or profileID in the search bar of the second column and clicking 'Invite Assignment'. If they are in conflict with that paper, a banner will alert you with an error. Otherwise, they will receive an email notifying them of their invitation with the option to accept or reject the assignment. Their status will change according to their response to your invitation ('Declined', 'Pending Sign Up', 'Accepted', or 'Conflict Detected').
Some reviewers have a custom reduced paper load which appears in the edge browser as 'Custom Max Papers'. You cannot directly assign a reviewer to more papers than their custom max papers, but you can 'Invite' reviewers if that option is enabled for you. You can also filter out reviewers who have met their quota with the checkbox 'Only show reviewers with fewer than max assigned papers.'
There are two ways to assign members after assignments have been deployed:
The option 'Assign' directly assigns the reviewer to the paper and sends an email notification.
The option 'Invite Assignment' sends an invitation email to the reviewer with an accept/decline link. If you would like to use this option, you will need to contact info@openreview.net to enable its functionality.
Any changes made after deployment are immediately visible to the assigned Reviewers or Area Chairs, and it is not necessary to deploy again.
The reviewer can then respond to the invitation. If you want to invite a reviewer from outside the reviewer pool, you can do so by searching for their email address or profileID in the search bar of the second column and clicking 'Invite Assignment'. If they are in conflict with that paper, a banner will alert you with an error. Otherwise, they will receive an email notifying them of their invitation with the option to accept or reject the assignment. Their status will change according to their response to your invitation ('Declined', 'Pending Sign Up', 'Accepted', or 'Conflict Detected').