How to setup paper matching by calculating affinity scores and conflicts
Last updated
Last updated
The Setup Matching is the first step needed to make assignments between Senior Area Chairs, Area Chairs, Reviewers and submissions. Once this step is completed you can run the Matching following the instructions here.
You can calculate affinity scores and conflicts for your venue using OpenReview's 'Paper Matching Setup' feature. Paper Matching Setup is enabled for any venue and the button is activated once the Submission Deadline has passed. This feature allows Program Chairs to compute or upload affinity scores and/or compute conflicts.
Calculating affinity scores can be lengthy process depending on the size of your venue. Therefore, only one setup matching can be run at a time.
You can find the 'Paper Matching Setup' button on your venue request form. The button will become available after the submission deadline.
Clicking it should bring up the following form. The 'Matching Group' is a dropdown menu of the groups you can use in the matcher (Reviewers, Area Chairs, Senior Area Chairs), depending on whichever you selected for your venue. You can select if you would like affinity scores and/or conflicts computed. Alternatively, you can compute and upload your own affinity scores using the OpenReview expertise API.
It is important that all Senior Area Chairs, Area Chairs, and Reviewers have a complete and updated Profile including their publications. Users can import publications from DBLP by editing their Profile. This will improve the quality of the affinity scores and conflicts.
Senior Area Chairs assignments can be done in two different ways (specified in the request form):
Senior Area Chairs are assigned submissions directly: This will compute affinity scores and conflicts between Senior Area Chairs and submissions.
Assignment to submissions through Area Chairs: Senior Area Chairs are assigned to Area Chairs and the submissions assigned to their Area Chairs are assigned to them. For this reason, when assigning Area Chairs to submissions, the corresponding Senior Area Chair conflicts need to be transferred to the Area Chairs. This guarantees that there are no conflicts between the submission and the assigned Area Chair and Senior Area Chair. Selecting this option will compute affinity scores between Senior Area Chairs and Area Chairs and conflicts between Senior Area Chairs and submissions. It is also required that the assignments between Senior Area Chairs and Area Chairs is done before starting the matching between Area Chairs and submissions.
If your venue has Senior Area Chairs and you are performing Senior Area Chair assignments based on their Area Chairs, make sure that the Area Chairs already have assigned Senior Area Chairs. This is needed so that the Senior Area Chair conflicts are transferred to the Area Chairs too. Refer to the Selecting Senior Area Chairs as the Matching Group section for more information
This will compute affinity scores and conflicts between Area Chairs and submissions.
This will compute affinity scores and conflicts between Reviewers and submissions.
Conflict detection uses the information of the users' coauthors from publications in OpenReview as long as they are publicly visible and the users' Profile. Therefore, the more complete and accurate the information in the Profile is, the better the conflict detection.
The sections of the Profile used for conflict detection are the Emails section, the Education & Career History section, and the Advisors, Relations & Conflicts section.
Another parameter that can be controlled is the amount of years you want to consider when looking at conflicts. For example, there might be two users who worked at company A at some point. However, User A worked at Company C ten years ago and User B just started working at Company C. If the amount of years is set to 5, for example, a conflict won't be detected between User A and User B because only the history, relations and publications from the past 5 years will be taken into consideration. By default, all relations, history, and publications are taken into consideration for conflict detection.
Since a lot of users use email services such as gmail.com, a list of common domains is used to filter them out before conflicts are computed.
There are two policies when computing conflicts: Default and NeurIPS.
Uses the domains and computes subdomains from the Education & Career History section.
Uses the domains and computes subdomains from the emails listed in the Advisors, Relations & Conflicts section.
Uses the domains and computes subdomains from the emails listed in the Emails section.
Uses the publication ids in OpenReview that the user authored.
Note that emails do not have a range of dates for when they were valid in the user's Profile. The NeurIPS policy addresses this issue.
Uses the domains and computes subdomains from the Education & Career History section. All intern positions are ignored.
Uses the domains and computes subdomains from the emails listed in the Advisors, Relations & Conflicts section, if the relation is that of a Coworker or a Coauthor.
Uses the domains and computes subdomains from the emails listed in the Emails section, if and only if no domains were extracted from the Education & Career History and Advisors, Relations & Conflicts sections.
Uses the publication ids in OpenReview that the user authored.
Depending on the value you selected for Compute Conflicts N Years, part or all the Profile information will be considered when computing conflicts. For example, if you use the value 5, only the most recent 5 years of the Profile's data will be used to compute conflicts.
Once all the information is extracted from the users' Profiles, the following rules apply to find a conflict between User A and User B:
If any of the domains/subdomains from the Education & Career History section of user A matches at least one of the domains/subdomains of the same section of User B, then a conflict is detected.
If any of the domains/subdomains from the Advisors, Relations & Conflicts section or the Emails section of user A matches at least one of the domains/subdomains of the same sections of User B, then a conflict is detected.
If any of the publications of User A is the same as one of the publications of User B. In other words, if User A and User B are coauthors, then a conflict is detected.
OpenReview has different models available to compute affinity scores between users and submissions. The current available models are:
specter+mfr
specter2
scincl
specter+scincl
If you want to learn more about the models, you can find the open source repository here.
You may also choose to upload your scores by selecting No. Directions for how to format uploaded scores are in the description of the field in the Paper Matching Setup.
Running the paper matching setup should output a comment on your venue request page. If there were members missing profiles or publications, the message will identify them and say 'Affinity scores and/or conflicts could not be computed for these users. Please ask these users to sign up in OpenReview and upload their papers. Alternatively, you can remove these users from the Reviewers group.' This message does not mean that the process failed, but that those members were excluded from the calculations. You have two options:
Remove reviewers without profiles from the reviewers group.
Remind the reviewers that they need OpenReview profiles and wait for them to create them. You can run the Paper Matching Setup as many times as you want or until all users have completed profiles.
Note that when a reviewer creates a profile, their email address will not be automatically updated to their profile ID in the reviewers' group. The matcher will still detect email addresses as users without profiles, so any email addresses will either need to be removed or replaced with tilde IDs. This can be done automatically by re-running Paper Matching Setup.
You can confirm that the affinity scores were computed by checking if an invitation for the scores was created: https://api.openreview.net/edges?invitation=your/venue/id/role/-/Affinity_Score. Next, you should be able to run a paper matching from the ‘Paper Assignments’ link in your PC console.
