Bids, assignments, affinity scores, conflicts, etc. are saved as Edges in OpenReview.
Simply speaking, Edges are links between two OpenReview entities (Notes, Groups, Profiles, etc.).
Besides the fields that define user permissions, an Edge would usually contain these fields: head, tail, weight, label.
For example, a OpenReview affinity score edge for a paper-reviewer pair may have the reviewer’s Profile id set in the edge.head field, paper id set in the edge.tail field, and OpenReview affinity score set in the edge.weight field.
All Edges respond to some OpenReview Invitation, which specifies the possible content and permissions that an Edge is required to contain.
Because the amount of Edges can be quite large, depending on the query used, you need to pass at least one of the following parameters: id
, head
, tail
.
Consider the following example which gets the first 10 Edges representing the “Personal” conflicts in ICLR 2020:
Note that since conflict data is sensitive, you may not have permissions to access conflict edges mentioned in the above example.
By default, get_edges will return up to the first 1000 corresponding Edges (limit=1000). To retrieve Edges beyond the first 1000, you can adjust the offset parameter, or use the function get_all_edges which returns a list of all corresponding Edges:
Since edges usually are very large in numbers, it is possible to get just the count of edges by using the function client.get_edges_count. Note that this only returns the count of edges visible to you.
Since most of the common tasks performed using Edges require Edges to be grouped, it’s also possible to query for already grouped Edges. Consider the following example that gets all reviewers grouped by papers they have conflicts with for the ICLR 2020 Conference.
Note that in this case it's not necessary to pass id
, head
or tail
as parameters. This is the advantage of using the get_grouped_edges
method.
Consider the following example that gets all papers grouped by reviewers they are in conflict with for the ICLR 2020 Conference. It returns a list of lists of the {head, weight, label} of each conflict edge for that tail.
To group Edges, one must already know what the edge.head and edge.tail represent in an Edge and that information can be seen from the Edge’s invitation.
Assignments are stored as edges. You can view all of the assignment edges for a user in two ways:
In your browser search bar, enter the following. If you want to view assignments for an Area Chair, change 'Reviewers' to 'Area_Chairs'. Replace your_venue_id with your full venue id. This will bring up all of the assignment edges for all of your venue's reviewers.
2. Now we want to filter by a particular reviewer. If you look at a specific edge, you will see that the field "tail" corresponds to the profile ID of the assigned reviewer. So to see only the assignments for reviewer ~User_One1, change your search query to the following:
If you have not done so, you will need to install and instantiate the openreview-py client.
Set a variable 'tail' to the profile ID of the user you are interested in, for example:
3. Set a variable 'invitation' to the invitation of the edges you are trying to view. If your user of interest is a reviewer, the invitation would be in the format <your_venue_id>/Reviewers/-/Assignment.
4. Retrieve all of the edges posted to that invitation for the user of interest.
In general, conflicts can and should be computed using Paper Matching Setup. However, there may be cases where you do not want to re-compute conflicts for all reviewers and papers; for example, if an author requested to add new coauthors after reviewers were already assigned to their paper. You can use the python client to manually check for conflicts between the reviewers and the new authors like so:
If you have not done so, you will need to install and instantiate the openreview-py client.
Get the note that you are interested in computing conflicts for. If you have a double blind venue, go to api.openreview.net/notes?id=<submission_forum> and get the id listed for "original". If you have a single blind venue, you can pass in the forum.
3. Get the profiles of the authors of the submission and the reviewers. The reviewer group id should be something like your conference id/Paper#/Reviewers, for example robot-learning.org/CoRL/2022/Conference/Paper99/Reviewers if the submission of interest is paper number 99.
4. Compute the conflicts. If no conflicts are found, an empty array will be printed. Otherwise, the array will contain the shared groups that put them in conflict with each other.
If you are using automatic matching, you can generate conflicts automatically with Paper Matching Setup. Sometimes Program Chairs additionally want to create custom conflicts between certain users and certain papers. This can be done by posting an edge to the conflict edge invitation for your venue.
Determine the conflict invitation you will be using. If you are creating a custom conflict edge for a reviewer, it would be <your_venue_id>/Reviewers/-/Conflict. If it is for an Area Chair, it would be <your_venue_id>/Area_Chairs/-/Conflict.
Set a variable 'tail' to the user you are generating the conflict for. Set a variable 'head' to the forum of the submission you want to create the conflict for. For example:
Set the readers of this conflict. In general, this should include the tail of the edge and anyone who might be making assignments for this group, such as Area Chairs or Senior Area Chairs. You can confirm who the readers should be by going to openreview.net/invitation/edit?id=<conflict_invitation> and checking the readers in the reply field.
Optionally set a label for your custom conflicts. This can help you query and retrieve them later.
Finally, create and post an edge with a weight of -1 between the user and the paper. This will make the conflict a hard constraint.
If you have not done so, you will need to .