arrow-left

All pages
gitbookPowered by GitBook
1 of 4

Loading...

Loading...

Loading...

Loading...

Computing Conflicts Between Users

In general, conflicts can and should be computed using Paper Matching Setup. However, there may be cases where you do not want to re-compute conflicts for all reviewers and papers; for example, if an author requested to add new coauthors after reviewers were already assigned to their paper. You can use the python client to manually check for conflicts between the reviewers and the new authors like so:

  1. If you have not done so, you will need to install and instantiate the openreview-py client.

  2. Get the note that you are interested in computing conflicts for. If you have a double blind venue, go to api.openreview.net/notes?id=<submission_forum> and get the id listed for "original". If you have a single blind venue, you can pass in the forum.

3. Get the profiles of the authors of the submission and the reviewers. The reviewer group id should be something like your conference id/Paper#/Reviewers, for example robot-learning.org/CoRL/2022/Conference/Paper99/Reviewers if the submission of interest is paper number 99.

4. Compute the conflicts. If no conflicts are found, an empty array will be printed. Otherwise, the array will contain the shared groups that put them in conflict with each other.

Edges

Bids, assignments, affinity scores, conflicts, etc. are saved as Edges in OpenReview.

circle-info

You can read more about Edges in the reference section depending on the API version: ,

Simply speaking, Edges are links between two OpenReview entities (Notes, Groups, Profiles, etc.).

Posting a custom conflict edge

If you are using automatic matching, you can generate conflicts automatically with Paper Matching Setup. Sometimes Program Chairs additionally want to create custom conflicts between certain users and certain papers. This can be done by posting an edge to the conflict edge invitation for your venue.

  1. If you have not done so, you will need to .

  2. Determine the conflict invitation you will be using. If you are creating a custom conflict edge for a reviewer, it would be <your_venue_id>/Reviewers/-/Conflict. If it is for an Area Chair, it would be <your_venue_id>/Area_Chairs/-/Conflict.

note = client.get_note(<submission_id>)
author_profiles = openreview.tools.get_profiles(
    client,
    note.content["authorids"],
    with_publications=True
)
reviewers = openreview.tools.get_profiles(
    client,
    client.get_group(reviewer_group_id).members,
    with_publications=True
)
Besides the fields that define user permissions, an Edge would usually contain these fields: head, tail, weight, label.

For example, a OpenReview affinity score edge for a paper-reviewer pair may have the reviewer’s Profile id set in the edge.head field, paper id set in the edge.tail field, and OpenReview affinity score set in the edge.weight field.

All Edges respond to some OpenReview Invitation, which specifies the possible content and permissions that an Edge is required to contain.

Because the amount of Edges can be quite large, depending on the query used, you need to pass at least one of the following parameters: id, head, tail.

circle-info

For more information on the available parameters, check the corresponding OpenAPI reference: Edges V1, Edges V2.

Note that the groupby parameter is only available in the method get_grouped_edges described below.

Consider the following example which gets the first 10 Edges representing the “Personal” conflicts in ICLR 2020:

Note that since conflict data is sensitive, you may not have permissions to access conflict edges mentioned in the above example.

By default, get_edges will return up to the first 1000 corresponding Edges (limit=1000). To retrieve Edges beyond the first 1000, you can adjust the offset parameter, or use the function get_all_edges which returns a list of all corresponding Edges:

Since edges usually are very large in numbers, it is possible to get just the count of edges by using the function client.get_edges_count. Note that this only returns the count of edges visible to you.

Since most of the common tasks performed using Edges require Edges to be grouped, it’s also possible to query for already grouped Edges. Consider the following example that gets all reviewers grouped by papers they have conflicts with for the ICLR 2020 Conference.

circle-info

Note that in this case it's not necessary to pass id, head or tail as parameters. This is the advantage of using the get_grouped_edges method.

Consider the following example that gets all papers grouped by reviewers they are in conflict with for the ICLR 2020 Conference. It returns a list of lists of the {head, weight, label} of each conflict edge for that tail.

To group Edges, one must already know what the edge.head and edge.tail represent in an Edge and that information can be seen from the Edge’s invitation.

Edges V1
Edges V2
  • Set a variable 'tail' to the user you are generating the conflict for. Set a variable 'head' to the forum of the submission you want to create the conflict for. For example:

    1. Set the readers of this conflict. In general, this should include the tail of the edge and anyone who might be making assignments for this group, such as Area Chairs or Senior Area Chairs. You can confirm who the readers should be by going to openreview.net/invitation/edit?id=<conflict_invitation> and checking the readers in the reply field.

    1. Optionally set a label for your custom conflicts. This can help you query and retrieve them later.

    1. Finally, create and post an edge with a weight of -1 between the user and the paper. This will make the conflict a hard constraint.

    invitation = "<your_venue_id>/Reviewers/-/Conflict"
    install and instantiate the openreview-py client
    tail = "~User_One1"
    head = "ggVj9Mmq2-a"
    for reviewer in reviewers:
        print(reviewer.id)
        conflicts = openreview.tools.get_conflicts(
            author_profiles,
            reviewer,
            policy='default',
            n_years=5
        )
        print(conflicts)
    conflict_edges = client.get_edges(
        invitation='ICLR.cc/2020/Conference/-/Conflict',
        label='Personal',
        tail='~Test_User1',
        limit=10
    )
    conflict_edges_iterator = openreview.tools.iterget_edges(
        client,
        invitation='ICLR.cc/2020/Conference/Reviewers/-/Conflict',
        label='Personal',
        tail='~Test_User1'
    )
    conflict_edges_count = client.get_edges_count(
        invitation='ICLR.cc/2020/Conference/Reviewers/-/Conflict',
        label='Personal'
    )
    grouped_conflict_edges = client.get_grouped_edges(
        invitation='ICLR.cc/2020/Conference/Reviewers/-/Conflict',
        groupby='head',
        select='tail,weight,label'
    )
    grouped_conflict_edges = client.get_grouped_edges(
        invitation='ICLR.cc/2020/Conference/Reviewers/-/Conflict',
        groupby='tail',
        select='head,weight,label'
    )
    readers = [
        "~User_One1", 
        "your_venue_id",
        "<your_venue_id>/Area_Chairs"
    ]
    label = "Custom Conflict"
    client.post_edge(openreview.Edge(
       invitation = invitation,
       label = label, 
       weight = -1, 
       head = head, 
       tail = tail,
       signatures = [
        "<your_venue_id>"
       ],
       readers = [
        "<your_venue_id>"
       ],
       writers = [
        "<your_venue_id>"
    ]))

    Getting all Assignments for a User

    Assignments are stored as edges. You can view all of the assignment edges for a user in two ways:

    hashtag
    View them at api.openreview.net

    1. In your browser search bar, enter the following. If you want to view assignments for an Area Chair, change 'Reviewers' to 'Area_Chairs'. Replace your_venue_id with your full venue id. This will bring up all of the assignment edges for all of your venue's reviewers.

    2. Now we want to filter by a particular reviewer. If you look at a specific edge, you will see that the field "tail" corresponds to the profile ID of the assigned reviewer. So to see only the assignments for reviewer ~User_One1, change your search query to the following:

    hashtag
    Retrieve them with the Python Client

    1. If you have not done so, you will need to .

    2. Set a variable 'tail' to the profile ID of the user you are interested in, for example:

    3. Set a variable 'invitation' to the invitation of the edges you are trying to view. If your user of interest is a reviewer, the invitation would be in the format <your_venue_id>/Reviewers/-/Assignment.

    4. Retrieve all of the edges posted to that invitation for the user of interest.

    api.openreview.net/edges?invitation=<your_venue_id>/Reviewers/-/Assignment&groupBy=tail
    install and instantiate the openreview-py client
    api.openreview.net/edges?invitation=<your_venue_id>/Reviewers/-/Assignment&tail=~User_One1
    tail = "~User_One1"
    invitation = "<your_venue_id>/Reviewers/-/Assignment
    edges = client.get_all_edges(invitation = invitation, tail = tail)