There are currently two APIs supported. Venue organizers can decide what API version to use before it is deployed.

While most operations will work on both APIs, pay careful attention when that is not the case.

1. You will need to install the openreview-py client.

git clone [email protected]:openreview/openreview-py.git
cd openreview-py
pip3 install -e .

2. Create a client object with your OpenReview credentials. If you do not yet have an OpenReview profile, you will need to make one now.

import openreview

# API V1
client = openreview.Client(
    baseurl='https://api.openreview.net',
    username=<your username>,
    password=<your password>
)

# API V2
client = openreview.api.OpenReviewClient(
    baseurl='https://api2.openreview.net',
    username=<your username>,
    password=<your password>
)

Bids, assignments, affinity scores, conflicts, etc. are saved as Edges in OpenReview.

Simply speaking, Edges are links between two OpenReview entities (Notes, Groups, Profiles, etc.).

Besides the fields that define user permissions, an Edge would usually contain these fields: head, tail, weight, label.

For example, a OpenReview affinity score edge for a paper-reviewer pair may have the reviewer’s Profile id set in the edge.head field, paper id set in the edge.tail field, and OpenReview affinity score set in the edge.weight field.

All Edges respond to some OpenReview Invitation, which specifies the possible content and permissions that an Edge is required to contain.

Because the amount of Edges can be quite large, depending on the query used, you need to pass at least one of the following parameters: id, head, tail.

Consider the following example which gets the first 10 Edges representing the “Personal” conflicts in ICLR 2020:

Note that since conflict data is sensitive, you may not have permissions to access conflict edges mentioned in the above example.

By default, get_edges will return up to the first 1000 corresponding Edges (limit=1000). To retrieve Edges beyond the first 1000, you can adjust the offset parameter, or use the function get_all_edges which returns a list of all corresponding Edges:

Since edges usually are very large in numbers, it is possible to get just the count of edges by using the function client.get_edges_count. Note that this only returns the count of edges visible to you.

Since most of the common tasks performed using Edges require Edges to be grouped, it’s also possible to query for already grouped Edges. Consider the following example that gets all reviewers grouped by papers they have conflicts with for the ICLR 2020 Conference.

Consider the following example that gets all papers grouped by reviewers they are in conflict with for the ICLR 2020 Conference. It returns a list of lists of the {head, weight, label} of each conflict edge for that tail.

To group Edges, one must already know what the edge.head and edge.tail represent in an Edge and that information can be seen from the Edge’s invitation.

In general, conflicts can and should be computed using Paper Matching Setup. However, there may be cases where you do not want to re-compute conflicts for all reviewers and papers; for example, if an author requested to add new coauthors after reviewers were already assigned to their paper. You can use the python client to manually check for conflicts between the reviewers and the new authors like so:

1. If you have not done so, you will need to install and instantiate the openreview-py client.

2. Get the note that you are interested in computing conflicts for. If you have a double blind venue, go to api.openreview.net/notes?id=<submission_forum> and get the id listed for "original". If you have a single blind venue, you can pass in the forum.

3. Get the profiles of the authors of the submission and the reviewers. The reviewer group id should be something like your conference id/Paper#/Reviewers, for example robot-learning.org/CoRL/2022/Conference/Paper99/Reviewers if the submission of interest is paper number 99.

4. Compute the conflicts. If no conflicts are found, an empty array will be printed. Otherwise, the array will contain the shared groups that put them in conflict with each other.

Assignments are stored as edges. You can view all of the assignment edges for a user in two ways:

View them at api.openreview.net

1. In your browser search bar, enter the following. If you want to view assignments for an Area Chair, change 'Reviewers' to 'Area_Chairs'. Replace your_venue_id with your full venue id. This will bring up all of the assignment edges for all of your venue's reviewers.

2. Now we want to filter by a particular reviewer. If you look at a specific edge, you will see that the field "tail" corresponds to the profile ID of the assigned reviewer. So to see only the assignments for reviewer ~User_One1, change your search query to the following:

Retrieve them with the Python Client

1. If you have not done so, you will need to .

2. Set a variable 'tail' to the profile ID of the user you are interested in, for example:

3. Set a variable 'invitation' to the invitation of the edges you are trying to view. If your user of interest is a reviewer, the invitation would be in the format <your_venue_id>/Reviewers/-/Assignment.

4. Retrieve all of the edges posted to that invitation for the user of interest.

If you are using automatic matching, you can generate conflicts automatically with Paper Matching Setup. Sometimes Program Chairs additionally want to create custom conflicts between certain users and certain papers. This can be done by posting an edge to the conflict edge invitation for your venue.

1. If you have not done so, you will need to .

2. Determine the conflict invitation you will be using. If you are creating a custom conflict edge for a reviewer, it would be <your_venue_id>/Reviewers/-/Conflict. If it is for an Area Chair, it would be <your_venue_id>/Area_Chairs/-/Conflict.

To get all meta-reviews for a venue, you can do the following:

1. Get all submissions for your venue. You can do this by passing your venue's submission invitation into get_all_notes. You should also pass in details = "directReplies" to obtain any notes that reply to each submission.

2. For each submission, add any replies with the Meta-Review invitation to a list of meta-reviews.

3. The metareviews list now contains all of the meta-reviews for your venue.

Program committee is represented as group, like Reviewers, Area Chairs, Action Editors, etc. Each of these group has a property members that can contain a set of groups. Email addresses and profile ids are also reprensted as groups. If you want to build you own group of reviewers you can just simply add them as members of the group.

You have two options to edit the Reviewers group:

1. Using the group editor. You can find the group editor using the url: https://openreview.net/group/edit?id=group_id

You can get a list of all of the venues in OpenReview like so:

venues =  client.get_group(id='venues').members
print(venues)

To get all decisions for a venue, you can do the following:

1. Get all submissions for your venue. You can do this by passing your venue's submission invitation into get_all_notes. You should also pass in details = "directReplies" to obtain any notes that reply to each submission.

submissions = client.get_all_notes(
    invitation="Your/Venue/ID/-/Submission",
    details='directReplies'
)

2. For each submission, add any replies with the Decision invitation to a list of decisions.

decisions = [] 
for submission in submissions:
    decisions = decisions + [reply for reply in submission.details["directReplies"] if reply["invitation"].endswith("Decision")]

3. The decisions list now contains all of the decisions for your venue.

Some venues with multiple deadlines a year may want to reuse the same reviewer and area chairs from cycle to cycle. In those cases, it is useful to use the python client to copy group members from one group to another rather than recruiting the same people each time.

1. If you have not done so, you will need to install and instantiate the openreview-py client.

2. Get the group you are taking members from. You can get an individual group by its id. If you are copying reviewers from one venue iteration to another, for example, do the following:

first_reviewer_group = client.get_group("<first_venue_id>/Reviewers")

3. Each group has a field 'members' which is a list of profile IDs or emails belonging to the members of that group. Extract the members:

4. Finally, you can use add_members_to_group to add those members to your new Reviewers group.

Most data in OpenReview are represented as Notes. Each Note responds to an Invitation, which specifies the possible content and permissions that the Note is required to contain.

Users can query notes using the ID of the Invitation that they respond to.

Consider the following example which gets the public Notes that represent the 11th through 20th submissions to ICLR 2019:

By default, get_notes will return up to the first 1000 corresponding Notes (limit=1000). To retrieve Notes beyond the first 1000, you can adjust the offset parameter, or use the function client.get_all_notes which returns a list of all corresponding Notes:

Venues using API V1

To get all reviews for a double-blind venue, you can do the following:

While a support request form can most easily be submitted through the UI, some venues that have multiple deadlines a year and need to submit multiple venue requests with the same settings may find it easier to do this programmatically through the API.

1. If you have not done so, you will need to install and instantiate the openreview-py client.

2. Familiarize yourself with the venue request form. Note which fields are required, and which are optional. You can view the form through the UI or view it in JSON.

3. Choose your program chairs, and create a list of their email addresses.

4. Now it is time to choose the settings for your venue. These make up the 'content' field of your note. Go through the fields on the form in UI, and cross reference the JSON of the invitation to make sure each key matches that in the invitation exactly. For the Program Chairs field, you can enter your program_chair_emails list. For example:

5. Now create your openreview . You need to include the invitation for the request form, as well as signatures, readers, writers, and content. Your signature should be your OpenReview profile ID that is linked to the email address you entered in the program_chair_emails.

6. Post your note.

7. You can check for your support request here:

Querying profiles

You can retrieve an individual's OpenReview profile object by their name or email:

profile = client.get_profile('~Michael_Spector1')
profile = client.get_profile('[email protected]')

If you want to query more than one profile at a time, you can use our tools module:

profiles = openreview.tools.get_profiles(
    client,
    ids_or_emails=['[email protected]',
    '~Melisa_bok1'
]

If you want to get all the profiles and their publication, you can use the previous call and add the parameter with_publications=True

Finding profile relations

Relations can be extracted in two ways: (1) from the Profile object itself, or (2) from coauthored Notes in the system.

Getting stored relations:

Getting coauthorship relations from Notes:

The submission deadline set through the venue request form is actually only the advertised due date. The true deadline is the expiration date, which is set to 30 minutes after the submission deadline. If you would like to change the expiration date to allow the authors more or less wiggle room around the submission deadline, you can do so using the python client.

1. If you have not done so, you will need to .

2. Choose your desired expiration date. You will need to convert it into epoch time in milliseconds using an For example:

Say you want to export all of the reviews for a given venue into a csv file.

1. If you have not done so, you will need to install and instantiate the openreview-py client.

2. Retrieve all of the Reviews. Reviews generally follow the invitation Your/Venue/ID/-/Official_Review. We can retrieve them by getting all of the direct replies to each submission and finding those with invitations ending in Official_Review.

Single blind venues can do this like so:

whereas double blind venues should replace "Submission" in the invitation with "Blind_Submission":

3. Next, get the super review invitation. This is the overall review invitation which each of the Paper#/-/Official_Review invitations are based off of, and it follows the format Venue/ID/-/Official_Review.

4. Generate a list of the fields in the content in the Review invitation. For reference, this is what the default review invitation content looks like in JSON:

so we would expect a list like ["title", "review", "rating", "confidence"]. This is how we get the list:

5. If you haven't already, import csv. Then iterate through the list of reviews stored in 'reviews' and for each one, append the values associated to the keys in your keylist. If a value does not exist for that key, put an empty string in its place.

6. The previous example only exports the content fields of each review. You may also want to know which submission each review is associated with. You can get the forum of each review, which corresponds to the forum page of its associated submission. For example, if a review's forum is aBcDegh, you could find that submission at https://openreview.net/forum?id=aBcDegh. To create a csv that includes the review forums, do this:

7. There should now be a csv of exported reviews in the directory in which you are working.

If you are testing your venue on the dev site, you may want to generate some test submissions. This can be accomplished manually through the UI, but it may be faster to do it with python using the guide below.

Venues using API V1

1. If you have not already, you will need to create a username and password on the dev site: . Note that sending emails through the dev site is not supported. Therefore, you will need to request the activation link at .

2. Instantiate the python client using your dev profile credentials:

3. Familiarize yourself with the submission invitation. Every content field of your note will need to be in the format specified by the submission invitation, and any fields marked as "required" will need to be present in the note.

4. Create your note. If you were submitting a note through the , it would look something like this:

   Invitation: The ID of your venue's submission invitation. It will take the form of your venue id + /-/Submission. Readers/Writers: A list of the venue ID + all author profile IDs. After the submission deadline, these values will be automatically updated to match the visibility settings selected in the venue request form. Signatures: Your profile ID. Content: The actual content of the submission, which must match the invitation. If the invitation calls for any file uploads, such as a pdf or zip file, you can build the url to the file using put_attachment and entering the path, the invitation ID, and the name of the field. You can then add that field to the note's content. The "authorids" field of the note's content should contain a list of the authors' profile IDs or emails, in the same order as the authors' names. If you registered dummy users, you can find their profile IDs by going to their profile page and copying the ID parameter of that page's url. For example, dev.openreview.net/profile?id=~Test_Author2 --> ~Test_Author2.

5. Post your note and output the resulting note's ID:

   You can now view your note in the UI by going to dev.openreview.net/forum?id=<note_id>.

Venues using API V2

• If you have not already, you will need to create a username and password on the dev site: . Note that sending emails through the dev site is not supported. Therefore, you will need to request the activation link at .

• Instantiate the python client using your dev profile credentials: