arrow-left

All pages
gitbookPowered by GitBook
1 of 11

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Data Retrieval and Modification

How to View Camera-Ready Revisions

How to Update a Reviewer's Custom Max Papers

How to add/remove bids programatically

How to Sync Manual and Automatic Assignments

How to Export all Submission Attachments

How to Undo Deployed Assignments

How to loop through Accepted Papers and print the Authors and their Affiliations
How to add/remove fields from a submission
How to get all the Reviews that I have written and their corresponding Submissions
How to get all Registration Notes

How to Update a Reviewer's Custom Max Papers

Reviewers have the option to submit a form requesting a custom number of papers to review. The range of this custom number is defined by the organizers of the venue. Reviewers can set this vale during the recruitment period by submitting a form that creates a Note with the Invitation <your_venue_id>/Reviewers/-/Reduced_Load. When paper matching setup is run, this note is converted to an Edge with the Invitation <your_venue_id>/Reviewers/-/Custom_Max_Papers.

To create a reviewers' Custom Max Papers after having run Paper Matching Setup, you can do the following:

  1. If you have not done so, you will need to install and instantiate the openreview-py client.

  2. Create an Edge object with proper params as in the example below:

circle-info

Creating an Edge varies depending on the API version you are using.

For API V1 use openreview.Edge

For API V2 use openreview.api.Edge

Please make sure the tail value is the profile id of the user. To get the profile id you can use the following call:

To change a reviewers' Custom Max Papers after having run Paper Matching Setup, you can do the following:

  1. If you have not done so, you will need to .

  2. Retrieve the custom max papers edge for the reviewer. This edge will have the invitation <your_venue_id>/Reviewers/-/Custom_Max_Papers and the reviewer's profile ID as its tail, so you can retrieve it like so:

The edge has a "weight" field. This represents the custom amount of papers they have agreed to review. To change it, you can set it to the new desired number of papers and repost the Edge.

Once the edge is posted, it will appear in the reviewers console as the following image:

install and instantiate the openreview-py client
Reviewer Console
# This example works for API V2
client.post_edge(openreview.api.Edge(
    invitation='<your_venue_id>/Reviewers/-/Custom_Max_Papers',
    head='<your_venue_id>/Reviewers',
    tail='~Reviewer_ICMLOne1',
    signatures=['<your_venue_id>/Program_Chairs'],
    weight=2
))
profile = openreview.tools.get_profile('[email protected]')
profile.id
edges = client.get_edges(
    invitation="<your_venue_id>/Reviewers/-/Custom_Max_Papers",
    tail="~User_One1"
)
custom_max_papers_edge = edges[0]
edge.weight = 2
client.post_edge(edge)

How to loop through Accepted Papers and print the Authors and their Affiliations

  1. If you have not done so, you will need to install and instantiate the openreview-py client.

  2. Get all the accepted papers for your venue. The value for 'venue' is the Abbreviated Venue Name which can be located on the venue request form and the venue homepage underneath the official venue name. For example, 'SaTML 2023' is an abbreviated venue name, be sure to include yours in its place.

accepted_papers = client.get_all_notes(content={ 'venue': 'SaTML 2023'})

3. Iterate through each accepted paper, printing the number and title of the paper. To get the affiliations for each author, get the profiles by passing the author IDs in the content of the accepted paper. Iterate through the author profile and print the author's preferred name and history containing the affiliation.

How to Sync Manual and Automatic Assignments

Using manual assignment adds Reviewers to PaperX/Reviewers groups without posting an assignment edge. If you originally used automatic assignment but then add assignments manually through the PC console instead of the edge browser, your groups and assignment edges will become out of sync. Here is how you can restore them:

  1. If you have not done so, you will need to install and instantiate the openreview-py client.

  2. Retrieve all of the assignment edges and all of the Paper#/Reviewers groups for your venue. Make two dictionaries: one mapping groups by their IDs, the other mapping edges by their head. The head of each edge is the forum of the paper that the reviewer is assigned to.

edges = client.get_all_edges(invitation = 'Your/Venue/ID/Reviewers/-/Assignment')
papers = client.get_all_notes(invitation = 'Your/Venue/ID/-/Blind_Submission')
groups = [client.get_group(f"Your/Venue/ID/Paper{paper.number}/Reviewers")for paper in papers]
groups_by_ids = {group.id: group for group in groups}
edges_by_paper = {}
for edge in edges: 
    if edge.head in edges_by_paper: 
        edges_by_paper[edge.head].append(edge.tail)
    else: 
        edges_by_paper[edge.head] = [edge.tail]

3. Iterate through all assignment edges. For each one, check that the reviewer in the assignment edge is in the correct PaperX/Reviewers group. For example, if an edge has a tail of ~User_One1 and a head corresponding to Paper2, then Paper2/Reviewers should include ~User_One1. If the reviewer is not in the correct group, add them.

4. Now we want to check for the opposite case, where a Reviewer has been added to a group but there is not an associated assignment edge. Retrieve all papers, then create a dictionary with paper numbers as the keys. Iterate through the papers, get the associated Reviewers' group for each, and check that there is an assignment edge for each group member. If there is not, remove the reviewer.

How to get all Registration Notes

If you enabled the Registration Stage for reviewers and/or area chairs, you will be able to programatically query these registration notes using the .

  1. Instantiate your OpenReview client

  1. Get all the registration notes for your venue. The invitation is composed of your venue's id (found in the request form), the group you want to query notes for (Reviewers

How to add/remove bids programmatically

Bids are represented by Edges where the head is the Note id of the submission and the tail is the Profile id of the user that is doing the bid. Bids have an optional property label and in this case it is being used to represent the type of bid, for example: "Very High", "High", "Neutral", "Low" and "Very Low".

Members of the official committee usually use the "Bidding Console" to create and edit these bids. However, bids can also be updated programmatically.

How to Undo Deployed Assignments

If you deployed automatic assignments but would like to roll them back, you can do it using the python client.

  1. If you have not done so, you will need to .

  2. You will first need to delete all of the assignment edges. You can do this by getting all edges with your venue's assignment invitation and then setting a ddate for each one.

3. Now all of the assignments have been reverted, but before you can deploy a new matching you will need to change the status of the deployed matching configuration from 'Deployed' to 'Complete'. In order to do this, you will first need to find the matching configuration ID. Go to api.openreview.net/notes?invitation=Your/Venue/ID/Reviewers/-/Assignment_Configuration and find the note with a status of 'Deployed'. Once you have its ID, do the following:

for accepted_paper in accepted_papers:
    print(accepted_paper.number, accepted_paper.content['title'])
    author_profiles = openreview.tools.get_profiles(client, accepted_paper.content['authorids'])
    for author_profile in author_profiles:
        print(author_profile.get_preferred_name(pretty=True), author_profile.content.get('history', [{}])[0])
count = 0
for edge in tqdm(edges): 
    # Check if reviewer is in paperX reviewer group 
    paper = client.get_note(edge.head)
    reviewer_group = groups_by_ids[f'Your/Venue/ID/Paper{paper.number}/Reviewers']
    if edge.tail not in reviewer_group.members: 
        print(edge.tail, reviewer_group.id, paper.forum)
        count = count+1
        # If edge exists but reviewer is not in group, add reviewer to group 
        reviewer_group = client.add_members_to_group(reviewer_group, edge.tail)
print(count)
papers_by_number = {}
for paper in papers: 
    papers_by_number[paper.number] = paper.forum
count = 0
# Go through every PaperX/Reviewers group and for each one check if reviewers have an edge 
# go through each paper, for each one get the reviewers group, for each reviewer in the group see if the edge exists 
for number, forum in papers_by_number.items(): 
    reviewers_group = groups_by_ids[f'Your/Venue/ID/Paper{number}/Reviewers']
    tails = edges_by_paper[forum]
    for reviewer in reviewers_group.members: 
        if reviewer not in tails: 
            count = count + 1
            print(reviewer, reviewers_group.id, forum)
            reviewers_group = client.remove_members_from_group(reviewers_group, reviewer)
print(count)
or
Area_Chairs
) and the
(by default, this is Registration):

{venue_id}/(Reviewers|Area_Chairs)/-/{registration_stage_name} , e.g., NeurIPS.cc/2023/Conference/Reviewers/-/Registration

  1. Iterate through every note to access the note's content. You will be able to access all fields you configured for the registration stage.

client = openreview.api.OpenReviewClient(
    baseurl='https://api2.openreview.net',
    username=<your username>,
    password=<your password>
)
python clientarrow-up-right
registration_notes = client.get_all_notes(invitation='NeurIPS.cc/2023/Conference/Reviewers/-/Registration')
name of the registration stagearrow-up-right
hashtag
Editing bids:

Every time you update a bid you need to set the signatures. The signatures must be a group id where you have signatory permissions. The bid Invitation specifies that the signature must be either a profile id or the venue id. In the example below, the signature is "ICML.cc/2023/Conference":

hashtag
Removing bids:

To remove bids, you need to a the ddate value using a timestamp in milliseconds. This change will perform a soft delete which means that this bid can be restored.

hashtag
Restoring bids:

To restore bids, you need to get the deleted bid and remove the ddate value. This is achieved by passing an object { delete: true } in case you are using API V2. Be aware that the invitation must allow a this operation by passing an object with "deletable: true", otherwise, this action cannot be performed.

If you are using API V1, then you need to pass the value None instead.

4. Now you should have the option to deploy a new matching configuration.

edges = client.get_all_edges(invitation = 'Your/Venue/ID/Reviewers/-/Assignment')
for edge in edges: 
    edge.ddate = 1643083220000
    client.post_edge(edge)
install and instantiate the openreview-py client
note = client.get_note('<note_id>')
note.content['status'] = 'Complete'
client.post_note(note)
for note in registration_notes:
    print(note.content['expertise_confirmed']['value'])
edge = client.get_edge(bid_id)
edge.label = 'Very High'
edge.signatures = ["ICML.cc/2023/Conference"]
edge.nonreaders = None
client.post_edge(edge)
edge = client.get_edge(bid_id)
edge.ddate = 1664467200000
edge.signatures = ["ICML.cc/2023/Conference"]
edge.nonreaders = None
client.post_edge(edge)
edge = client.get_edge(bid_id)
edge.ddate = { 'delete': True }
edge.signatures = ["ICML.cc/2023/Conference"]
edge.nonreaders = None
client.post_edge(edge)
edge = client.get_edge(bid_id)
edge.ddate = None
edge.signatures = ["ICML.cc/2023/Conference"]
edge.nonreaders = None
client.post_edge(edge)

How to view Camera-Ready Revisions

If you enabled a Submission Revision Stage to allow authors to revise their submissions or submit Camera-Ready Revisions, the authors will have used these invitations to directly revise their existing submissions. This means that in order to see the final versions you can simply go to the forum of each submission. You can also click 'show revisions' to see the revision history for each paper.

If you need a way to see which authors have submitted camera-ready revisions, you can use the following code:

  1. You will first need to .

Next, retrieve all of the revision invitations for your venue. You will want to replace the super invitation with the submission revision invitation for your venue, which is your conference id + /-/ + the name you chose for your Submission Revision stage.

Note: if the deadline has passed for revisions, add the parameter expired=True to the call to get expired invitations.

2. Create a dictionary mapping the submission number to each submission for your venue. Replace the submission invitation with that of your venue:

Iterate through all of the camera-ready revision invitations and for each one, try to get the revisions made under that invitation. If there aren't any, don't add them to the dictionary revisions by forum. Finally, print the number of revision invitations vs the number of actually completed revisions so that you know how many papers are missing revisions.

install and setup the python clientarrow-up-right
revision_invitations = list(openreview.tools.iterget_invitations(client, super = 'Your/Conference/ID/-/Camera_Ready_Revision'))
submissions_by_number = {p.number: p for p in client.get_all_notes(invitation = 'Your/Conference/ID/-/Submission')}
revisions_by_forum = {}
for invitation in revision_invitations: 
    number = int((invitation.id.split('/-/')[0]).split('Paper')[1])
    submission = submissions_by_number[number]
    references = client.get_references(referent = submission.id, invitation = invitation.id)
    if references:
        revisions_by_forum[submission.forum] = references
    else:
        print(f'no revisions for {number}')
print(f'Number of revision invitations: {len(revision_invitations)}')
print(f'Number of revisions submitted: {len(revisions_by_forum.keys())}')

How to get all the Reviews that I have written and their corresponding Submissions

OpenReview does not currently provide an official document that certifies that a Reviewer has participated in the review process of a conference. However, it is still possible to retrieve the public reviews and public submissions that the Reviewer reviewed using the python client from OpenReview.

hashtag
Steps for getting Reviews

  1. Instantiate your OpenReview client

  1. Run the following method

How to add/remove fields from a submission

circle-info

This documentation is only valid for the API V2

hashtag
Editing a submission as an organizer

client = openreview.api.OpenReviewClient(
    baseurl='https://api2.openreview.net',
    username=<your username>,
    password=<your password>
)
result = openreview.tools.get_own_reviews(client)

# Result is a list of dictionaries that have the following schema
# {
#     'submission_title': <Submission title>,
#     'submission_link': <Link to submission>,
#     'review_link': <Link to review>
# }
Organizers of the venue have access to an Invitation that lets them edit any part of a submission associated to their venue. This Invitation is called "Meta Invitation".

Meta Invitations are admin Invitations which are created when the venue is deployed and are available to venue organizers. There is only 1 available per venue and should be used with care and sparingly. Therefore, before using the Meta Invitation, make sure there is no other Invitation that can be used instead to avoid errors when modifying a submission.

hashtag
Removing fields

Fields can be removed from the note using the value { delete: true }

Only the Meta Invitation, which usually has an id that ends with "Edit", can be used to edit any part of a submission. Other Invitations can be used, but the property "deletable: true" must be declared in the field to be removed as in the example below.

Editing a submission using an invitation that is not the Meta Invitation:

client.post_note_edit(
    invitation = 'ICML.cc/2023/Conference/-/Edit',
    readers = ['ICML.cc/2023/Conference'],
    writers = ['ICML.cc/2023/Conference'],
    signatures = ['ICML.cc/2023/Conference'],
    note = openreview.api.Note(
        id = note_id,
        content = {
            'supplementary_material': { 'delete': True }
        }
    )
)
"supplementary_material": {
    "value": {
        "param": {
            "type": "file",
            "extensions": [
                "zip",
                "pdf",
                "tgz",
                "gz"
            ],
            "maxSize": 500,
            "optional": True,
            "deletable": True
        }
    },
    "description": "All supplementary material must be self-contained and zipped into a single file. Note that supplementary material will be visible to reviewers and the public throughout and after the review period, and ensure all material is anonymized. The maximum file size is 500MB.",
    "order": 8
}
client.post_note_edit(invitation='ICML.cc/2023/Conference/-/PC_Revision',
    signatures=['ICML.cc/2023/Conference/Program_Chairs'],
    note=openreview.api.Note(
        id = submission.id,
        content = {
            'title': {
                'value': submission.content['title']['value'] + ' Version 2'
            },
            'abstract': submission.content['abstract'],
            'authorids': {
                'value': submission.content['authorids']['value'] + ['[email protected]']
            },
            'authors': {
                'value': submission.content['authors']['value'] + ['Melisa ICML']
            },
            'keywords': submission.content['keywords'],
            'pdf': submission.content['pdf'],
            'supplementary_material': {
                'value': { 'delete': True }
            },
            'financial_aid': {
                'value': submission.content['financial_aid']['value']
            }, 
        }
    ))

How to Export all Submission Attachments

How to extract PDFs and zip files associated with submissions.

  1. If you have not done so, you will need to install and instantiate the openreview-py client.

  2. First, get all of the submissions for your venue. If your venue is double blind, do the following:

notes = client.get_all_notes(invitation = "Your/Venue/ID/-/Blind_Submission", details = 'original')

3. If your venue is single blind, do the following:

notes = client.get_all_notes(invitation = "Your/Venue/ID/-/Submission")

4. Iterate through each submission. For each one, check if it has the attachment you are looking for, and if it does, export it. In this example, we are exporting pdfs and naming them with the format paper#.pdf.

5. If you wanted to extract a field called supplementary_material which authors uploaded as zip files, you could do the following instead:

for note in notes:
    if(note.content.get("pdf")):
        f = client.get_attachment(note.id,'pdf')
        with open(f'paper{note.number}.pdf','wb') as op: 
            op.write(f)
for note in notes:
    if(note.content.get("supplementary_material")):
        f = client.get_attachment(note.id,'supplementary_material')
        with open(f'paper{note.number}_supplementary_material.zip','wb') as op: 
            op.write(f)