How to upload assignments with Python

Overview

Follow these instructions if you are not using OpenReview’s matching system and are computing assignments locally to upload in bulk.

General steps to follow:

1. Upload assignments as Proposed Assignments

2. Use the assignment browser and stats page to spot check assignments

3. Deploy

Prerequisites

1. In your request form, Submission Reviewer Assignment should be set to Automatic.

2. Let the full paper deadline pass

3. Set up the Python client

4. Create the necessary invitations by calling setup_committee_matching() for all roles:

# Use API 1 client
venue = openreview.helpers.get_conference(client_v1, request_form_id)
venue.setup_committee_matching(committee_id=venue.get_reviewers_id())
# For ACs: venue.setup_committee_matching(committee_id=venue.get_area_chairs_id())
# For SACs: venue.setup_committee_matching(committee_id=venue.get_senior_area_chairs_id())

Uploading assignments as Proposed Assignments

Proposed Assignments are a way to view your assignments in OpenReview without making them visible to the committee. This can be helpful if you want to spot check assignments, use the stats page to view the assignment distributions, and manually adjust them in the UI.

In this example, we will upload Reviewer assignments. The exact script depends on how you have your assignment data structured. We will assume you mapped paper IDs to a list of assigned reviewer profile IDs, e.g.:

paper_id_to_assigned_users = {
    'paperid1': ['~profile_id1', '~profile_id2', ...]
}

The user IDs need to be their profile.ids. Any time you are adding users to groups or posting their edges, you use profile IDs. This is how we ensure data is synced between the consoles, assignment browser, and more. If you're unsure, get all profiles with: openreview.tools.get_profiles().

1. Setup

First, go to the Assignments page for your group and create a new Assignment Configuration. Since you are not using our matcher, only the title and Max Papers is important. title will be used to label the edges and Max Papers will be used as the default quota in case any users are added to the group later. You can input any values for the other settings.

Next, change the status of the configuration to Complete. Get the assignment configuration note using the title and post a note edit updating its status:

assignment_config_note = client_v2.get_all_notes(
    invitation='venue_id/role_name/-/Assignment_Configuration', 
    content={'title': 'insert-assignment-config-title'})[0]

client_v2.post_note_edit(
    invitation='venue_id/role_name/-/Assignment_Configuration',
    signatures=[venue_id],
    note = openreview.api.Note(
        id=assignment_config_note.id,
        content = {
            'status': {
                'value': 'Complete'
            }
        }
    )
)

2. Build Proposed Assignment edges

Now we need to build Proposed Assignment edges for each user. All the edges need to be posted with: invitation, head, tail, label, readers, nonreaders, writers, signatures, and weight.

Since we're uploading assignments for reviewers, we will use the following edge configuration:

• invitation: The reviewer proposed assignment invitation ID.

• head: The ID of the paper.

• tail: The reviewer's profile ID.

• label: The assignment configuration title. This is how we differentiate between edges of different matchings.

• readers: List containing: venue ID, assigned SAC ID, assigned AC ID, the assigned reviewer's profile ID.

• nonreaders: List containing: paper authors group ID.

• writers: List containing: venue ID, assigned SAC ID, assigned AC ID.

• signatures: List containing the Program Chair group ID.

• weight: 1

  • Weight typically represents the aggregate score, but when we're manually uploading assignments we can just set this to 1.

To see how Proposed Assignment edges should be configured for each role, you must view the respective Proposed Assignment invitation by going to: https://openreview.net/invitation/edit?id=proposed_assignment_invitation_id

If SACs are assigned to ACs, the head and tail will be profile IDs. Refer to their invitations for more info.

We will create a function that processes the assignments for each paper and builds the edges using the title of the assignment configuration to label the edge:

# Retrieve proposed assignment invitation ID
proposed_assignment_invitation_id = venue.get_assignment_id(
    committee_id = 'venue_id/role_name',
    deployed = False
)
# Result: venue_id/role_name/-/Proposed_Assignment

def process_paper_assignments(paper): 
    # Initialize assignment edge list
    paper_assignment_edges = []
    
    # Get group IDs for each role for this paper
    paper_reviewer_id = venue.get_reviewers_id(number=paper.number) # E.g. "venue_id/Submission5/Reviewers",
    paper_ac_id = venue.get_area_chairs_id(number=paper.number)
    paper_sac_id = venue.get_senior_area_chairs_id(number=paper.number)
    paper_author_id = venue.get_authors_id(number=paper.number)
    
    # Get users assigned to this paper
    assigned_users = paper_id_to_assigned_users[paper.id]
    
    # Build Edges for each reviewer
    for user_id in assigned_users:
        paper_assignment_edges.append(openreview.api.Edge(
            invitation = proposed_assignment_invitation_id,
            head = paper.id,
            tail = user_id, # profile.id
            label = 'insert-assignment-config-title',
            readers = [ venue_id, paper_sac_id, paper_ac_id, user_id ],
            nonreaders = [ paper_author_id ],
            writers = [ venue_id, paper_sac_id, paper_ac_id ],
            signatures = [ venue.get_program_chairs_id() ], # E.g. venue_id/Program_Chairs
            weight = 1
        ))
    return paper_assignment_edges

3. Post Proposed Assignment edges

Gather all the proposed assignment edges. We use concurrent_requests which takes in a function and a list, where each item in the list will get passed to the function. Here, the list should be the list of submission notes.

First, get all active submissions. This means we are only posting assignments to active papers.

Next, call concurrent_requests with the previously created function and use reduce and concat to get a list of edges:

assignment_edges = reduce(concat, openreview.tools.concurrent_requests(process_paper_assignments, submissions))

Finally, check if it is the expected number of assignments and post the edges in bulk:

print('Number of assignment edges', len(assignment_edges))

openreview.tools.post_bulk_edges(client=client_v2, edges=assignment_edges)

4. Check the data

Check the edge count in your browser to verify that the API posted all the edges: http://api2.openreview.net/edges/count?invitation=invitation_id

Use the stats page by clicking View Statistics next to the assignment configuration. From here, you can view distributions such as papers to number of users or users to number of papers.

5. Deploy assignments

When you've finalized assignments, you can deploy them. This will make papers visible in the committee's consoles and give them access. This does not send emails.

Call venue.set_assignments() and pass:

• assignment_title: Title of Assignment Configuration.

• committee_id: ID of the Group getting assigned.

• overwrite: Boolean that determines if the current assignments should be overwritten.

  1. If False, it will only post the new assignments.

  2. If True, it will delete all current assignments and replace them with the new assignments.

Only use overwrite = True if you want to completely redo assignments.

venue.set_assignments(
    assignment_title = 'insert-assignment-config-title',
    committee_id = 'venue_id/role_name',
    overwrite = False
)

How to undo deployed assignments

If you want to undo assignments for any reason, you can call venue.unset_assignments(). This will undeploy assignments based on the Proposed Assignment edges in the specified Assignment Configuration. So if you made changes to the assignments after deployment, those reassignments won't be removed. You will need to handle them manually either through the UI or by deleting their edges and removing the users from the paper groups.

Use this if you deployed assignments, made no modifications, then want to undeploy:

venue.unset_assignments(
    assignment_title = 'new_assignment_configuration_title',
    committee_id = f'{venue_id}/role_name'
)