TABLE OF CONTENTS

• Introduction
• Prerequisites
  • API Key: Allow Include Parameters
  • Survey: Grade Passback Settings
• Configuration
• Calculation Breakdown
  • Glossary
    • Submission record
    • Submission
    • Distribution
  • Steps

Introduction

Due to the nature of distributing surveys through Qualtrics LTI, Qualtrics itself does not know how many distributions there are for a given survey. On top of that, responses to the survey may not always contain user data that would allow, for example, duplicate responses to be filtered out during data analysis on the Qualtrics side. These facts can make it tricky to calculate the response rate of a survey.

The Response Rate feature, if enabled, adds a Response rate page to the Institute level of Qualtrics LTI. This page allows you to select a Survey and 1 or more Courses in order to calculate how many student Submissions and Distributions there are for a Qualtrics survey. More on the precise terminology below.

✏️ Note

If you want sub-account admins to also be able to use this feature, it is also possible to enable this page on Department levels by navigating to <Institute level> → Admin Settings → Enable the use of the 'Response rate' page on sub-accounts.

Even though the Response rate page is available in Qualtrics LTI in many (sub-)accounts, we request all course and submission data through Canvas using Canvas' permissions and authorisation system. This means that you will always see a list of courses that your user account has permissions to see, regardless of whether you are on the Institute level or Department level, and Canvas will also check your permissions for any assignments and courses that you request submission data from. To that end, you can consider the Response rate pages to exist separately to the account that you're viewing them on. They are not scoped to that particular account in any way.

Prerequisites

✏️ Note: The Response Rate feature is in Beta and is not available to all customers at this time. If you are interested in beta testing the feature, feel free to get in touch!

In order for the Response Rate feature to work, the following settings are required.

API Key: Allow Include Parameters

If your API Key for Qualtrics LTI has Enforce Scopes enabled, then it must also have Allow Include Parameters enabled.

Fig 1. A screenshot of Canvas → Admin → <Root account> → Developer Keys → <API Key> → Enforce Scopes, with checkbox Allow Include Parameters checked.

This allows us to fetch the assignment and user data associated with each submission record. Without this, we cannot calculate the response rate.

✏️ Note: If you use our Inherited Key ("Qualtrics by Drieam"), then you do not need to do anything.

Survey: Grade Passback Settings

All data that we use to calculate the response rate comes from Canvas.

The assignments that you want to gather the response rates for must be attached to a survey instance that has one of the With Canvas submission grade passback settings.

If a survey instance does not have this setting, then when a student completes the survey in an assignment that that survey instance is attached to, Qualtrics LTI will not update the corresponding submission record in Canvas to be 'submitted'. Therefore, we cannot determine whether responses to that survey instance are submitted or not for the purposes of the response rate calculation.

Fig 2. A screenshot of the Qualtrics LTI → <Survey> → Settings → Survey grading section, with With Canvas submission highlighted in a dropdown.

For example: If you have added the same Qualtrics survey on the Institute level and a Department level, but the survey instance on the Department level does not have the With Canvas submission grade passback setting, then any submissions by students for assignments attached to the Department level's survey instance would not be included in the response rate that we calculate — but assignments attached to the survey instance of the Institute level would be.

Configuration

The Configuration section of the Response rate page allows you to select a Survey and 1 or more Courses. It also allows you to filter out certain submissions with Submitted from/until dates.

The Survey list shows the names of all Qualtrics surveys across the Institute level and all Department levels that you use. It will not show any Qualtrics surveys that have been added to users' Personal Surveys areas.     

The Courses tree shows the names and course codes of all Canvas courses that Qualtrics LTI is aware of for your institution — in other words, all courses that have at least 1 assignment with a survey attached via Qualtrics LTI. 

The Submitted from/until date fields allow you to exclude submissions that are not made between your chosen dates. If you leave Submitted from blank, then we will include all submissions up to and including the Submitted until date. Likewise, if you leave Submitted until blank, then we will include all submissions made after your chosen Submitted from date. If you leave both blank, then we will simply include all submissions ever made.

Calculation Breakdown

After clicking the Calculate button, a precise sequence of steps occurs in order to gather and process the data required to calculate the response rate of your chosen Qualtrics survey.

We want you to know exactly how this data is processed so that you can make informed decisions with the information we provide. Therefore, we show you a breakdown of how the response rate was calculated, which will be specific to your configuration settings for the most recent calculation.

The following sections present a more fleshed-out and technical version of the calculation breakdown which is seen on the Response rate page.

Glossary

The terminology that we use throughout the response rate process is very specific and is hereby described so as to avoid confusion with terminology used by Canvas or Qualtrics.

Submission record

You can think of this as a combination of 'student + assignment'. Whenever an assignment is created in Canvas and assigned to students, Canvas creates a submission record for each of these students. By default, the submitted_at property of these records is null and the workflow_state is "unsubmitted" — in other words, it has a "default status of 'incomplete'". See Canvas' documentation for the Submissions API for more information.

If you create 2 assignments in a course with 3 students, and 1 assignment in a course with 5 students, then there are (2*3) + (1*5) = 11 submission records, ergo 11 distributions.

Submission

Confusingly, Canvas refers to submission records as "Submissions", even if they haven't actually been submitted. When we refer to submissions in Qualtrics LTI, we are referring to submission records that have the workflow_state of either: "submitted", "graded", or "pending_review".

As mentioned before, we do this assuming that the instances of your chosen Qualtrics survey (in the Institute level and all Department levels) have their grade passback settings set to With Canvas submission. With this setting, whenever a student completes a survey within an assignment, Qualtrics LTI tells Canvas to update the submission record for that student + assignment. If any assignments were created with an instance of your Qualtrics survey with No Canvas submission set as the grade passback setting, then Canvas will not be informed when a student completes a submission, and therefore this means that we cannot determine whether the student has responded via that assignment.

Distribution

For regular Qualtrics surveys, Qualtrics refers to the number of contacts that the survey has been distributed to as the number of 'Distributions'. For many types of surveys, including those distributed via Qualtrics LTI, Qualtrics cannot know this number as it does not have access to that population data, e.g. the number of students + assignments in Canvas.

To that end, the Distribution count is equal to the number of all submission records from Canvas for your chosen survey's assignments.

Steps

These are the steps taken by Qualtrics LTI to retrieve and process the data from Canvas:

1. We find course records in our database that match your selected courses.
   • If a course has never had an assignment with a survey attached to it created within it, then we will not know about the course.
2. We fetch all submission records from Canvas for those courses, including associated assignment and user data.
   • We do this using the List submissions for multiple assignments endpoint.
   • The amount of time that the response rate calculation takes is thus directly proportionate to the number of submission records that exist in your selected courses.
3. We filter out the submission records that:
   • have an associated User object with a name resembling "Test Student";
     • This is the student that you impersonate when you use the View as student feature in Canvas.
   • have a submitted_at date that does not fall between your chosen Submitted from/until dates;
   • do not have an associated Assignment object with an external_tool_tag_attributes['url'] value that is associated with the survey you selected in your configuration.
     • E.g. if a submission record has an associated Assignment object with external_tool_tag_attributes['url']: "https://qualtrics.drieam.com/lti/launches?assignment_id=bfadc59a-0aac-449d-ad89-47b37ccfaabc", we check if the assignment record in our database associated with that ID was created for an instance of the survey that you selected.
4. For each remaining submission record, we determine whether or not the user has responded:
   1. If the submission record has an external_tool_url of Qualtrics LTI, we:
      1. check if the workflow_state of the submission record is "submitted", "graded", or "pending_review";
      2. internally mark that submission record as being 'submitted'.
   2. If the submission record does not have an external_tool_url, or does but it is not of Qualtrics LTI, we internally mark that submission record as being 'unsubmitted' (as we cannot guarantee whether or not they responded).
      • In principle, this should never occur. It is a defensive measure.
        
5. We group each submission record by assignment and establish the response rate for that assignment:
   1. We add up all submission records that we marked as 'submitted' to form the Submissions number of that assignment.
   2. We count the number of total submission records for that assignment to form the Distributions number.
   3. We divide Submissions over Distributions to form the percentage for that assignment.
6. We further group each assignment's set of submission records by the course that the assignment(s) are under:
   1. We add up the Submissions number of each assignment to form the Submissions number of the course.
   2. We add up the Distributions number of each assignment to form the Distributions number of the course.
   3. We divide Submissions over Distributions to form the percentage for that course.
7. We establish the response rate for the survey as a whole:
   1. We add up the Submissions number of each course to form the Submissions number for the survey.
   2. We add up the Distributions number of each course to form the Distributions number of the survey.
   3. We divide Submissions over Distributions to form the percentage for the survey.
8. We render a breakdown of the response rate by survey, course, and assignments on the Response rate page.