Why Write a Rubric? Focused Feedback, with Consensus Across Raters that’s why!

Focused Feedback

After many discussions with other therapists wanting to learn how to write a rubric. I decided that a blog post was needed to clarify why a rubric is important, not only to ourselves but most importantly, to our clients, other professionals and parents/caregivers. As I began my research for this blog, I came across the term “focused feedback.” I think that it is a great way to express the intent of a rubric. I think, though, that a rubric is more than just feedback to the client or ourselves. I am thinking in lateral terms, from therapist to therapist or teacher for example.  A rubric provides clearly understandable expectations for goal progress, from professional to professional, professional to parent, professional to student or client. A well written rubric is easy to understand by all parties, including the client and caregivers, with clearly identified expectations for progress without having to search for more information. My book, Using Rubrics to Monitor Outcomes in Occupational Therapy, discusses, in more detail, the importance of writing rubrics clearly.

Across Raters

When this term is used, mainly in research, it generally means inter-rater reliability or the degree of agreement among raters. A rater is someone that is using your rubric to assess the client’s performance. Think about times when you are not available, someone is covers for you. It may also be that a classroom aide is collecting data using your rubric. Data collection on all goals occurs throughout the day and is the responsibility of everyone working with the student. You want all the raters to be consistent, considered reliable and see the same progress that you do regardless of their training. Raters can vary in judgment without clear and consistent guidelines [definitions]. According to Hung and Wang (2012), individuals judged by the most severe raters are at a disadvantage, because their ratings are always lower. Conversely one might assume the same is true of a lenient rater in that the same disadvantage may occur because their ratings were always higher. Neither the severe or the lenient rater actually provide usable data. The only way to accomplish this is to have clear and concise definitions for scoring. It is difficult to gain a consensus between raters when the definitions for scoring are vague. Typically, one word definitions [such as fair, poor, and good, maximum, moderate and minimal] are very subjective, leaving the rater to guess at what that means. It the heat of a classroom or therapy room, it is difficult to flip back and forth between explanations and rubrics, possibly leading to non-compliance of a rater or that dreaded subjective assessment. This really drives home the point of doing a thorough task analysis, keeping the rubric clear and concise with definitions for scoring defined within the rubric itself.

I recently assess a student for assistive technology. One of the staff concerns was the student’s lack of compliance with the current device.  Upon interview, the student revealed sensory issues [sound and proprioceptive feedback] with the use of a standard QWERTY keyboard. The student also revealed a preference for a virtual keyboard, such as on a tablet. The student typed equally as well on a standard keyboard and a virtual keyboard so that switching devices would most likely not cause a problem.  So a ‘Compliance Rubric’ was developed to determine if the student would use the device in class. I would like you to look at two rubrics exploring her compliance. Check out the definitions for scoring. Which rubric do you think will reveal more accurate data on compliance?

Rubric 1

Criteria	Score 0	Score 1	Score 2	Score 3
Student will bring  tablet to class	Refused or maximum assistance	Moderate assistance	Minimal assistance	Independent

Rubric 2

Criteria	Score 0	Score 1	Score 2	Score 3
Student will document that tablet is requested	Student refused to document in agenda, staff will document	Student documents the request for tablet with 2 or 3 prompts	Student documents the request for tablet with one prompt	Student documents the request for tablet without prompts
Student will bring tablet to class	Student does not have the tablet with and refuses to retrieve it when prompted	Student does not have tablet but will retrieve it with prompts	Student does not have tablet but will ask to retrieve  tablet without prompts	Student brings tablet to class without prompting
Student will use  tablet to work on assignments in class	Student refuses to use the tablet to complete assignment	Student will open the tablet and activate the app with prompts	Student will open the tablet and activate the app without prompts	Student will open tablet, activate the app and begin to work on assignment without prompts

It is very difficult to assess compliance without clear and concise expectations. One of the criterion that should be considered is, does the student know that the tablet is expected in class during a particular period on a particular day? We need to clearly indicate that this student was notified of this expectation. Then and only then can we have the expectation that the device will be there and make judgments on her ability to comply with the request.

Data Collection

Data collection should also be on the same page with a location to provide notes provided. This can be on the back of the page or in a separate notebook. When the data collection table is not on the same page, we run the risk of either delayed entry or no data collected at all. The data collection table needs to be as clear and concise at the rubric itself. It should include the date, time or period, the criteria being assessed and a place to insert the score.

http://rubistar.4teachers.org/index.php?screen=WhatIs

http://www.rasch.org/rmt/rmt261e.htm

Compliance Rubric