Our take on the mastery workflow

After some great conversations at #TMC15 about formative assessment, mindset, and flipped instruction, I wanted to put to words what we're currently doing and why.

A school year ago, all of our classes assessed with a one-try final, one-try unit tests, and one-try quizzes.  We did our best to build formative feedback into our daily lessons and homework, but avoided retake assessments in fear of students entering a death-spiral of retakes and falling behind.  Our classes were all fully-flipped, and I started moving away from guided notes into only making example videos for the homework packet.

This past year, with our resources allowing us to make classes self-paced, we made the leap into self-paced quizzing (only when students were ready to try).  Each quiz was hyper-focused on a narrow concept and thus grading was very binary -- either you essentially "got it" or you didn't.  We also offered a B and C quiz to retry mastery of a concept if they did not pass the first time.  To qualify for a requiz, students needed to show us evidence of new learning, ranging from a set of problems to a productive conversation.

Since the assessments were short and the grading was simple, we started grading the quizzes immediately upon completion.  Students loved the instant feedback.  I loved being able to immediately jump into a feedback conversation while the material was fresh in the students' minds.  Near the end of class, it was impossible to keep up and still circulate / help with questions, so those quizzes were returned the next day.

We found with our grading scale that many students were passing all of their quizzes, but still doing poorly on tests.  My colleague Rob realized that the pass/not-yet scoring was sending the message that "mostly right" on a quiz would lead to a successful test.  To correct for this, he implemented a 0-4 rubric score (0=didn't try, 1=getting started, 2=conceptual errors, 3=minor errors, 4=perfect or very close).  Students didn't like it at first, but then their test scores started going up, and thus their course grades, so they came around.

Efficiency was a growing challenge in class.  Even when dedicating 80% of class time to self-paced practice (in groups) and quizzing, it was hard to offer mini-lectures to groups, answer questions, check-in with everyone, grade quizzes, and go over quiz errors all during class.  In some classes, we were able to get a student assistant to help with real-time grading, but it wasn't always as helpful as needed.  Rob made another leap by setting his keys out for students to self-correct.  They did it in another color with pencils away so they would not change answers.  Once self-checked, they brought him the quizzes to him, discussed the errors and reasons behind them, and agreed on a 0-4 score for the quiz.  After holding out for a couple weeks, I tried it too and immediately loved the extra time and student empowerment that quickly followed.  The follow-up conversation after quizzes saw an improvement as well as students often figured out their errors and the root causes on their own.  They did most of the talking and I just nodded and handed out high fives.

Within a unit (typically 8-10 block periods long), there were 3-6 sections with their own A-B-C quizzes.  We expected all students to get a 4 on all sections.  Some students were able to reach this in the first half of the unit without taking work home, leaving them a lot of extra time.  We added the revised expectation that students with 4's do the retake quizzes again to confirm that they still have a mastery of the material before the test.  There was no risk of lowering the grade, but most students bought into the idea of a double-check before the test.

Despite our flexible approach to quizzing, we kept the unit test dates fixed.  Since the quarter had fixed dates, we needed to keep this structured so students would finish the course.  However, we offered a single retake for each unit test during the week of the quarter final, a perfect time to go back and study old content that was not mastered the first time.  This kept everyone on pace during the year but offered hard-working students one more chance to relearn the material and make major grade improvements before the end of the quarter.

The classroom layout played a major role in facilitating the mastery workflow.  My room, due to being a computer lab, did not work particularly well.  Rob cobbled together tables from across the building to replace most of his desks, creating collaborative work areas.  He kept 8 desks against the window for students to take quizzes individually.  The teacher desk is used for self-grading quizzes with the keys.  I think this is the ideal layout (see below):

The course grades for our Geometry and Algebra 1 courses (the ones using mastery) were composed fully of assessments: 30% mastery quizzes, 50% unit tests, 20% final test.  I thought this was a fair breakdown given that it all reflected what students could do, 30% could be infinitely repeated (before the unit test), and 80% could be repeated at least once if needed.

WINS: Looking back on the year, there were a lot of wins:

• Student learning (as measured by unit test proficiencies) were the highest they had ever been in nearly all sections.
• Student mindsets appeared to become more open to revision and relearning with effort.
• Student engagement during work-time was up (likely due to nearly-daily quizzing and instant, actionable feedback on the quizzes)
• Conversation quality between teacher and student went through the roof -- it was so focused on learning!  Complaints about points nearly disappeared, feedback was mostly student-generated with only teacher refinement, and the ability to measure real progress in learning eliminated the mystery in the process.
• Anxiety was way down amongst students on quizzes and tests.  They loved being able to take a quiz when they were ready.  However, all of the practice with the quizzes and re-quizzes also made them more comfortable to take the scheduled unit tests.
• I could predict test scores pretty well from the sum of the max quiz scores for each section, meaning that they were actually aligned and that relearning was as effective as someone who mastered it on the first try

UNCERTAINTY: I'm not sure how to think of the assessments:

• Dylan Wiliam and other experts in the formative assessment are heavily opposed to attaching grades to formative assessments.  We grade ours 0-4, and even though a 3 is "proficient", it is a 75% (C) in the gradebook.  The up-side to this is that it is very motivating to work for a 4/4 through retakes (which most students choose to do).  The down-side is that we are calling a "C" proficient.
• Quizzes are only 30% of the grade, but that can still have an impact.  The ideal case is to make it 0% of the grade and make it purely formative assessment.  However, that makes the grade 100% determined by 6 big tests.  That adds a lot more anxiety and reduces grade control for students without really shifting the grade (remember that quiz and test grades are highly correlated).  In addition, from talking to students about it, there is likely less motivation to shoot for 4's on each section, unraveling some of the wins we made this year through mastery.
• Regardless of the grading situation, I think the quizzes are true formative assessments.  They inform students with exactly how they are doing in a test-like scenario, with test-level questions, and can be taken whenever they are ready.  If they self-correct and find they did many problems wrong, they have a conversation with a teacher.  They zero in on what went wrong and what they need to do to learn.  The grade is only applied at the end of that and students usually know what they are getting based on the conversation, so the teacher is just part of grading for calibration and integrity of the system.

INTENSE FEAR: Are we trapped in a local maximum?

• With all of the great process work happening, I worry that our curriculum is simply bad math.  We have nearly no word problems and no contextualized situations -- just naked numbers, expressions, and equations.  The example videos have concept instruction embedded, but due to the way some units are divided up, it can be learned as procedures, not mathematics.
• One fear is that embedding multiple-entry-point, contextualized discussion problems that groups can work on will not lend themselves to this assessment structure.  A greater use of whole-group time would reduce time for individual practice and quizzes.  It would also be harder for the class to be self-paced with more whole-group activities.  This fear may be unfounded.
• A larger fear is the utility of the flipped videos -- if I stop telling them how to directly do everything to move to inquiry and discussion in more interesting problems, the videos are not only unhelpful but actually destructive to learning.  The practice problems would need to be fully re-written and the videos would need to be isolated to vocabulary or the rare procedure that just needs to be memorized.  Part of this fear may be unfounded as there are a decent number of "just memorize it" terms and procedures, but I think a long road is ahead of me here.
• A huge fear is getting trapped by student success.  If we start teaching real mathematical thinking everywhere, grades will drop.  We got really good at teaching procedures efficiently and effectively in a way that the vast majority of students could learn and repeat.  Students (claim to) hate word problems, contextualized situations, and ambiguity because they are difficult and unfamiliar in their math courses.  If we upped the rigor and taught through discussion, analysis, and inquiry, students (and parents) would get upset, preferring to go back to "what worked".  I'm still committed to moving in this direction, but I'm terrified.

If you made it all the way to the end, you rock.  I would love to hear your thoughts on any or all of this -- there is a lot here.  Special thanks to Lisa @lmhenry9 for pushing me to get this all down in writing and to Princess @MathPrincessC for talking through parts of this with me tonight after her TMC session.  Huge thanks to Rob W. for trying out and then teaching me most of the ideas I'm talking about in our mastery workflow, to our student teacher Rob N for serving as the core of my reflecting all semester and cross pollinating ideas between Rob W and I, and to our entire PLC for reflecting with us and pushing us forward.

Using the mastery quiz work-flow to eliminate project grading

**WARNING: UNTESTED IDEA**

When my colleague Rob and I discussed Stats projects and reflections, we were reviewing the value of each and time we need to invest in them.  On a parallel track, we have both been using mastery quizzes in our other classes and are moving them now into Stats.

The projects in Stats class (Infographic, Prove-it video, Minute to Win It paper, Ultimate Frisbee spreadsheet) are fun to do, and students seem to learn a lot, but the grading process is tedious and doesn't seem to correlate with individual learning (reflecting only the effort of the hardest worker on the team).  The reflections, which are individual, are a great opportunity to see what each person really learned, but students rarely take them seriously.  Since they always come at the end after I went through multiple revision cycles of the project with students, they either get no direct response from me or only something small.

My thought was to stop grading projects altogether.  I would keep the rubrics up as the ideal to work towards, but I would remove the points.  Instead, I would create mastery "quizzes", let's call them "check-ins", that ask students to explain (in writing) specifically how the project taught them the skill in question.  When finished writing, they would bring that check-in sheet and their project (on computer/iPad) and show it to me.  If I think they understand it AND their project is an artifact of that understanding, that individual is checked off with that skill.  Each person in the group needs to do this.  They can all share the project itself, even if they divided and conquered some of the work, but everyone needs to defend their understanding of the core components individually.  That is part I would track.

Since this check requires that the work gets completed (except any busywork that didn't lead to any objectives of mine), I don't have to directly grade the project.  Since the check-in involves self-reflection on the desired outcome, I can also skip a separate reflection assignment.  Most importantly, the time I spend assessing would be mostly in direct dialogue with students.  This is the change we saw in our department when we transitioned into mastery quizzes elsewhere, and class becomes very productive.

I think the hardest part will be properly sizing the assessments so there are not too many to handle each class, but being comprehensive enough to ensure that the key learning is taking place.

If you think this makes sense, please let me know.  Also please tell me if this is a terrible idea so I don't invest too many hours in it, or at least tweak it as soon as possible.  Thanks!

A more interesting way to learn experimental design

My colleague Rob and I were talking last week about how he used to introduce the big ideas of stats.  He told the story about the farmer (whom I suppose has a name, this is a true story) a few centuries back who wanted to understand cause and effect with his crops.  Since the problem was mixing together all the input variables, he broke his land into plots and tested each variable separately.  This was (as far as I know) the start of modern experimental design.  I loved the story but didn't think most students would buy-in.  They had to feel the experience of the farmer somehow.

Since I love simulations for everything, especially in stats, I thought a simple farm simulation with a few plots and different parameters you could set would be a good starting point.  The idea was to build a simulator with too many variables (inputs) and a poor understanding of detailed cause and effect.  Students would work in groups of 2 or 3 and have 10 "growing seasons" to learn all they could about how the simulation worked.  Then each group would have one shot to try their inputs on my screen and advance in a tournament bracket (everyone loves a good bracket) to the final farming round.  The most profitable team wins.

With only 10 rounds to learn from, only teams with a disciplined approach would be able to learn enough to give them a strong shot at winning.  Haphazard clicking would leave a lot up to luck.

The first time through, I would introduce the game before I taught them anything, using the simulation as an intro event.  After playing and finding the winners, we would talk through their approaches during the 10 practice rounds and what they were able to learn.  We would discuss the challenges they found in trying to learn things.  THEN, I would introduce some of the core concepts of experimental design.  After some teaching, I would throw them back into the simulator (with a different crop / parameters) to see if they are able to better learn the ideal conditions and everyone can be more profitable.  I would assess them on how they approached the second game (they could write a summary of what they did, how it compared to the first time through, and why they made changes).

Here is an ugly but functional prototype to get the idea across.  I would love it if you can play around with the farm simulator and my description above on the lesson intent and offer feedback / ideas / links / anything to help me make this better.  Both the lesson and the simulator are very rough at the moment, but I think they could be awesome if it became a community-designed lesson.  Of course all of it will be shared freely if you want to use it.  Thanks in advance!