Monday, January 19, 2015

EQAO Reflection

Our grade nine students wrote their Grade 9 Assessment of Mathematics (EQAO) last week. Often during this time I reflect on the process, because really what else are you going to do for two hours while supervising. This year my thinking wasn't about the pros or cons about the test but rather the way we evaluate it. The test is sent off to be marked provincially but before that happens schools have the option to evaluate it in order to include some or all of the mark as part of the student's final grade. The thinking here is that if it counts for something then perhaps students will take it seriously. At my school we count the test for 10% of a student's final grade. Then about a week later they will write the final exam that counts for 20% of their grade.

The test consists of two booklets that each must be completed in an hour. Each booklet is made up of 7 multiple choice questions, followed by four longer 'open response' questions then finishes with 7 more multiple choice questions. Once the second booklet is completed students are asked to complete a questionnaire.

My observation has been that more often than not students come into the test under prepared and it serves as a bit of a wake up call to them. They then (hopefully) use the remaining classes to prepare for the final exam.

This year I have decided that I am not happy with counting the test for any portion of the students' final marks. In fact, my students did so poorly that after the fact I told them that I was not going to count it at all towards their mark and here's why:


1. Time

Many students did not have time to complete the test. They had an hour to complete each of the two booklets. For the second year in a row my strongest students did not complete the booklets on the first day. These students were very concerned about the impact it was going to have on their overall grade. Rather than providing incentive to do well it caused a great deal of anxiety. As a math teacher my goal is to help students reduce their anxiety towards math not contribute to it. I also try to evaluate what a student knows and does not know. If a question is left blank I have no idea if it was because the student ran out of time or because they did not know how to do it. By removing time from the equation I can make a better judgement of what the student know.


2. Multiple Choice

I have decided that I disagree with the multiple choice questions. They obstruct my view of what the student does or does not know. Some students will get the correct answer by guessing. Others will get the incorrect answer by guessing. In either case, I am unable to see the process that allowed them to arrive at their answer and as a result I am unable make a true judgement of their understanding of the material.


3. Feedback

I don't know much about the official feedback students get so if I'm wrong here let me know. I believe that tests get marked in the summer (the rest of the cohort will write in June) and a mark is returned to the students in the fall. This is far from immediate feedback and is anything but descriptive. Not very useful in my mind. As a teacher I can mark the work, but I'm not allowed to copy anything. This means that I can't show students where they went wrong. I can tell them that they messed up on the bicycle question but unless they can see where, I'm not sure that's useful.


4. Justification

I'd be hard pressed to justify any mark to a student or a parent given that the tests get sent off, never to be seen again. Students should be able to look at their marked work and question my judgement, which is sometimes right and sometimes wrong. In fact, I enjoy when students start questioning my evaluation as it often brings out what they truly meant to write or allows me to better understand their misconceptions.


5. Rationale

When students ask why the test has to count for a portion of their grade I struggle to give a valid reason. I typically say something along the lines of "If you're going to spend two days writing it, we may as well give you some credit for it". It's not an answer I'm comfortable with but it's all I have. One of the reasons I'm not comfortable with it is that the vast majority of my students perform much worse on the test than they do on the final exam. We could probably discuss what that says about my teaching, but let's save that for another post. The real reason that we count the test as a portion of a student's grade is that we believe that this will make them take it more seriously, which means they will perform better, which will make the school look better. Given that twelve out of eighteen students in my colleague's class said on the survey at the end  that they didn't know if the test was going to count (and yes he did let them know on numerous occasions), I'm not sure that counting it is a good motivator. Besides, is this in the best interest of the student or the school?

I'm curious to know whether counting the test as a portion of a student's grade makes them perform better. What does the data say? Do schools that count the test outperform those that don't? Is this information publicly available? Are there any schools that don't count the test?  Or does everyone count the test so that they don't look bad? Is this in the best interest of the students?

What does your school do about EQAO testing in grade nine math?

10 comments:

  1. Great post, Dave, and very timely for me as we start our EQAO test tomorrow. I can't speak for all the teachers in my school, but I count the multiple choice sections of the test for 5% of the students' year. I believe it does "motivate" them somewhat. Today, as I was reminding students of a couple key formulae and to TRY EVERYTHING (as in, every jot something down for every single question), one student said "But the long answer questions aren't being counted!" I think mine might be more likely to leave those questions blank just because I am not counting them.

    I agree with a lot of what you say, especially when it comes to time. We did a board-mandated practice test last week (I wrote in part about that here: http://byodasap.blogspot.ca/2014/11/the-false-pressure-of-standardized-tests.html), which was 14 M/C and 4 Open Response, and many of my 1D's ran out of time. In our class, we've been focussing a lot on perseverance and problem solving (ie. making good use of resources to figure things out), but it feels all for naught since we take pretty much every resource away from the students AND slap a time limit on them for this test.

    I don't know what the students receive re: their level on the final test, but I have heard of a new EQAO portal where you can go in and view a students' results (on this and every EQAO test they've ever taken), as well as their questionnaire results, if I remember correctly. You may find that of interest? Good luck as you decide how to proceed. What will you replace the test with for your summative mark?

    ReplyDelete
    Replies
    1. Thanks for the reply Heather. I'm curious to know if counting questions really changes students behaviour.

      Thank for the information about the student portal. I may have to explore further. I don't think I would replace the test with anything. Currently, the test counts for 10% and the exam counts for 20%. I would just make the exam count for 30%.

      Delete
  2. While using "it will count as part of your final mark" as motivation for students is part of the picture, the other part is over-evaluation. Our students must all write a final exam and do some kind of summative. We count EQAO as the summative for MPM1D. If we did not count it, students would have to do a summative in addition to EQAO and the exam. That would be crazy, in my opinion.

    ReplyDelete
    Replies
    1. I can see why you would want to count EQAO rather than introducing another summative assignment. We don't have any requirements for summatives, so we could just use the exam to make up the 30%. It's great hearing about all the different ways that schools or boards approach not just EQAO but semester end in general. Thanks for sharing.

      Delete
    2. The summative I have my students do is more a self-reflection of the year than anything else, going back over learning goals, reflecting on which learning resources worked best for them throughout the semester. We're doing EQAO Tuesday and Thursday this week - Monday, Wednesday and Friday is the self-reflection assignment to finish off the semester.

      Delete
    3. I like the idea of a self-reflection. Good luck this week.

      Delete
  3. Hi Dave – My name is Debra Rantz and I’m currently the Chief Assessment Officer for EQAO (I’m on loan from my position as superintendent for the Renfrew County DSB). I read your blog post with interest and I’d like to offer some information in response to the questions you raised.

    EQAO, too, wondered whether counting the Grade 9 math assessment as a portion of students’ grade made them perform better. So the agency did some research in 2011 and found that counting the assessment did indeed motivate students to take the assessment more seriously and that it was related to higher achievement. You can check out the bulletin about this research at www.eqao.com/Research/pdf/E/ResearchBulletin9_en.pdf.

    Our annual teacher questionnaire is one public source of evidence about how widespread the practice of counting the Grade 9 assessment as part of students’ class marks is. Last year, 98% of the Academic math teachers and 96% of the Applied math teachers surveyed said they count the assessment. There is a breakdown of how they weighted it in the questionnaires as well. You can access them at http://www.eqao.com/results/results.aspx?grade=9&year=2013&Lang=E&submit=View+Results.

    Your post raised some questions about the value of multiple-choice questions. We believe that multiple-choice questions are a valid and important way of assessing student learning—when used in conjunction with open-response questions. Generally speaking, a great deal of thought goes into responding to multiple-choice questions. And educators can actually learn a lot about student thinking from the incorrect answers that are selected.

    This brings me to the final point I’d like to offer. Your post questioned how much feedback about student learning can be obtained from the provincial tests. As Heather mentioned in her comment, the agency offers an online application, called “EQAO Reporting,” that enables schools to do detailed analyses of their students’ results, including achievement results for every single question, as well as contextual and behavioural information from the student questionnaires. All school principals in Ontario have access to this application and can grant teacher access as well. I would encourage you to speak to your principal to learn more. EQAO regularly offers training sessions on the use of the application, as well.

    I appreciate your thoughtful and reflective post. We are deeply committed to making sure the provincial assessments can support improvement planning in addition to their legislated goal of providing public accountability. I hope the information I have provided is helpful.

    ReplyDelete
    Replies
    1. Hi Debra. Thank you for your comment and thank you for the link to the study about student motivation. It's not surprising that it's a motivator, but worth confirming.

      I agree with your statement about thought going into student responses to multiple choice questions, but this isn't always true. Some students just guess either because they have no idea or because they are running out of time. Educators can only learn from a student's response (either right or wrong) if they know that the answer was deliberately chosen. I like that the wrong answers highlight common mistakes but if a student has guessed (and there's no way to determine this) then the wrong answer tells us nothing.

      I am aware of the application "EQAO Reporting" and have used it a little. There is a ton of information there. Some day I'd like to find some time to explore it in greater detail. My concern about feedback dealt more with direct feedback to the student. When students submit work to me I hand back their work and show them exactly where they went wrong. The hope is that they can see their error, understand where they went wrong and make note of it in order to learn. Without being able to copy their work to show them where they went wrong, students aren't able to learn from their mistakes. This is where my concern lies.

      Delete
    2. Hi again, Dave – When it comes to direct, detailed feedback at the individual student level, it’s important to remember that large-scale and classroom assessment have very different goals. If you’d like, I’d be happy to arrange a tutorial on “EQAO Reporting” for you some time with a member of our Outreach team.

      Delete
    3. Thanks Debra. I might have to take you up on that tutorial some time.

      Delete