All 6 entries tagged Maea

e–assessment – interesting video

Writing about web page http://www.teachersmedia.co.uk/videos/e-assessment-where-next

This is interesting. Good for the e-assessment question. 6 years old now - we should be so much further on.

http://www.teachersmedia.co.uk/videos/e-assessment-where-next

Juliet Nickels : 11 Apr 2012 17:13 |  Tags: Assessment E-Assessment Ma Maea Masters |  Comments (4) |  Close comments |  Report a problem

Foundation research methods and mini–projects

Trying to do a research project on the basis of a 5 day summer school is something like trying to captain an oil tanker on the basis of a weekend in a sailing dinghy! The task vastly outweighs the input.

Juliet Nickels : 05 Aug 2011 11:57 |  Tags: Assessment Education Educational Ma Maea Masters Module Research School Summer University Warwick |  Comments (8) |  Close comments |  Report a problem

ARG

Writing about web page http://www.assessment-reform-group.org/publications.html

Some of the publications

J

Juliet Nickels : 12 Feb 2011 13:48 |  Tags: Assessment Education Educational Formative Ma Maea Masters University Warwick |  Comments (1) |  Close comments |  Report a problem

APP staff meeting 14/10/10

Notes on APP staff meeting 13/10/10

Background

APP was introduced to our school last year primarily in a 'top down' manner (it came from senior management without preamble or discussion among staff) with some dissemination by subject leaders who had had brief training in order to pass on the procedural information to the rest of the staff.

In this session we were told how we were to continue with using APP, adding reading and writing to the maths we were doing last year. Teachers are to focus on 3 children per subject for their class and acquire a body of evidence for levelling those children. During the meeting we were provided with some written work from a y2 child and instructed to use the APP guidlines to derive a level for that child.

Issues arising in the meeting

During the meeting, several issues arose as bulletted below:

• We are using the system for mainly summative purposes – that is we are to derive levels which are to be used for all our data, including 'high stakes' purposes such as judgements of teachers, cohorts and the school

• Some mention is made of using APP assessments to help plan next steps (although little mention of feed back to pupils themselves) suggesting a confusion between formative and summative purposes

• Levels must be generated each term

• All children are to be given a 'teacher assessed' level but evidence need only be collected for 3 in each subject

• Individual teachers are to make judgements about their pupils with some advice that we should work with year group partners

• Interpretation of particular statements varied widely, e.g. 'with support' (support to keep going, support because it is work done during lesson, support because the work is done shortly after lesson)

• In English writing, it is a requirement that 50% of the writing evidence must be in books other than Literacy books

• Overall judgements about level are dictated by the number of assessment focuses (Afs) which have been ticked with a stated number of the required Afs in the guidance

• There was no agreement about how much of the box in an AF needed to be ticked to constitute the level or the achievement of the AF

• Teachers working in small groups or pairs came to significantly different decisions about the level of the work they were looking at (from high 2 to nearly a low 4)

If the system is for mainly summative purposes then a serious question arises about the use of this data to judge teachers' performance since it is very difficult to achieve reliability and comparability by assessing in this way. Individual teachers are required to gather evidence and to perform the assessment themselves. There is no opportunity for teachers to discuss at length the children and the evidence of their levels. Furthermore, the summative assessments for the rest of the cohort will not be backed up by any evidence but it will be taken as read that if the judgement about 3 children is 'accurate' then the rest can be assumed. It is difficult to see how this is a convincing argument, especially as there is no reliability in a system which requires each teacher to make a summative judgement of work based on criteria referencing. The recent SATs writing problems are a prime example of the likelihood of disagreement between individuals about judgements based on criteria. Is it acceptable or ethical to use data derived in this way to make judgements about staff and to use that in performance management?

Another question to consider is, if we are using APP for summative purposes, then why are levels to be derived 3 times a year, rather than at the end of the teaching period? It might be suggested that this is to track the progress of the children, as there is no requirement to report data to any outside agencies until the end of the year. Since the children can only cover a certain amount of material before the end of the year, the criteria can not be fully addressed until that material has been covered.

Staff found it very difficult to achieve consensus on many aspects of the guidelines, including some fundamental statements as bulletted above. They also failed to agree on a level for the work presented but this was not addressed as an issue. Staff were subsequently informed that the pupil was 'a secure 3' without any empirical evidence being presented to support that statement.

It seems like APP should have an 'assessment for learning' purpose – that is it is intended to be used primarily for formative assessment – in which case it needs to be applied to all the pupils rather than a selected 3. As such it has some obvious usefulness and can be a powerful tool. Criteria within the AFs provide some clarification of next steps in learning and can be helpful in guiding and motivating pupils as to what they hope to achieve. Used in this way, reliability is less of an issue. Problems arise when it is used in a 'high stakes' environment and no acknowledgement is made either of its unavoidable unreliability in determining levels and in the unmanageability of attempting to achieve reliability.

The images below represent annotation performed during the meeting. We identified evidence for the various AFs in the writing APP guidelines at the level we judged the work to be at.

Juliet Nickels : 14 Oct 2010 21:15 |  Tags: App Assessment Education Educational Formative Ma Maea Masters Reflection Summative University Warwick |  Comments (0) |  Close comments |  Report a problem

MAEA

Writing about web page http://www2.warwick.ac.uk/fac/soc/wie/teaching/masters/assessment/maeablogs/

(copied from my student blog… the picture of the graph is visible on that one – along with formatting!)

Excercise from MA in Educational Assessment, Summative Assessment module, session 1

Responses to Goldacre’s article in ‘The Guardian’ (Aug 21 2010) National: Bad Science: Mystery of the A-level

Initial reaction to the A-level results

Firstly – a confession. Saturday morning’s session was not the first occasion on which I had considered the A level results issue. Watching the media coverage, my initial reaction was two-fold – derisive and subsequently a tad peevish. Most of us have done A-levels at some time in the past. I sat mine in 1980 at a very well-regarded comprehensive (it was once listed as one of the top state schools in the country), with (at the time) supposedly high-achieving pupils. We would have been dumbfounded at anyone achieving 4-5 A grades at A-level in a year. That would have represented academic ability to a staggering degree worthy of a good deal of awe and even veneration! I think it would have been nigh on physically impossible to have covered the material necessary in the time given, for a start. I was intrigued to see how results had changed over time and came across this graph from the University of Buckingham Centre for Education and Employment Research.

I found it on the BBC website in their response to the ‘issue’.

http://www.bbc.co.uk/news/education-11012369

There are more of them here:

http://www.bbc.co.uk/news/education-11011564

More of that later.

Goldacre’s article

So does Goldacre have a particular argument? He seems to be saying that there is a definite issue in the public consciousness and that reactions to the high % of A grades at A level are polarised between ‘getting easier’ and ‘they are not’. Goldacre writes, ‘how do you know?’ and proceeds to outline the two positions and present some evidence.

Goldacre’s evidence is:

Students getting cleverer
IQ scores have to be adjusted for the Flynn effect.

Exams getting easier/students’ performance not improving
Royal Society of Chemistry research – Five Decade Challenge in 2008 – students’ scores in O-level papers from the 60s onwards showed a steady increase in time

TDA scores have declined and flattened out indicating no increase in performance ability over time in spite of increased A-level scores

Steady decline in scores in undergraduates’ basic maths skills from 60 university depts in maths , physics and engineering

Goldacre presents some arguments which may support either the notion that exams are easier or that pupils may just be performing better for other reasons, such as improved teaching methods, differences in subject choices, changes in exam focus etc. He pleads for research to clarify these issues.

What evidence would be appropriate – what research required?

Let’s assume for this purpose of considering how we might address his plea, that Goldacre’s presentation of it represents the actual issue. What would constitute evidence for either view and how could it be obtained? Given that the whole area of reliability in assessment is one fraught with problems, it’s difficult to see exactly how we could reasonably compare students’ performances across the decades in any convincing way. My own cohort of 1980s students are no longer the same people we were – sitting a current A-level in our subject (can one still do Zoology?) wouldn’t necessarily show how able we would have been to have sat the same paper back in the day. Similarly, the project to investigate how contemporary students coped with O-level papers of previous decades was not able to address all the variables that might account for the differences in results (though how much more interesting it would have been if the modern pupils had performed better in the old papers!). Is it possible that certain subjects (maths for example) have remained sufficiently static over time and the requirements for national qualifications so consistent, that papers from 1960 onwards could be used in a widespread test on performance with maths students, with levels awarded according to the relevant system of the time? (On a personal note, there must have been some assumed stability when I was revising for my O-levels at the end of the 70s; our practice papers reached back into the 1950s and were still considered relevant!)

Maybe, but the relevant system of the time presents a problem. In his response letter to the Guardian, KJ Eames stated that there had been a move from ‘norm referencing’ to ‘criterion referencing’ in awarding levels. So in analysing changes over time, do we apply one or either or both methods for grading in our investigation? Even the methods for monitoring comparability year on year have changed. For instance, Paul Newton writes (August 2007) that examining board researchers largely stopped using the common test method to monitor comparability in the 1980s because of the likelihood of plausible challenge on the grounds of uncontrolled variables. There is no comparability, therefore, between papers of the millennium and papers of the 70s.

Is research for evidence to support or to refute either position in this polarised debate even appropriate, however difficult? Possibly, in a general sense – there is some desire to know how the nation’s students are performing year on year – but probably not in this case specifically, because there is no like comparison. The A-level of 2010 is fundamentally not the same exam or qualification I was entered for in 1980. Even the way in which students study for it is different – the rise of coursework and the interference of the AS level are two examples.

That said, there clearly was a change in the 1980s – did we make a big move from norm-referenced levelling to criterion-referenced? That could certainly account for the nature of the graph above – stasis for two decades followed by a steady rise. If we had retained a norm-referenced system would we not have kept our A grades at roughly 8% forever?

How do I feel about the article and the issues?

I think Goldacre is presenting his perceptions of public opinion which may or may not be true. Perhaps someone should actually investigate whether the public really does hold those opinions. I don’t think the actual issue, however, is about the ease of the exams or the relative intelligence or academic ability of the students over time (I’m skeptical about the degree to which this can change in a human population of this size over a few decades, anyway). However, the statistics clearly show it’s easier to achieve an A grade at A level for whatever reason. This is what Robert Coe (April 2007) had to say about the 2006 results:

A level grades achieved in 2006 certainly do correspond to a lower level of general academic ability than the same grades would have done in previous years. Whether or not they are better taught makes no difference to this interpretation; the same grade corresponds to a lower level of general ability.

If that is true, is it important? It is in this respect: there is an impact on perception of the value of the qualification by the public, the students themselves, further education establishments, employers and previous students. The question arises as to what is the value of an A-level. If 25% of students achieve an A, then how do institutions award those who could demonstrate higher achievement? With an A*? Will there be a continuous devaluation process? A**? If the A-level of today is not the same as that of 1980 then why is it still called an A-level since that implies that it has retained certain characteristics over time? What is it even for as a qualification? Maybe it is the vestige of something which no longer serves its originally intended purpose.

Refs:

Coe, Robert (2007) Changes in standards at GCSE and A-Level: Evidence from ALIS and YELLIS – Report for the ONS by, CEM Centre, Durham University

Newton, Paul (2007) Techniques for monitoring the comparability of examination standards – QCA

Juliet Nickels : 11 Oct 2010 12:52 |  Tags: Maea Reflection |  Comments (1) |  Close comments |  Report a problem

Excercise from MA in Educational Assessment, Summative Assessment module, session 1

Excercise from MA in Educational Assessment, Summative Assessment module, session 1

Responses to Goldacre's article in 'The Guardian' (Aug 21 2010) National: Bad Science: Mystery of the A-level

Initial reaction to the A-level results

Firstly – a confession. Saturday morning's session was not the first occasion on which I had considered the A level results issue. Watching the media coverage, my initial reaction was two-fold – derisive and subsequently a tad peevish. Most of us have done A-levels at some time in the past. I sat mine in 1980 at a very well-regarded comprehensive (it was once listed as one of the top state schools in the country), with (at the time) supposedly high-achieving pupils. We would have been dumbfounded at anyone achieving 4-5 A grades at A-level in a year. That would have represented academic ability to a staggering degree worthy of a good deal of awe and even veneration! I think it would have been nigh on physically impossible to have covered the material necessary in the time given, for a start. I was intrigued to see how results had changed over time and came across this graph from the University of Buckingham Centre for Education and Employment Research.

I found it on the BBC website in their response to the 'issue'.

http://www.bbc.co.uk/news/education-11012369

There are more of them here:

http://www.bbc.co.uk/news/education-11011564

More of that later.

Goldacre's article

So does Goldacre have a particular argument? He seems to be saying that there is a definite issue in the public consciousness and that reactions to the high % of A grades at A level are polarised between 'getting easier' and 'they are not'. Goldacre writes, 'how do you know?' and proceeds to outline the two positions and present some evidence.

Goldacre's evidence is:

Students getting cleverer

• IQ scores have to be adjusted for the Flynn effect.

Exams getting easier/students' performance not improving

• Royal Society of Chemistry research – Five Decade Challenge in 2008 – students' scores in O-level papers from the 60s onwards showed a steady increase in time

• TDA scores have declined and flattened out indicating no increase in performance ability over time in spite of increased A-level scores

• Steady decline in scores in undergraduates' basic maths skills from 60 university depts in maths , physics and engineering

Goldacre presents some arguments which may support either the notion that exams are easier or that pupils may just be performing better for other reasons, such as improved teaching methods, differences in subject choices, changes in exam focus etc. He pleads for research to clarify these issues.

What evidence would be appropriate – what research required?

Let's assume for this purpose of considering how we might address his plea, that Goldacre's presentation of it represents the actual issue. What would constitute evidence for either view and how could it be obtained? Given that the whole area of reliability in assessment is one fraught with problems, it's difficult to see exactly how we could reasonably compare students' performances across the decades in any convincing way. My own cohort of 1980s students are no longer the same people we were – sitting a current A-level in our subject (can one still do Zoology?) wouldn't necessarily show how able we would have been to have sat the same paper back in the day. Similarly, the project to investigate how contemporary students coped with O-level papers of previous decades was not able to address all the variables that might account for the differences in results (though how much more interesting it would have been if the modern pupils had performed better in the old papers!). Is it possible that certain subjects (maths for example) have remained sufficiently static over time and the requirements for national qualifications so consistent, that papers from 1960 onwards could be used in a widespread test on performance with maths students, with levels awarded according to the relevant system of the time? (On a personal note, there must have been some assumed stability when I was revising for my O-levels at the end of the 70s; our practice papers reached back into the 1950s and were still considered relevant!)

Maybe, but the relevant system of the time presents a problem. In his response letter to the Guardian, KJ Eames stated that there had been a move from 'norm referencing' to 'criterion referencing' in awarding levels. So in analysing changes over time, do we apply one or either or both methods for grading in our investigation? Even the methods for monitoring comparability year on year have changed. For instance, Paul Newton writes (August 2007) that examining board researchers largely stopped using the common test method to monitor comparability in the 1980s because of the likelihood of plausible challenge on the grounds of uncontrolled variables. There is no comparability, therefore, between papers of the millennium and papers of the 70s.

Is research for evidence to support or to refute either position in this polarised debate even appropriate, however difficult? Possibly, in a general sense – there is some desire to know how the nation's students are performing year on year - but probably not in this case specifically, because there is no like comparison. The A-level of 2010 is fundamentally not the same exam or qualification I was entered for in 1980. Even the way in which students study for it is different – the rise of coursework and the interference of the AS level are two examples.

That said, there clearly was a change in the 1980s – did we make a big move from norm-referenced levelling to criterion-referenced? That could certainly account for the nature of the graph above – stasis for two decades followed by a steady rise. If we had retained a norm-referenced system would we not have kept our A grades at roughly 8% forever?

How do I feel about the article and the issues?

I think Goldacre is presenting his perceptions of public opinion which may or may not be true. Perhaps someone should actually investigate whether the public really does hold those opinions. I don't think the actual issue, however, is about the ease of the exams or the relative intelligence or academic ability of the students over time (I'm skeptical about the degree to which this can change in a human population of this size over a few decades, anyway). However, the statistics clearly show it's easier to achieve an A grade at A level for whatever reason. This is what Robert Coe (April 2007) had to say about the 2006 results:

A level grades achieved in 2006 certainly do correspond to a lower level of general academic ability than the same grades would have done in previous years. Whether or not they are better taught makes no difference to this interpretation; the same grade corresponds to a lower level of general ability.

If that is true, is it important? It is in this respect: there is an impact on perception of the value of the qualification by the public, the students themselves, further education establishments, employers and previous students. The question arises as to what is the value of an A-level. If 25% of students achieve an A, then how do institutions award those who could demonstrate higher achievement? With an A*? Will there be a continuous devaluation process? A**? If the A-level of today is not the same as that of 1980 then why is it still called an A-level since that implies that it has retained certain characteristics over time? What is it even for as a qualification? Maybe it is the vestige of something which no longer serves its originally intended purpose.

Refs:

Coe, Robert (2007) Changes in standards at GCSE and A-Level: Evidence from ALIS and YELLIS - Report for the ONS by, CEM Centre, Durham University

Newton, Paul (2007) Techniques for monitoring the comparability of examination standards - QCA

Juliet Nickels : 10 Oct 2010 16:52 |  Tags: A-Level Assessment Easy Education Educational Exams Goldacre Guardian Ma Maea Masters Performance Reflection Standards Student University Warwick |  Comments (1) |  Close comments |  Report a problem