Tag Archives: Schools

Reforming teacher training

This week the House of Commons Education Select Committee published its report on the teaching profession. This post gives the main points of our evidence to the Committee.

We think of Initial Teaching Training (ITT) as encompassing both the initial training and the probationary year. How should this be set up to produce the most effective teachers who will have the greatest impact on pupil progress? ITT plays two roles for the profession – training and selection with the emphasis typically placed on the former. Both are important and neither should be neglected, but we argue that the evidence suggests that if anything, selection is the more important, and this is our focus here. An important role for selection is completely standard for any professional accreditation system in either public or private sectors.

The key argument is this: the sharpest selection should be made at the point when the evidence on ability is strongest. The final decision on who can become a teacher should be made when we have accumulated enough evidence on the candidate’s teaching effectiveness. Where is this point in teaching? The two central relevant facts are that variations in teacher effects on pupil progress are very substantial, and that the future effectiveness of a potential teacher is hard to judge from their own academic record.

We believe that the current operation of selection in ITT (tight at the beginning, negligible thereafter) is the wrong way round. Instead, we should let a broader group try out to be teachers, but enforce a much stricter probation policy based around measures of teacher effectiveness in facilitating pupil progress. Full certification and an open-ended first job would only be granted once performance data showed a teacher to be effective. The expectation would be that only the most effective teachers would make it through to full certification.

Selection into ITT is about gaining a place on a course. The difficulty faced in identifying people likely to be good teachers is very relevant here. It is very hard to tell who will be a good teacher and therefore a high degree of agnosticism would be appropriate when faced with applicants. This is certainly true for selection based on objective criteria from the applicants’ own academic records. We know that these are unrelated to teaching ability, and so should be irrelevant in selection into ITT. Beyond that, even if selectors are highly skilled at spotting potential, and it is not clear that they are, it is impractical to ask each applicant to teach a practice lesson. Therefore, selection into ITT should be very broad, with a relatively low academic entry requirement. This of course is not the situation now, nor the direction of travel of current policy. The tightening of academic entry requirements into teaching is not helpful: it will restrict the quantity of recruits and have no impact at all on average teaching effectiveness.

Graduation from ITT should also be tough. Given that much of an ITT course is now school-based, time spent in the classroom will form an important part of the assessment. Arguably the classroom experience is the key part of the course. However, in such a short space of time it will not generate sufficient data for a robust and objective view of the trainee’s effectiveness. It will nevertheless allow the trainee to discover whether teaching is for them.

Once in a job in a school, the progression to being a qualified teacher should be very different to the typical experience now. The key decision on final certification should be made after a probation period of say three years and ideally, the probation should involve classes of varying ability and year group. The period probably cannot be less, though the appropriate length of the probation would need to be analysed properly, depending on the statistical reliability of any pre-hire indicators, school-based performance data, and the cost of being wrong. This is the point when enough data is available to make a reliable judgement on the effectiveness of the teacher. There should be an expectation that not all will make it through to final certification, and indeed only the most effective should be retained. The key judgement should be a minimum threshold of progress that the probationer’s pupils make. Obviously, the measurement of that progress and the parameters of the threshold require a great deal of careful work. Like any statistical data, estimates of teacher effectiveness will never be perfect, and a good deal of evidence over a number of years will be necessary to reach a decision, but this is clearly necessary to raise the average effectiveness of the teaching profession in England.

Another innovative route into teaching is through Teach First. In some ways this is a positive development, as it allows a lot of people to try out teaching and also gives the schools which employ them an ‘out’ at the end of the two years. On the other hand, it restricts entrants based on their academic background.

It is important to see the teacher labour market as a whole, and to see how the different stages of a teacher career fit together. It seems to be very hard to fire ineffective teachers. While the regulations on this have recently changed, generating a culture that encourages headteachers to take a more proactive stance seems harder. While this may change, it may be that the best way to reduce the problem of low-performing teachers is to make it very difficult for ineffective teachers to get into the profession in the first place.

These changes would make starting out on a teaching career much more risky financially. In order to maintain the same average lifetime expected income from the profession, the pay rate of those making it through to final full certification will need to be higher. And the lower is the chance of making it through, the higher is the full professional pay.

In summary, we think that the evidence shows that the selection aspect of ITT is completely the wrong way round. Selection is tight to get into ITT in the first place, but once in, progression to full certification is normal and expected. The process needs to be more appropriately agnostic about likely teaching ability in the first place. It should also allow a broader group of people to try out teaching, but have a much tougher probation regime before trainees be given final certification. It makes much more sense to make final decisions later once more evidence on effectiveness has accrued.

Who fails wins? The impact of failing an Ofsted Inspection

2 Replies

Rebecca Allen and Simon Burgess

What is the best way to deal with under-performing schools? This is a key policy concern for an education system. There clearly has to be a mechanism for identifying such schools. But what should then be done with schools which are highlighted as failing their pupils? There are important trade-offs to be considered: rapid intervention may be an over-reaction to a freak year of poor performance, but a more measured approach may condemn many cohorts of students to under-achieve.

This is the issue that Ofsted tackles. Its inspection system identifies failing schools and supervises their recovery. How effective is this? Is it even positive, or does labelling a school as failing push it to ever lower outcomes for its students?

It’s not clear what to expect. Ofsted inspections are often dreaded, and a fail judgement seen as being disastrous. It has been argued it triggers a ‘spiral of decline’, with teachers and pupils deserting the school, leading to further falls in performance. But it might also be a fresh start, with renewed focus on teaching and learning, leading to an improvement in exam scores. Equally, we might expect nothing much to happen: after all, the policy ‘treatment’ for those schools given a Notice to Improve is very light touch. It is neither strongly supportive (typically no or few extra resources) nor strongly punitive or directive (schools face no sanctions nor restrictions on their actions). Schools are instructed to focus intensively on pupil performance, and are told to expect a further inspection within a year. In addition – and possibly the most important factor – the judgement that the school is failing is public one, usually widely reported in the local press.

Our research shows that the Ofsted inspection system works. Schools that just failed their Ofsted significantly improved their performance over the next few years, relative to schools that just passed. The impact is statistically significant and sizeable. In terms of the internationally comparable metric of effect sizes, our main results suggest an improvement of around 10% of a standard deviation of pupil scores. This is a big effect, with a magnitude similar to a number of large-scale education interventions. Translated into an individual pupil’s GCSE grades, this amounts to a one grade improvement (for example, B to A) in one or two GCSEs. From the school’s perspective, the gain is an extra five percentage points in the proportion of pupils gaining five or more GCSEs at grades A*-C.

Our findings suggest that the turn-around arises from proper improvements in teaching and learning, not gaming to boost exam performance through switching to easier courses. First, the impact is significantly higher in the second year post visit than the first, and remains level into the third and fourth year after the inspection. So it is not simply a quick fix to satisfy the inspectors when they return twelve months later. Second, we find a stronger effect on the school’s average GCSE score than on the headline measure of the percentage of students gaining at least 5 good passes; if the schools’ responses were aimed at cosmetic improvement, we would expect the reverse. We also find similar positive effects on maths results and on English results.

It could be argued that these results are implausibly large given that the ‘treatment’ is so light touch and schools are given no new resources to improve their performance. The instruction to the school to improve its performance may empower headteachers and governors to take a tougher and more proactive line about school and teacher performance. This may not be a minor channel for improvement. Behavioural economics has provided a good deal of evidence on the importance of norms: the school management learning that what they might have considered satisfactory performance is unacceptable may have a major effect. The second part of the treatment derives from the fact that the judgement is a public statement and so provides a degree of public shame for the school leadership. Ofsted fail judgements are widely reported in local press and this is usually not treated as a trivial or ignorable announcement about the school. It seems plausible that this too will be a major spur to action for the school.

Where do we go from here? Our results suggest Ofsted’s identification of just-failing schools and the use of Notice to Improve measures is an effective policy, triggering the turn-around of these schools. We need to be clear that our research does not address the question of what to do about schools that comprehensively fail their Ofsted inspection. Possibly this light-touch approach can be extended. Since leaving the Headship of Mossbourne school to become the new Director of Ofsted, Sir Michael Wilshaw has argued that schools just above the fail grade should also be tackled: that ‘satisfactory’ performance is in fact unsatisfactory. Such interventions in ‘coasting’ or ‘just-ok’ schools are very likely to be of the same form as Notice to Improve. Our results suggest that this is potentially a fruitful development with some hope of significant returns.

This research is available on the CMPO website and the IoE website.

Why the new school league tables are much better … but could be better still

Leave a Reply

Rebecca Allen (IOE) and Simon Burgess (CMPO)

Tomorrow the new school league tables are published, with the usual blitz of interest in the rise and fall of individual schools. The arguments for and against the publication of these tables are now so familiar as to excite little interest.

But this year there is a significant change in the content of the tables. For the first time, GCSE results for each school will be reported for groups of pupils within the school, groups defined by their Keystage 2 (KS2) scores. Specifically, for each school the tables will report the percentage of pupils attaining at least 5 A* – C grades (including English and maths) separately for low-attaining pupils, high attaining pupils and a middle group. This change has potentially far-reaching implications, which we describe below.

This is a change for the better, one that we have proposed and supported elsewhere. Why? We believe that in order to support parents choosing a school, league tables need to be functional, relevant and comprehensible. The last of these is straightforward (though not all league table measures in the past have been comprehensible: Contextualised Value-Added (CVA) being the perfect example). ‘Relevant’ means that a measure has some relevance to the family’s specific child. A simple school average, such as the standard whole-cohort % 5 A* to C, is not very informative about how one specific pupil is likely to get on there. By ‘functional’ we mean a measure that does actually help a family to predict the likely GCSE attainment of their child in different schools. If a measure is not functional it should not be published at all.

The new group-specific component is comprehensible and is more relevant than the whole-cohort %5 A* to C measure. In our analysis of functionality, we show that it is as good as the standard measure, and much better than CVA.

It also addresses in a very straightforward way the critique of the standard league tables that they simply reflect the ability of the intake into schools, and not the effectiveness of the school. By reporting the attainment of specific groups of students of given ability, this measure automatically corrects for prior attainment, and in a very transparent way. This is therefore much more informative to parents about the likely outcome for their own children than a simple average. This of course is what value-added measures are meant to do, but they have never really become popular, and as we show they are not very functional.

However, the details of the new measure now published are problematic in one way. The choice of groups is important. We defined groups by quite narrow ten percentile bands, the low attaining group lying between the 20^th and 30^th percentiles in the KS2 distribution, the high attaining group between the 70^th and 80^th percentiles, and the middle group between the 45^th to 55^th percentiles. While clearly there is still variation in student ability within each band, it is second order and the main differences between schools in performance for any group will come from variation in schools’ teaching effectiveness.

However, the DfE has chosen much broader bands, and have defined the groups so that they cover the entire pupil population: the low attaining group are students below the expected level (Level 4) in the KS2 tests; the middle attaining are those at the expected level, and the high attaining group comprises students above the expected level.

This has one significant disadvantage, set out in detail by Rebecca Allen here. The middle group contains around 45% of all pupils, and so there is very significant variation in average ability within that group across schools. This in turn means that differences in league table performance between schools will reflect differences in intake as well as effectiveness, even within the group, thus partly undermining the aim of group-specific reports.

The chart below illustrates this for the middle attainment group (see here for more details). Each of the three thousand or so tiny blue dots shows the capped GCSE attainment for a group of mid-attaining pupils (on the DfE’s measure of achieving at the expected level at KS2) against the average KS2 score (i.e. prior attainment) of pupils at the school. The red dots plot the same relationship for our narrow group of middle attainers (the 45^th to the 55^th percentile). The chart shows very clearly that the performance among our narrow band is essentially unrelated to prior attainment, but the DfE measure for the very broad group does still favour schools with higher prior ability pupils.

We can speculate as to why the DfE chose to have much broader groups. There may be statistical reasons, pragmatic reasons or what can be termed “look and feel” reasons. Using narrow KS2 bands will correctly identify the effectiveness of the school, but will almost always be averaging over a small number of students. So the estimates will tend to be “noisy”, and induce more variation from year to year than averaging over bigger groups. The trade-off here is then between a noisy measure of something very useful against a more stable measure of something less useful. Our original measure was intended to balance these, the DfE have gone all the way to the latter.

A pragmatic reason is that some schools may not have any pupils in a particular narrow percentile band of the KS2 distribution. The narrower the band the more likely this is to be true. This would mean either null entries in the league tables, which might be confusing, or some complex statistical imputation procedure, which might be more confusing. The broad groups that cover the entire pupil population are likely to have very few null entries. Finally, the broad groups feel more ‘inclusive’, they report the performance of all of a school’s students. This is a red herring – the point of the tables is to inform parents in choosing a school, not to generate warm glows.

The new measures hold out the promise of improvements in two areas: choices by parents and behaviour by schools. Parents will have better information on the likely academic attainment of their child in a range of schools. Second, parents will be able to see more directly whether school choice actually matters a great deal for them: whether there are worthwhile differences in attainment within the ability group of their child.

The key point for schools is that performance measures have consequences for behaviour. If this new measure is widely used, it will give schools more of an incentive to focus across the ability distribution. It is still the %5 A* – C measure that is the focus of attention for each group, but now schools will have to pay attention to improving this metric for high and low ability groups as well as simply the marginal children with the highest chance of getting that crucial fifth C grade.

If one believes that gaming and focussing of resources within schools is a very big deal (and there is little quantitative evidence either way) then the new measures could have a major impact on such behaviour. Even if such resource focussing is second order, performance measures send signals on what is valued. These new league table measures will explicitly draw widespread media and public attention to the performance of low- and high-ability children in every school in England.

“All we want is a good local school”

2 Replies

Rebecca Allen and Simon Burgess

Two articles in the Times Education Supplement (TES) last Friday nicely illustrate the debate on school choice and school competition.

The first reports results from the British Social Attitudes Survey (BSAS), citing research by Sonia Exley, at the LSE, showing that most respondents thought that school choice was not a priority.

A familiar refrain in the school choice debate is that “all we want is a good local school”. There should be little doubt that this is indeed what most parents do want. We have used data from the Millennium Cohort Study to estimate the relative weights that parents place on the characteristics of primary schools. Unsurprisingly, school academic quality is positively valued, and distance between home and school is highly negatively valued. This makes a lot of sense: many parents have to make this journey four times a day. So, yes, people, do want a good local school.

But where does this take us? It is often said to imply that school choice is a distraction, an irrelevance. There is a side issue of whether choice is a good thing per se, as opposed to being functionally good. This is the thrust of the point above, that choice itself was not a priority, though the study also reports that 68% agreed that parents should have a basic right to choose their child’s school. Choice per se may become valuable once contrasted with the alternative of no choice.

But the main issue should be whether using school choice is a better way to allocate children to schools than alternatives. One alternative is implicit in the statement – children should go to their local school. In fact this gets a lot of support in the survey: the TES reports that 85% of respondents in the BSAS believe parents “should send their children to their local school”.

This idea would work well if families were not permitted to move house after the school admissions rule was changed. It is surely obvious why. We know that parents care a great deal about the school their child goes to. If the school allocation rule was simply “you will attend your local school”, then parents who were able to would ensure that their local school was the one they wanted by moving house.

It is quite possible that this would in fact lead to no less social segregation in schools, and almost certainly greater social segregation by neighbourhoods. While we found the relationship between school quality and moving house to be weaker than many might expect, this would undoubtedly be stronger in a world where your residence determined your school. It also does not do away with the concern about having to actually exercise a choice – it simply transfers it to a choice of neighbourhood and school combined.

So neighbourhood-based schooling would be very unlikely to resolve the issues of social segregation and choice-angst associated with choice-based schooling. It would also hand each school a local monopoly and, in the case of poorer families at least, a captive audience with no escape.

This connects to the second TES article, a leader on school competition. As the article notes, “Few things exercise critics of education policy more than the spectre of increased competition in our school system.” The argument balances the “un-school”-like, unorganised, chaotic and generally messy nature of competition with the potential for this to improve outcomes for students.

In fact, there is some evidence on this trade-off and what the net result of competition is (the article is mostly about competition for 6^th form entrants and allegations of mis-information, but the available evidence is about compulsory schooling). While the international evidence is mixed, the UK evidence suggests that there is at the very best a weak and small positive effect of competition on student outcomes; a review is here. The interesting question is why competition doesn’t appear to do much. The answer appears to lie in market failures in the schools market. If these could be addressed, it may be that a competitive threat might do more to raise standards in poorly performing schools.

Much of the furore about school ‘choice’ or ‘competition’ is misplaced. It is not choice between schools per se, relative to other allocation rules, that causes the perceived unfairness. The focus for objections should be the way that places in over-subscribed schools are allocated. The proximity criterion – who lives closest gets in – is operated in almost every non-selective school. This directly relates the chances of getting in to the most popular schools to family income, damaging social mobility in a very clear way. If some or all places at an over-subscribed school were filled by a random ballot, then school choice would seem a very different beast.

Finally, the competition article talks of ruined lives: “If no authority oversees admissions, plots likely pupil numbers or configures special needs support, the results won’t just be missed targets or dicey operating margins, but ruined, real pupil lives.” It is also true that that poor communities trapped with low-performing schools ruins lives, that unaccountable and coasting schools also ruin lives. The debate is about how best to avoid ruined lives, not whether or not they should be ruined.

A Report of Two Halves

1 Reply

Simon Burgess

We published some research last Friday showing that students perform less well in their crucial GCSE exams in years when there is a major international football tournament taking place at the same time. For example, the FIFA World Cup in the summer of 2010, or the UEFA European Championship next summer, both overlap in part with the GCSE exam timetable.

With the draw for the groups in the European Championship taking place earlier that day, much of the comment naturally and sensibly focussed on the specific issue of the impact of next year’s tournament on exam scores. This is important: we estimate that the concurrence of the exams and saturation media coverage of the football reduces exam scores on average by around 0.12 standard deviations of pupil performance and by a lot more for some groups who reduce their effort a lot. These groups tend to be from poorer areas and predominantly (but by no means exclusively) male students. Since these groups are already lower-performing groups, this means that education gaps will widen. We think of this impact arising through a reduction in student effort, with that time being spent instead on watching the football tournament. The variation in impact arises because of differing tastes for football, arising in turn from cultural norms and idiosyncratic factors, and from the differential effectiveness of an hour of study on exam performance.

However, there is also a broader significance to the research: finding that effort matters matters.

Recent research by economists has broadened out from the previous focus on cognitive ability, and a great deal of work has investigated the role of non-cognitive factors in educational attainment. Non-cognitive factors can be identified with personality traits (see Heckman), and one of the ‘big 5’ personality traits is ‘conscientiousness’, with the related traits of self-control, accepting delayed gratification, and a strong work ethic. Conscientiousness has been shown to be an excellent predictor of educational attainment and course grades. These aspects of self control and ability to concentrate are clearly related to the broad notion of effort we are using here. Our results on the importance of effort strengthen this evidence by isolating the effect of decisions on effort and time allocation in addition to the general ability to concentrate and exert self-control.

There is a great deal of policy interest in England arising from recent studies of US Charter schools with what is called a “No Excuses” ethos. This includes the KIPP (Knowledge Is Power Program) network of schools and schools in the Harlem Children’s Zone. These schools all feature a long school day, a longer school year, very selective teacher recruitment, strong norms of behaviour, as well as other characteristics. Some of the profession’s very top researchers have produced evidence showing that such schools produce very powerful positive effects on student achievement. While this overall effect could be due to different aspects of the KIPP/HCZ ethos, part of it is very likely to be increased effort from the students. Our results complement this by showing the impact of just a change in effort, and that that can have very substantial effects.

This matters for a number of reasons. First, unlike genetic characteristics, cognitive ability or non-cognitive traits, effort is almost immediately changeable. Our results suggest that this could have a big effect. The fact that we find changes in student effort to be very potent in affecting test scores suggests that policy levers to raise effort through incentives or changing school ethos are worth considering seriously. Such interventions would be justified if the low effort resulted from market failures due to lack of information on the returns to schooling, or time-inconsistent discounting. Second, the importance of a manipulable factor such as effort for adolescents’ educational performance provides evidence of potentially high value policy interventions much later than “early years” policies. This is encouraging, offering some hope that low performing students’ trajectories in life can perhaps be effectively improved even after a difficult environment early in life.

What are Free Schools for?

Rebecca Allen and Simon Burgess

As children start their lessons in the 24 Free Schools opening this week, a new experiment begins in English education. The founders and staff will have been working hard for this day over many months and no doubt all will wish the pupils and staff well. There has been a lot of political passion on both sides of the debate, but what is the significance of the Free Schools experiment likely to be?

It is not an experiment because of the “free” part – Free schools will enjoy essentially the same freedoms as Academies do. It is an experiment because now anyone can propose to set up a school, and attract the same per-pupil funding from the state as other local schools. Anyone can propose … but the vetting process to check whether the applicants are “fit and proper” to run a school is (rightly, in our view) so stringent that around 90% of the original applications were turned down.

Will the experiment work? Work for whom? The focus cannot just be on the 5000 or so pupils starting now, less than a twentieth of 1% of their cohort. If so, this would be an incredibly expensive and high profile way to change the education chances of such a small number.

The focus has to be on the systemic impacts of the reform. What impact will the new option of opening a Free School have on the local schools?

The main idea, the big hope for the policy, is that this will raise the “competitive threat” to under-performing local schools. Economists talk about market entry: the theory that the threat of new suppliers entering a market works to keep the incumbents efficient and responsive. Any un-served niches or excessive costs would draw in new companies to offer a better deal. Arguably, the prime function of the Free Schools experiment is to make the schools market “contestable”. This might sharpen the efforts of schools to raise their game to avoid losing their pupils to the new schools.

This might work. It is conceivable that really poorly performing schools may not wish to see their school supplanted by a new entrant, and push to raise attainment standards. But the UK evidence on the effects of competitive pressure on school effectiveness is not encouraging. Put at its best, there is only very weak evidence for small effects of competition. This is not true in every country, nor in every service in the UK, and there appear to be important market structural reasons why competition does not work to raise standards in schools.

This is not at all to say that all competitive mechanisms are pointless. Clearly any school system needs a strong policy for dealing with failing schools, something to act as a discipline on the performance of Headteachers and school governors. Free schools offer this in principle: in extremis, parents very dis-satisfied with their state school can opt out and set up their own school. But there are two reasons why Free Schools are unlikely to be the best answer to this. First, there are very significant set-up costs, both in time and energy from the founders, but also in the straightforward sense of acquiring premises. While currently these are being funded (very generously) by the government, this simply cannot continue if the policy matures and spreads. But secondly, it seems inconceivable that any local area with one Free School would be offered the resources for any others. So as discipline device, this is a one-shot game, not an on-going continued pressure on low performing schools, which is what is needed.

Despite the first rush of enthusiasm, it seems unlikely that Free Schools will act as a major stimulus to systemic higher performance. Perhaps the main problem was not the barriers to entry preventing many highly competent and motivated school-starters from setting up. In which case, we will need other policies to address poorly performing schools in disadvantaged neighbourhoods.

There are other important potential systemic effects, interactions between Free Schools and their neighbouring schools.

The first is school admissions. Free Schools will take part in coordinated admissions and, although, like Academies, being their own admissions authority, they will have to adhere to the Admissions Code. The big issue is whether the founders have any special rights to get their own children into ‘their’ school. It may be that this is one aim of the clause in the new draft Admissions Code allowing schools to give admissions priority to the children of “staff”. This is a difficult issue: it is quite understandable that those putting in time and effort to set up a new school want to get their own children in. But there are dangers: if this becomes the norm, we have a new route for particular individuals to get their child into a good school, by founding or acquiring a stake in, a Free School. In any case, the admissions criteria and the highly related question of how to pass on a “stake” in a Free School need to be clarified.

Another important spill-over effect concerns funding. The on-going current expenditure of Free Schools is funded on the same basis as Academies, so is not much different from maintained schools (just differentiated by not having to pay the local authority for central services). But the capital funding is another matter. Funding the acquisition and refurbishment of premises, estimated by the DfE at £120m, has been a substantial drain on a much-reduced schools capital budget. The funding for Free Schools’ buildings is a direct diversion of funds from other capital funding projects, many in far more needy schools and disadvantaged areas than the Free Schools.

But finally what Free Schools are likely to offer is scope for pedagogical innovation. This will not always be the case – some of the groups setting up schools may have rather conservative ideas on what constitutes a good education. But others will likely be set up by innovators with radical ideas on how to teach.

So perhaps that’s what Free Schools will end up being for: not really working as a spur to higher standards but acting as incubators for radical new teaching ideas. Very valuable indeed, but perhaps not what everyone was expecting.

Revising the Draft School Admissions Code

1 Reply

Rebecca Allen and Simon Burgess

From next week, officials in the Department for Education are going to be busy sifting through responses to the consultation exercise around the new School Admissions Code.

Two important issues in the proposed code relate to the priority given to school staff, and to random allocation. We believe that as they currently stand, these provisions will set back the goals that the Government has set for its education policy.

1. Prioritising the children of staff

Paragraph 1.33 of the code says: “If admissions authorities decide to give priority to children of staff, they must set out clearly in their admission arrangements how they will define staff and on what basis children will be prioritised.”

This suggests that admissions authorities are to be allowed to prioritise the children of staff, reversing the policy of recent Admissions Codes.

One group very likely to be included in most definitions of “staff” are teachers. For those teachers with children, this will add a new aspect to their decision on which school to seek a job at. Like many other parents, teachers will be keen for their children to attend high-performing schools.

Following the White Paper “The Importance of Teaching”, one of the leading education policy issues is how to attract the particularly effective teachers into the more challenging schools. Research evidence does not tell us whether teachers who are parents are on average more effective teachers, but there are two points to make:

This policy change will differentially increase the flow of applicants to high-performing schools. If the Headteachers of those schools are skilled at spotting effective teachers, then simply having access to a much bigger applicant pool will raise the average effectiveness of teachers the hired at those schools.
They are less likely to be novices, which is one of the few clear findings on teacher effectiveness, so in that sense alone teachers who are parents are likely to be more effective.

Given that, this policy change is very likely to work against any efforts to attract effective teachers to challenging schools, and thus set back the Government’s stated educational policy goals of narrowing the outcome gap between affluent and disadvantaged pupils.

The proposed code change will also complicate disciplinary procedures because firing a teacher from a school would also have implications for her/his children. This is likely to make it even less likely that headteachers will engage in robust performance management.

We know that any work-based privileges that are specific to particular establishments tend to cement people in that job and reduce turnover. Such privileges include health insurance, pension rights, and so on. This reduces labour mobility and typically will make the labour market less efficient. This proposed change will have the same effect in the teacher labour market as teachers will be less willing to move as it will disrupt their children’s education.

The core issue is where the most effective school staff work. It would clearly be in high-performing schools’ interests to define staff quite widely to attract highly motivated and effective governors and other staff. School leadership is key to excellent schools, ever more so as more schools become autonomous. If the children of governors were also prioritised for admission, then we would likely see an improvement in the quality of governors at high-performing schools and a decline at more challenging schools.

These points are about the effectiveness and efficiency of schools. There is also a fairness point about equity of access to high-performing schools. Some schools might define “staff” quite widely to include Teaching Assistants, lunchtime supervisors, and governors as well as teachers. In such cases, becoming one of those staff is an attractive route of access into the school, as an alternative to buying a nearby house. In extreme circumstances, individuals could offer to work for free as lunchtime supervisors, or could offer to provide goods or services to the school to become governors. While many may apply for such positions, schools will naturally choose the more able and articulate applicants, particularly those with a professional background. This will not help to foster more equal access to high-performing schools.

It may be that the provision to allow this prioritisation of staff was a way of finessing the problem of how to guarantee that the founders of free schools can get their children into ‘their’ school. If so, it might solve a problem affecting possibly 5% of pupils at the cost of the problems outlined here for the rest.

2. Random allocation

Paragraph 1.28 says “Local authorities must not use random allocation as the principal oversubscription criterion for allocating places at all the schools in the area for which they are the admission authority.”

This paragraph bans the use of lotteries as the main mechanism for resolving over-subscription across an LA. The problem of who is admitted to the high-performing over-subscribed schools is highly relevant to the Government’s goal of closing the outcome gaps between affluent and disadvantaged pupils.

The Government has stated that the primary goal of its social policy is to raise social mobility. One of the central ways in which where you were born influences your life chances is through the assignment of children to schools, governed by the Admissions Code.

Currently, the main mechanism to resolve over-subscription is proximity. It is clear that the widespread use of the proximity rule widens the socio-economic gap in the available quality of schooling. In fact, the gap in accessible school quality between rich and poor families widens by over 50% once a proximity criterion is imposed.

We can illustrate this straightforwardly, using data from the annual Census of all state school pupils in England. We can approximate the neighbourhood of a pupil as the area within 3km of her home, and calculate the average difference in the academic performance of schools in the neighbourhood of poor and rich pupils. As we would expect, more affluent families tend to live in neighbourhoods with higher quality schools. We can also use the database to estimate an approximate proximity criterion, and to look at the schools that a particular pupil could reasonably expect to be admitted to. The difference in school “quality” is now much starker, over 50% higher in fact. This is the impact of the proximity rule, and operates over and above the simple fact that rich and poor live in different places.

This shows the mean academic quality of primary school attended (measured by the mean Keystage 2 score achieved in that school) available to high and low socio-economic status (SES) families in England. ‘Neighbourhood’ is defined simply as an area within 3km of the student’s house; ‘Catchment’ is defined as schools that the student would have a very high chance of getting into based on residential location and the school ’s catchment area. Standard error bars are displayed on the levels.

Clearly some criterion has to be used to resolve over-subscription. One possibility for use in cities is a lottery. But by its very nature, a lottery ensures that places are allocated in a way that ignores social background. All those families who have applied to an over-subscribed school are entered into a ballot, and as many names are drawn out as there are free places available (that is, once looked-after children, children with special educational needs, and siblings of current pupils have been assigned slots).

It is clear that lotteries are not a panacea and are not without difficulties, as our own research has shown. However, they do offer one potentially important way of raising social mobility.

We wait to see which way the new Admissions Code turns in the end.

Reforming school performance tables

Leave a Reply

Rebecca Allen and Simon Burgess

Yesterday the Government published its response to the Wolf Review on Vocational Education. The Response sets out a number of proposals, accepting all of the Review’s recommendations. These include the eye-catching scheme to ensure that young people who do not achieve C grade in English and maths at age 16 continue studying them to age 19.

The response also proposes reforms to school performance tables. This is based on a recognition that schools’ behaviour in selecting qualifications for their students is strongly influenced by the incentive structure they face. A crucial component of this structure is the published school performance tables. These tables are important in influencing parental choice of school, and school leadership teams pay them a lot of attention.

From this year, the content of the performance tables will change quite significantly. The long-standing measure of the percentage of students achieving at least 5 A* to C grades will be retained. But in addition, a differential average points score will be published for each school, which provides information on how well the school does for students at the lower and upper ends of the ability distribution, as well as at the average:

“It is vital that performance indicators do not inadvertently cause schools to concentrate on particular groups of pupils at the expense of others. To avoid this we will continue to include performance measures, like average point scores, which capture the full range of outcomes for pupils of all abilities. In addition, from 2011 the performance tables will show for each school the variation in performance of low attaining pupils, high attaining pupils and those performing as expected.” (Wolf Review of Vocational Education, Government Response, p. 6).

This is a step forward. In our analysis we argued for exactly this measure: average GCSE points score, presented at three points in the ability distribution, low ability, average and high ability. Our criteria were functionality of the performance measure, relevance to parents and also comprehensibility. A measure is relevant if it informs parents about the performance of children very similar to their own in ability and social characteristics. It is comprehensible if it is given to them in a metric that they can meaningfully interpret. It is functional if it helps parents to answer the question: “In which feasible choice school will my child achieve the highest exam score?”. Overall, this performance measure came out on top. We also described ways that the information could best be displayed for parents: paper-based and web-based delivery mechanisms.

A second issue is that the “price” or GCSE-equivalent points of the new vocational exams seems set to change. Precise details of this are unclear at the moment. It is worth making the point again that schools will have an eye on the performance table impact of courses they offer to students. If vocational qualifications are to be worth less than they are at present receive, there is a danger that schools will not be keen to promote them to students who may be unlikely to score highly on more academic courses. In turn, this may make schools less keen to accept low ability pupils.

Of course, the old league table measure of percentage with 5 A* to C grades is staying. Perhaps there is a performance management version of Gresham’s Law and good performance measures will drive out bad ones. If parents come to rely more on this measure, the media will give it more prominence and the grip of the “%5A-C” measure on the public mind will finally begin to weaken.

Are school league tables any use to parents?

Leave a Reply

Rebecca Allen and Simon Burgess

Today is “school league tables” day. Performance tables are released for schools and colleges in England, reporting a number of different measures of the exam performance of their students. While much attention this year will focus on the reporting of the new “English Baccalaureate”, we ask a more fundamental question: are school league tables in general any use to parents? One of the major aims for school league tables is to support and inform parents in choosing a school for their child: but are they fit for this purpose? The answer is “yes” – we show that using school league tables does help parents to identify the school in which their own specific child will do best in her future exams.

Parents consistently rank academic standards as being one of the most important criteria for choosing a school. The performance tables provide outcome measures that are very widely reported and easy to get hold of. The idea is that parents can scrutinise the results and weigh up the merits of the local schools, considering the academic performance, travel distance, the child’s own wishes and other factors before deciding which schools to write down on their application form.

But this idea has been subject to a number of critiques. There are three main lines of argument. First, it is argued that differences in raw exam performance largely reflect differences in school composition; they do not reflect teaching quality and so are not informative about how one particular child might do at a school. Second, schools might be differentially effective so that even measures of average teaching quality or test score gains may be misleading for students at either end of the ability distribution. Different school practices and resources might be more important for gifted students or others for low ability. Third, it is argued that the scores reported in performance tables are so variable over time that they cannot be reliably used to predict a student’s future performance. After all, today’s league tables reflect last year’s students’ exams, but a parent wants to know how her child will do in five years time.

It is an empirical question how quantitatively important these points are: are league tables helpful or not? The question on academic standards that parents want answered is: “In which feasible choice school will my child achieve the highest exam score?”. We argue that the best content for school performance tables is the statistic that best answers this question.

To answer this question, we use the long run of pupil data now available to researchers. We can follow students through their years at secondary school and see how they did in the exams at the end; that is standard. But we can also use statistical procedures (details) to estimate the counter-factuals of how that student would have done if s/he had gone to a different local school. We can then ask: if families had picked schools according to the league table information available at the time, would that have turned out to have been a good choice in terms of subsequent exam performance for that specific child? Focussing on the simplest measure of the school’s %5A*-C score, the results show that while it certainly does not produce a good choice for everyone, it produces a good choice for twice as many students than it produces a poor choice for. So on average, a family using the schools’ %5A*-C scores from the league tables to help identify a school that would be good academically for their child will do much better than the same family ignoring the league table information.

So are the league tables useful for parents? Definitely. Can they be improved? Certainly. The measures included in the performance tables should be judged according to their functionality, relevance, and comprehensibility. The test of functionality is the analysis just described. A measure is relevant if it informs parents about the performance of children very similar to their own in ability and social characteristics. It is comprehensible if it is given to them in a metric that they can meaningfully interpret. In fact, none of the current leading performance measures score very well across our three criteria. We have proposed an alternative measure that performs better on these criteria. No measure can be perfect because there are important trade-offs between relevance, functionality and comprehensibility: the more disaggregate the form in which performance tables are provided (increased relevance), the less precision they will have (decreased functionality). The more factors are taken into account in describing school performance for one specific child (increased relevance), the more complex the reported measure will be (decreased comprehensibility). Any choice on the content of league table information has to make decisions on these trade-offs.

“Systemic failure” in Welsh Education

Leave a Reply

Simon Burgess

The release this week of the latest round of international comparative education results produced some fascinating results. Not least of these was the outcome for Wales, characterised by the Wales’ Education Minister as alarming and “unacceptable”.

The PISA (Programme for International Student Assessment) results derive from a standardised international assessment of 15-year-olds, run by the OECD. They show that Wales has fallen further behind since the last tests in 2006, and scored worse than before in each of reading, maths and science. Scores in Wales have fallen relative to England and are now “cast adrift from England, Scotland and Northern Ireland”. The Wales Education Minister, Leighton Andrews, described the results as reflecting “systemic failure”.

What might that systemic failure be? One leading candidate is highlighted in our recent research on accountability mechanisms for state schools. We argue that the decision in 2001 by the Welsh Assembly Government (WAG) to stop the publication of school performance tables or “league tables” has resulted in a significant deterioration in GCSE performance in Wales. The effect is sizeable and statistically significant. It amounts to around 2 GCSE grades per pupil per year; that is, achieving a grade D rather than a B in one subject. This is a substantial effect, equivalent to the impact of raising class size from 30 to 38 pupils.

Although our results are based on a study of the GCSE scores school-by-school, Figure 1 gives a very stark impression of the overall effect. Students in England and Wales were performing very similarly up to 2001, but thereafter the fraction gaining 5 good passes has strongly diverged.

We take each secondary school in Wales, and match it up to a very similar school in England. This “matching” is based on pupils’ prior attainment, neighbourhood poverty and school funding among other factors. We then track the progress (or value added) students make in these schools before and after the league tables reform, comparing the Welsh school with its English match. Our analysis explicitly takes account of the differential funding of schools in England and Wales, and the greater poverty rates found in neighbourhoods in Wales.

Why should the removal of school league tables lead to a fall in school performance? Part of the effect is though the removal of information to support parental choice of school. The performance tables allow parents to identify and then apply to the higher scoring schools, and to identify and perhaps avoid the low scoring schools. This lack of applications puts pressure on the latter schools to improve. But this is not all of the story. Perhaps as important is the simple public scrutiny of performance, and in particular the public identification of the low scoring schools. This “naming and shaming” means that low scoring schools in England are under great pressure to improve, whereas the same schools in Wales are more able to hide and to coast.

Our work has attracted criticism, including a charge of using an “ideological theory” from teacher unions . A more thoughtful critic has accused us of a “howler” in the analysis: not noting the introduction of the original GCSE-equivalent qualifications. In fact, since these were introduced equivalently in both countries they simply net out of the comparison.

Responding to our research, the Welsh Assembly Government said “wait for the PISA results”. These results are now in, and do not make happy reading. No doubt there are many factors underlying the relative performance of Wales and England, but the diminution of public accountability for the country’s schools is surely one of them.