Category Archives: Education

Interpreting the numbers of school admissions – is the first preference offers rate too high?

Figures were released yesterday showing how many families received an offer from their first preference school. The headline number was 84% for secondary schools: that is, 84% of families were offered a place in the school that they put top of their application form. The coverage yesterday mostly focussed on the overall supply of school places. Maybe there is also a sense that “only” 84% got their first choice.

Acknowledging the disappointment or worse that individual families will feel at missing out, how should interpret that 84%? It’s obviously affected by two things, both the choices made, and the number of places available. ‘Demand and supply’ if you like – and that’s not inappropriate here as the school admissions algorithm is what acts as the market clearer.

Imagine a country a bit like England, but much simpler, more abstract. And also imagine that all that families care about in schools is their academic quality. In that country, as in England, 79% of schools are rated as Outstanding (26%) or Good (53%). If all schools are about the same size, then 79% of school places are in Outstanding or Good schools. Suppose that everyone can access at least one Outstanding or Good school, and that there everyone applies to one of those schools. If schools and people are spread around the country in a reasonably even and regular way, then 79% of them will get in and the remainder will be offered places in schools less than Good.

In that situation, 84% getting their first choice seems ok.

But what if everyone was a bit more ambitious in their choice, and everyone put an Outstanding school as their top choice? Why not?

Only 26% would get their first choice. Suddenly, in this abstract, regular country, 84% seems to suggest a lot of unambitious choices, ones that are likely to succeed rather than a ‘true’ first choice.

Of course, our country is not simple and regular like that. There is geographical clustering of Outstanding schools, so some families will face a much higher chance of getting into an Outstanding school. In other places, there may be no Outstanding schools, so even families looking for the very best school academically can only out a Good school top, and they too have a pretty high chance of getting that. So taking that into account, even “everyone chooses an Outstanding school” would result in more than 26% getting their first preference, but not as high as 84%.

So I think we can say that the 84% might be too high for comfort. Maybe it reflects a school admissions system that favours those who can buy access to the best schools through their home’s location. The key role of proximity in resolving who gets into the popular schools keeps many of the Outstanding schools out of the choice set of poorer families. We need to change this, as I discussed here.

Part of the solution to the teacher shortage?

 

What can we do about teacher recruitment? Yesterday’s National Audit Office Report has provided some useful clarity and confirmed that the Department for Education has missed its recruitment targets for four years now. A number of commentators have highlighted the shortfall of new teachers through the year, though the DfE has argued to the contrary. The range of views on the shortfall suggests the gap may be between zero and 18% (quite a range!)

A number of suggestions for policy responses have been made: pay all teachers more; pay some teachers more; review the teacher workload; reform the Ofsted process to make it less stressful. These may well be great ideas; but they are all expensive or very expensive, and slow or very slow to implement.

Here’s another idea, which could contribute to raising recruitment, though undoubtedly would not wholly solve the problem. This is more or less costless (apart from the cost of the extra teachers obviously) and easy to implement.

We have imposed a major but pointless restriction on the pool of potential teachers – we can just drop that restriction. We can, effectively, stop shooting ourselves in the foot.

The point is this: there is a general view threading through the teacher recruitment system that applicants with better degrees will make better teachers. I’ll illustrate that in a moment. But all the statistical evidence we have on teacher effectiveness says that that is not true: a teacher’s ability to raise the attainment of her pupils is unrelated to her own academic qualifications.

There are a number of explicit points in the system in which the boundary between getting a II.1 degree and a II.2 degree is crucial. I would argue that these create a mindset on appropriate qualifications for good candidates that pervades the system much more widely. For example, in terms of bursaries for teacher training, these are only available for people holding a II.1 or better in some subjects. The official ‘Get into Teaching’ website makes this clear. This is not true in all subjects: for sciences, maths and languages, the applicant’s degree class makes no difference to the bursary. So I repeat that this proposal would have only an indirect effect on those subjects.  Another example is Teach First, which requires a II.1 or better for its applicants.

So while the II.1 restriction certainly does not apply universally for all teacher recruitment, it is likely to have a much broader impact on the views of recruiters and selectors on what a good teacher looks like.

Over the last decade or so, economists have focussed a lot of research effort on teacher effectiveness. The research evidence shows clearly that teacher effectiveness is unrelated to the teacher’s own academic qualifications. Teachers who themselves got a First class or a II.1 degree are no more effective teachers than those who got II.2s. The NAO Report hints at this too.

The one study for England that measures this (our own) makes this point. The much more numerous research studies in the US show this (see my review here). Among researchers, it is an uncontroversial finding. Even researchers who set out to show that having a Master’s degree should help, end up finding it doesn’t (Ladd and Sorensen, 2015, reference in here).

So the explicit or implicit restriction of teacher recruitment to those getting at least II.1s is pointless – it does not achieve its aim of raising average effectiveness. But it is harmful, it restricts the hiring pool significantly. At the risk of repetition: this is not about a quantity – quality trade-off in hiring – by relaxing this constraint we can seek more quantity at no cost in quality.

How big a difference might this make? It’s very hard to say at a high level of generality. The NAO commented that uncoordinated data sources make this a difficult area to track.

Reaching for a very small envelope, turning it over and starting to scribble: the percentage increase in recruits is equal to the percentage increase in the hiring pool times the relative likelihood of applying from the new group times the relative likelihood of someone in the new pool being acceptable. If we assume the current hiring pool is all with a II.1 or better, and we are proposing to expand this to include people with II.2s as well, using HESA data (chart 9) that’s an increase of 35%. Of course there are other routes in as well as from the flow of new graduates, as well as people from outside the UK, so let’s call it 30%.

We know that students with II.2 degrees have lower rates of return than those with higher classes, so presumably face worse alternative job opportunities. They might therefore be more likely to apply to teaching posts. But to be cautious, and underestimate the likely effect, I assume a relative application rate of 1. For relative acceptability, I need to account for the fact that in some subjects, applicants with II.2s are already accepted. I will also assume that the marginal II.2 candidate is less acceptable than a II.1 candidate. Overall, let’s try a relative acceptability rate of 0.4.

Multiplying these numbers together (0.3 x 1.0 x 0.4) yields a potential increase in recruits of 12%. The previous paragraphs make clear how very rough an estimate that is, but you can choose your own numbers to try. So it might be that this proposal might increase recruits by around 10%, and recruits of the same expected effectiveness at teaching.

Why not remove the II.2 restriction? And work to counteract the view that teachers with II.2s will be ineffective teachers. It won’t reduce average teacher effectiveness and it will increase the applicant pool.

Just to finish, it is worth re-emphasising that while teacher numbers are important, much more important is the average effectiveness of teachers. All the evidence shows that being taught by an effective teacher relative to an ineffective teacher has a dramatic impact on attainment. To illustrate: having all effective teachers relative to all ineffective teachers for just one GCSE year wipes out half of the poverty gap in attainment. Getting more teachers into our classrooms matters, but understanding how to raise average effectiveness is the big prize.

 

 

Teacher performance pay without performance pay schemes

Author:  Simon Burgess

Teacher performance pay without performance pay schemes

Amid the macroeconomic gloom, the Autumn Statement contained a line about teachers’ pay. The School Teachers’ Review Body recommends “much greater freedom for individual schools to set pay in line with performance”. Consultations and proposals are expected in the near future.

But simply giving schools the freedom to do this may be a rather forlorn hope of anything much happening. It is not clear that there is a substantial demand from schools for performance-related pay (PRP) schemes that has only been thwarted by bureaucratic restrictions. It is hard to see high-powered, tough-minded PRP schemes being introduced by more than a handful of schools, not least because we have not seen large scale deviations from national pay bargaining in academies in England despite their new freedoms to do so.

If that path seems unpromising, there are other ways of facilitating a greater reflection of performance in pay, discussed shortly. But first – is PRP for teachers a good idea in the first place? Does it raise pupil attainment? What are the ‘side effects’?

This is a question that economists have produced a good deal of research on. And to summarise a lot of diverse work briefly, the international evidence is mixed. Those on both sides of the argument can point to high quality studies by leading researchers that find substantial positive effects, or no effects. In both cases, interestingly, there appeared to be little evidence of gaming or other unwanted effects of the incentives.

There is little evidence specifically for England. Our own research found a substantial positive effect of the introduction of a PRP scheme, but given the varied results found elsewhere it would seem unwise to place too much weight on this one study. The underlying performance pay scheme was poorly designed but nevertheless had a positive effect on the progress of pupils taught by eligible teachers relative to ineligible ones.

And design is key. There are many reasons why a simple high-powered incentive pay scheme might be detrimental to pupil progress, which we have discussed here and here. These include the fact that teachers have multiple tasks to do, the problems of measuring the outcomes of some of those tasks, the complex mixture of team and individual contributions, and the potential impacts on implicit motivation. The overall message is that incentives work, but schemes have to be very carefully designed to achieve what the schemes’ proponents truly intend.

There is another way to facilitate a closer link between pay and performance that does not require any school to introduce a performance pay scheme.

Published performance information in a labour market can change the way that the market rewards that performance. The critical features are first that the organisation’s own output depends in an important way on this performance characteristic of an individual; second that the organisation has some discretion in the pay offers it can make to new hires; and thirdly that the performance information is public – is available and verifiable outside the current employer. In this case, the pay structure of the market will reflect the performance rankings: high-performing individuals will be paid more.

In teaching, the first two of these three conditions are met: teacher quality matters hugely for schools, and schools have some discretion over pay. Now, suppose we had a simple, useful and universal measure of each teacher’s performance in raising the attainment of her pupils (obviously we don’t at the moment; I come back to this below), and that this was published nationally, primarily for the attention of Headteachers. The idea is that Headteachers trying to improve the attainment of their pupils would be on the look-out for high performing teachers when they had a vacancy to fill. Armed with this performance information, they might try offering a higher wage (or something else – it doesn’t have to be money) to tempt them to join their own school. Equally, the teacher’s current school may respond by raising the offer there.  Over time, this process will tend to raise the relative pay of high-performing teachers relative to low-performing ones, whom no-one is trying to bid for.

This idea should not be a strange one. A number of professions have open measures of performance. Just today it is reported that performance measures for more surgeons will be made public in the summer of 2013; this is already true for heart surgeons.

It is well-known that PRP does two things: it motivates and it attracts. The outcome for pay described here will tend to make teaching more attractive to people who are excellent teachers and less attractive to those who aren’t.

There are a number of problems with this idea, though perhaps less than might appear at first glance.  First, it could be argued that a performance measure derived from teaching in one school is not relevant to teaching in another school. Obviously each child and each school is unique, but it seems very unlikely that there is no commonality of context between one school and the next. Observation suggests this: teachers moving from one school to another are not counted as having zero experience, and Headteachers are often appointed from outside a school.

Second, there might be a fear that the teacher labour market would become chaotic, with everyone churning around from school to school in search of a quick gain. We have to recognise that there is substantial turnover of teachers now < http://www.bristol.ac.uk/cmpo/publications/papers/2012/wp294.pdf >. But the main point is that it does not require much actual movement to make the market work. Schools can make counter offers to try to retain their star teachers and the end result is the same – higher salaries for high-performing teachers.

Third, any measure would be noisy, partial and imperfect. Of course, all such measures are. Whether a measure is perfect is not really the question, the question is how noisy and imperfect is it, and whether it contains enough information to be useful. One advantage in this case is that the consumers of these performance indicators are the people best able to judge their usefulness and their shortcomings: Headteachers. If such metrics are not useful, Headteachers will simply ignore them; there would be no compulsion to use them.  Even in labour markets with some of the most detailed and finely measured performance indicators (for example, football or baseball) there are many moves between employers that do not work out. It is worth re-emphasising that these performance measures are bound to be imperfect and incomplete, but broad measures of performance may nevertheless be very useful.

There are useful parallels to be drawn from another profession: academics. For academics, the combination of very detailed and public performance information and a context where research performance matters a great deal to universities seems to have had a substantial effect on academics’ pay.

The Research Assessment Exercise (RAE) and more recently the Research Excellence Framework (REF) have made a strong research performance very important to a university’s standing and its income. But the critical factor for academics is that an individual’s research performance is public knowledge, through very detailed recording of the impact of their research papers. Departments and universities aiming to improve their ranking seek out star researchers and attempt to bid them away with higher salaries (plus other things such as research facilities). These offers may well be matched by their current employer, but the end result is that salaries now seem to be much more closely correlated with research productivity than before the RAE/REF (I say “seem” as there does not appear to be any evidence on this, so this is casual empiricism). This is a lot of what drives many young researchers to put in very long work hours: having a paper published in a top scientific journal early in a career has a substantial lifetime payoff even in a world with few or low-powered incentive schemes. If you check out academics’ websites you will invariably see their academic output prominently displayed.

Again, an important feature is that these indices of research output are largely consumed by other academics who are aware of their strengths and weaknesses. So although they are far from perfect, they are used by precisely the people best placed to calibrate their usefulness appropriately.

If we are to go down a path of tying teacher pay more closely to performance, and yet respect the rights of increasingly autonomous schools to determine their own pay systems, then this might be an option to consider.  The challenge is to devise a measure that is simple, useful and universal. It would measure the progress made by the pupils that teachers taught, it would have to deal with normal variations in performance by averaging over a number of classes and a few years, and be on a common metric.  This is not straightforward, but if it gave rise to a robust broad measure of performance it could form a part of performance pay for teachers, and performance management more broadly. It could also have substantial effects on the pay of high-performing teachers.