So good afternoon everyone; it’s very good to be here. Very good to meet you. My name is Ian Bauckham, Chief Regulator at Ofqual, and what I’m going to be talking about this afternoon is a little bit about qualifications in general, something specifically about V Levels, and then some reflections on issues like on-screen assessment and the role of AI in assessment. So I’d like to just start by pausing for a moment an...
So good afternoon everyone; it’s very good to be here. Very good to meet you. My name is Ian Bauckham, Chief Regulator at Ofqual, and what I’m going to be talking about this afternoon is a little bit about qualifications in general, something specifically about V Levels, and then some reflections on issues like on-screen assessment and the role of AI in assessment. So I’d like to just start by pausing for a moment and thinking about what my job involves. So, my job is to safeguard a national asset that we have, which is our qualifications system. Our qualifications do have, as many of you will know, a global reputation, and are actually a central part of this country’s soft power network globally and internationally. And the qualifications our 16 to 18-year-olds take certainly have the function of opening doors for them, for the people who take them – being a passport to opportunity – and command levels of public trust that many politicians would be deeply envious of. And I think they’ve really got a place in our national culture, in our national conscience as well. Today’s teenagers’ parents, for example, will have done GCSEs themselves and in many cases both their parents and grandparents will have done A Levels. So, these things are deeply rooted in our collective psyche. But they also play a role in shaping the educational experience that students have. So quite rightly in a democracy, through our governments, we make collective decisions about what young people should learn at school and we reflect those ambitions that we have for them in the qualifications we design. And those qualifications in turn shape teachers’ and schools’ activities and their priorities. The public’s trust and confidence is essential for our qualifications. But public confidence isn’t just an abstract throwaway, something we pick up from opinion polls. The reality of it is things like: a young person, receiving results in the summer that they can be genuinely proud of; a teacher, literally bursting with pride, as I was often myself when I was a teacher in school, at their class’s achievements when they get those results in August; a university confidently making decisions about admissions, trusting the qualifications that students bring to tell them what they’ve mastered; and an employer, likewise, confidently making decisions about who to employ. So public trust is really central, really important. But sustaining all this, both the trust domestically and the reputation of qualifications globally, doesn’t mean standing still. It absolutely means adapting, improving, meeting the challenges of a changing world. But all the while conserving the core values, which are at the heart of our qualification system, which I would say include fairness, trust and transparency. So I want to start then, as I said at the beginning, just by taking a little bit of a deeper dive into one of the central planks of the changes that are currently in scope; and that’s V Levels. So for decades now, since they were first introduced in 1952, A Levels have dominated the post-16 qualification landscape. They’re the qualifications most people are likely to have heard of internationally. Then in 2020, the introduction of T Levels marked a shift in the way we think about qualifications post-16. And now the interim report of the Milburn review has rightly highlighted the need for much more to be done in that 16 to 18 area, with the number of NEETs that we have breaching the one million mark. Clearly something is needed between the large, technical T Level on the one hand and the smaller academic A Level on the other. So, a vocational pathway to help guide young people towards work, higher education and training, and something that’s straightforward to navigate and understand. And the introduction of V Levels will create that stronger, vocational route alongside A Levels and T Levels. So we’ll have A, V and T, that will be the new post-16 landscape. This clear shaping of the landscape will bring clarity to choices that students make and parity across the 3 routes. So clarity and parity. Combined with improved opportunities also for those not yet ready for more advanced study, this all speaks directly to some of the challenges that Milburn has so powerfully pointed us towards. So let’s just look for a moment at what V Levels are actually for. You might be asking: are these just rebranded BTECs? Or are they really for students in my kind of school or college? These are common questions. So let’s look at the detail. V Levels will be level 3 qualifications. So at the same tier, if you like, in the qualification structure as A Levels. They’ll be for students who have got a decent clutch of stronger GCSE grades already. In other words, have securely proven that they can work at level 2 and are ready for the next level up. But only a minority of teenagers at that stage know with certainty what they want to focus on later on, what career path they’re seeking. Most want, to some extent at least, to keep their options open. So, with that in mind, the V Level is a smaller qualification than a T Level – taken with maybe 2 other V Levels, or indeed, with A Levels – to make a tailored study program for students at this age. A V Level will introduce students to a vocational field, it’ll teach them knowledge and skills associated with that field as well as point them to the range of jobs and occupations available in that field. And although V Levels are absolutely applied qualifications with a strong, practical element, they don’t actually mandate an industry placement as T Levels do. They will be entirely teachable in school sixth forms as well as in larger colleges. And that’s a really important part of ensuring that as many young people as possible can access the opportunities that these courses bring. So isn’t a V Level just a rebranded BTEC? You might still have that question in your head. So, let’s look for a moment at what a BTEC actually is. Well, first and foremost, BTEC itself is a private, proprietary brand name that belongs to Pearson. Only Pearson-owned courses can be called BTEC . And look, sure, all qualifications have to be approved by the government if they’re going to get public funding, but Pearson like any other board operating in this area currently has significant freedom to determine what is taught in a BTEC, in common with other boards and courses at this level. And I don’t know about you, but I for one think that the proper place for critical decisions to be taken about what a large proportion of 16 to 18-year-olds actually learn is our democratically elected governments, rather than individual organisations, experienced though many of them may well be. It certainly wouldn’t be a principle we’d accept at GCSE, because all GCSE students, regardless of which exam they take, have the quality guarantee of knowing that what they’re learning and what they’re being graded on, regardless of the board they take, is the same underlying content. And I don’t see why post-16 vocational students shouldn’t benefit from that same guarantee. They should be given knowledge and skills we know the jobs market and the economy need, and that’s good for them as well. What our students learn is as much a public good, a public asset, as the qualification system as a whole. And I’ve mentioned BTEC a few times – while BTEC is the most well-known of the privately owned brands currently, it’s by no means the only one. Other boards, other awarding organisations also offer level 3 vocational courses. But here’s the thing, you can’t make a direct comparison currently between the grades from one awarding organisation to another in these qualifications. There’s no requirement for them to use either the same grading scale, or the same standard setting. And that means that employers and others who use these qualifications cannot be absolutely sure that, currently, say, a merit from Pearson is the same thing as a merit from OCR, or any of the other boards in this area, because the content will vary and the standard setting approaches may also vary. And again we wouldn’t accept that situation in GCSE, so I don’t see why it’s acceptable for post-16 students. It’s this kind of long-standing issue with these qualifications that the V Level reform will set right. And I think that’s a good thing for us to be doing as a country. So the first 3 V Levels will be in education in early years, accounting and finance, and digital systems and data, and they’ll be available for first teaching in September 2027 - and the second much larger tranche will follow next year. Now, looking to the future is not just about what qualifications are available, it’s also about the how. And the how, of course, must include how qualifications work in the fast evolving world of technology, including AI. So let’s start with one apparently straightforward and specific question – should exams move on screen? Straightforward, surely? Well, on closer scrutiny, it turns out maybe it’s not quite so straightforward. We consulted as Ofqual on this question precisely earlier this year and we’ll publish our decisions on this in the light of that consultation soon. Now it’s absolutely no secret, because they tell the educational media often, that some exam boards would like us to “be braver, move faster”, prevent us “getting behind the curve” in this area. But of course, my job is not to do what the exam board wants me to do. I must act in the interests of the qualification system, which, as I’ve told you, I consider to be a public asset. The public has to want and trust innovation of this sort and believe it can be done fairly. And it turns out even that the simple lines that say “the world is going on-screen so it’s about time exams caught up”, those and similar challenges actually beg as many questions as they answer. We do sometimes, I feel, in this country, have a slight tendency to assume that we must be somehow behind the curve internationally. We should look at what’s happening in other countries, I get told, and that’s an appeal that really means “catch up and move on-screen”. So let’s take a little look around the world. When we do that, actually it turns out there are very few examples of whole large systems like ours, of high-stakes qualifications, moving directly on-screen. There are certainly some examples of small systems doing so, or trying to do so, and there are as many examples of such small systems pausing or reversing as well. And in fact, Sweden and Norway in slightly different contexts have just done precisely that, both of them. Sometimes the question that’s behind a decision to pause or reverse is, can we do this? In other words, can we do the sheer deliverability at scale? Do we have sufficient confidence it will work on the day? So can we do this? But sometimes, and actually I think increasingly, the question is not so much “can we do this?” but rather, “should we do this?” And behind that question is concern about, for example, screens in schools, screens in childhood, whether from the perspective of mental well-being, or whether from the perspective of impact on real learning. As everybody will know, there is now a significant global momentum behind the move to get mobile phones out of schools for both sets of reasons. Some of you might know the work of someone called Jared Horvath, who has written about this, or maybe you’ve listened to him online. Anna Stokke’s excellent podcast Chalk and Talk, I can recommend, has interviewed him. It’s well worth a listen. When I listened to it, I was particularly struck by the potent, distraction power of digital devices that he points to. Qualifications do exert a strong influence on what goes on in classrooms. So a key question has to be, if high stakes exams, high stakes assessments, move on screen, then what effect will that have on what happens in classrooms? And there are other questions, of course, that we need to consider including. Fairness, social justice, particularly difference in access to devices, familiarity with digital platforms, all levels of digital skills and confidence. Another is the readiness, the ‘equipped-ness’ of schools and colleges to introduce large-scale, simultaneous computer-based exams. And dare I say it, and I’m not thinking of anything in particular here at all, dare I say it, another is the capacity of the system, both exam boards and technology providers, to design interfaces which are genuinely accessible, and their capacity to run large tech systems reliably enough and to be trusted with the aspirations of thousands of young people, as well as handle their nervous energy on the day. We will deal with all these questions when we make our decisions over the summer and publish in the autumn. So, let me turn then to the major question of AI, which I’m separating out from on-screen assessment, because I think it is a different question. On AI, I think there are 3 main areas for us to think about in relation to qualifications. Firstly, do we even still need qualifications that prove that what students know and can do when we are told AI will be able to do everything anyway? If the answer to that question is, no, we don’t need them, then obviously, it’s game over. I’ll go home. But if the answer is, as I will argue, yes, we do need them, then a second question that flows from that is: what should we do about the misuse of AI in qualifications? Or, to give it perhaps a more accurate designation – AI-driven fraud. And the third, very big question actually, is: to what extent should we let AI run the whole exams and qualifications system for us? Should we allow that end-to-end, in part, or not at all? So let me reflect on each of those 3 questions in turn. Do we still need qualifications that assess and prove what students know and can do? Now look, the workplace of the future, and increasingly the workplace of the present of course, is going to be AI-enabled. No discussion there. But that doesn’t mean, as is sometimes assumed, that humans won’t be needed anymore. The future workplace will actually need higher level expertise to augment and add value to AI. Specialised knowledge, finely-balanced judgement, they’ll be absolutely central to that. So I think actually, we can rapidly put that question to bed. We won’t need less education, we’ll actually need more of it and we will still need qualifications that evidence it. And incidentally, one of the areas where we will need to educate humans better is about AI itself, so the content of AI itself. So my second question was AI and cheating, or fraud in other words. There is an easy answer to this: no, we cannot allow it. Extended writing coursework is especially vulnerable, as I’ve said many times before. But you know, every time I say that someone somewhere tells me to look at the marvellous things that are happening in universities and how many are supposedly embracing ways of assessment that are more suitable for the AI age. Well, of course, once again the real picture when you peel away the surface is much more complex than that. So for example, just a couple of weeks ago, I noticed that Princeton University, obviously an Ivy League, prestigious Ivy League University in the US, has moved to introduce proctoring – AKA invigilation – in all in-person exams and that reverses a long-standing tradition at Princeton going back 130 years for non-invigilated exams under its ’Honor Code’. And they’ve had to do this because of widespread concerns about AI and academic integrity. On this side of the pond, UCL’s law school last year announced a major initiative to overhaul its exams to make them more AI-proof by making greater use of in-person formats and including reintroducing in-person invigilated exams again, amid concerns about academic insecurity in the AI world. So universities, colleagues, are fast waking up, I would argue, to the harsh reality that if they want their qualifications to continue to be taken seriously, then assessments do need to tell the truth about their students’ achievements. And I could add that that applies as much to unmanaged grade inflation in universities as well as AI-fuelled subversion of assessment. So as we design new 16 to 18-year-old qualifications and reform existing ones, you should expect to see a far, far higher level of scrutiny for any proposals for extended writing coursework because, for the qualifications to be worth more than the paper they’re written on, we simply cannot normalise the idea that AI-generated output is somehow a substitute for genuine human endeavour. And we can’t allow that idea to influence schools and colleges either. Because, while AI is genuinely supremely promising in helping to ease the burden on teachers, improve the quality of the curriculum and improve teaching resources, offloading students’ thinking to AI doesn’t develop the human mind. Ultimately it leads to less learning, less satisfaction, less usefulness in the workplace. Actually, it’s really a covert assault on the realisation of human potential. So, my third and final question on AI, then, is, in a sense, the big one for us. Should we let AI run our qualifications system for us? Surely, some of you might be thinking, it could do a better job than the humans currently in charge, whether they’re organised into exam boards or into government departments. Well, I think there are actually 2 ways of approaching this question. The first is really an argument from accuracy, efficiency and cost. The second is an argument from ethics. Let’s start with accuracy, efficiency and costs. So AI does have enormous potential to improve both efficiency and cut costs, and increase accuracy. And it is doing that in lots of ways already. Lots of upstream processes, question generation, quality assurance, comparing performance across years and students, marshalling summary material to help experts and examiners make their judgements. All of that is happening already. And as far as I can see, properly managed, this is good. But we do also often hear arguments, mostly from the point of view of accuracy, for AI taking over the marking of high-stakes exams. Now, actually AI marking – and I’ll just contrast that with machine marking – machine marking is the automated marking of questions where there’s a very limited range of possible answers that can easily be pre-specified in advance. That’s one thing that’s not really AI. Properly understood, AI marking of assessments is not actually as good as you might think, despite what the many tech enthusiasts, including at this festival, would have you believe. It does still make mistakes, it does still hallucinate, it does still reflect biases. All the current research supports that. More importantly, some AI systems have the capacity to extend beyond the rules initially set by humans, particularly when dealing with cases not anticipated in advance. So such a system could extend and develop a human-agreed mark scheme to deal with cases it encounters that are not covered by the mark scheme. And as soon as it starts doing that, the expert human is out of the loop and no longer in charge. And at that point, colleagues, we confront the ethical question. Qualifications, indeed education as a whole, exists for humans, for human benefits – not actually the other way around. In my view, there’s a very strong argument for saying that the protection of the human person requires that consequential judgements made about another human being’s future require direct human control. And just imagine if an AI bot, an AI marking bot, started to legislate for itself on edge cases, not covered by a mark scheme. Given the influence of qualifications on teacher behaviour, we’ve effectively set up a route whereby AI, outside human control, is effectively influencing decisions about what we teach our students in classrooms. And it’s also the case that using AI marking in this way also makes the right – I would say this is a right from natural justice – to contestability very difficult to exercise. How would I contest a judgement made by a self-evolving AI black box that no one really understood? And if my challenge were upheld by a human, if I did contest it, what does that mean for all the other judgements that are made by the same AI black box? So in short, I think the AI question in high-stakes qualifications goes way beyond efficiency, accuracy, and cost savings – important and desirable though all those things are. It’s fundamentally an ethical question. It’s a trust question. AI will have an important place, but we must remain in charge and we must legislate and regulate to create clarity about what AI can be allowed to do, to bring about benefits for humans, and what it absolutely cannot be allowed to do. And this to keep humans in control, safeguard the dignity of human beings. To do otherwise, I would say, is to hand over the power to select who is worthy, and who is not, without any human bearing responsibility for those judgements – and that, I think, is not acceptable. So to draw to a close then, I’ve covered the main reforms, one of the main reforms underway for students wanting the vocational routes and options in education, post-16. I’ve talked about why that represents a radical break with legacy approaches for these students. We’ve looked at the not-so-simple question, as it turns out, of whether exams should be on-screen or not, and we’ve reflected on three key questions posed by the advance of AI. My conclusion at the end of all of this is that there are fundamental ethical questions that need to be confronted if we’re going to deal with the AI revolution well, and they go beyond cost efficiency and accuracy. We need to make decisions about what is central, both in education and in assessment, and we need to establish standards to ensure that the essential human-centeredness of both is safeguarded. These are, of course, complex debates, and difficult decisions will be required. As we evolve to meet those challenges, we must hold true here to the values that maintain trust and confidence in our national and public asset – in other words our qualification system – be they long-standing or new. So those are my reflections, colleagues. Thank you very much for your attention.
Speech Brief