How Universal Design could change assessment: TTR Special

0:00Welcome to TITUS TORK radio! Thank you for listening! Thanks for downloading this episode from TITUS TORK radio. You can find the full schedule and listen back to all our shows at ttradio.org. Enjoy the podcast! Good evening everybody and welcome along to TITUS TORK radio once again. And we're delighted to be here tonight to talk about assessment design, which is actually a topic. I don't think we may have touched on it once or twice before, but this is going to be very much a deep dive into it. So really, really look forward to it. We're going to look at what it actually is, what it means, the barriers and assessment can unintentionally create, I guess, to put it that way, and how those barriers might be removed, whilst maintaining sort of, I guess, fairness and academic rigor and all the things we want in an assessment.

1:00Absolutely delighted to be joined by Louise, Badan, this evening, good evening to you Louise, how are you? No, I'm very well. Thank you. Yes, thank you very much for having me. Excellent, thank you very much for joining us. And also my esteemed co-host, Christopher Vowels, Chris, how are you this evening? Oh well, thanks Tom. Brilliant. Yes, we're going to, let's get straight into it. A lot of our viewers and listeners at TTR, and anybody joining us now, by the way, if you're joining us just now, don't forget to give us a like and share it. And, you know, yeah, join in. But if you are a first time listening to teachers taught radio and you're wondering what on earth is assessment design? Louise, you're going to tell us what is this, what is this thing called universal design and what does it mean when we apply it to assessment? So universal design is a very specific approach to designing inclusive and accessible

2:00assessments. And it's really about embedding accessibility from the very beginning when you're first thinking about what it is you want to test, so what kind of knowledge or skills you're interested in measuring in the assessment. And the idea is that it's sort of a framework really that you can use to design an assessment that is as accessible and usable as possible to the greatest number of students possible. So regardless of any individual characteristic, so that's language or cultural background or gender, a disability, neurodivergence, socioeconomic background and so on. The idea is that universal design can help you, as you mentioned, their introduction. Think about the purpose of what assessment is, what you're trying to measure and then try to identify any potential, unintended barriers that might disadvantage any of those groups of learners and then try to remove them at the earliest point of design. So we don't just want to rely on some reasonable adjustments to accommodations later on the process, but for the very earliest stages when you're thinking of the assessment, trying to remove those barriers from the very outset. Oh sorry, Tom, I can't hear you.

3:03Holidays, why is the thinking behind universal design coming from? Yeah, so that's a good question. The universal design isn't originally from assessment. It's kind of concept that you've used in lots of different areas. It came from construction originally, so when you're thinking about designing a building and trying to make sure that it's as accessible as possible for everyone, so it might be things like putting a ramp at the entrance, so that wheelchair users can more easily access the building. But the idea is that by improving accessibility for one particular group, you improve it for everybody, so a ramp is particularly helpful, obviously wheelchair users, but then it also helps other groups of people be able to access the building more readily. So anybody else with sort of more minimability challenges, for example, or some of you with a Pram or a Prush chair, so it's a similar sort of idea with assessment that by removing those unnecessary barriers for a smaller group, you actually improve the accessibility for everybody and improve the quality assessment overall. So sort of

4:04commonly used examples, if you think about a maths exam, when you're interested in students being able to looking at ratios or something like that. But if you have a really long complicated question that's got lots of complex language in it, maybe the long sentences and is your maths language or something like that, then you start testing something else as well. You're starting to test students' region comprehension and their linguistic ability when actually what you really want to know is can they engage with as mathematical skills. So by making those questions more straightforward, more concise, more clear, then you're helping students in particular who might have some challenges of literacy as students who selects you, for example, but then you'll end up supporting other students who give students who may have, we don't have English sort of as a first or best language, we'd also find more straightforward language easier to access. And actually, you probably help all students because every single student is going to have less reading load and they can more easily get into the questions to your expectations are. The important thing is there, as you

5:08said, you don't change what you're testing, you're not changing the level of demand, you're not making the maths easier, you're just giving them access to be able to show what they can do. So before Universal Design has a, has Universal Design as a concept always existed, or is it something that has recently become more prominent that you've perhaps thought more about? And obviously you work with AQA, so yeah, maybe that's an opportunity just because we didn't really ask you to introduce yourself at the beginning in terms of your role and your background as well. Sure, so yes, I'm head of assessment design principles at AQA and so the role of my team is really to look at of connecting research and practice really is to improve assessment quality. So we look at things like recommendations coming out of research and try to translate that into practical guidelines and principles to support the development of assessments in practice. So that does include things like accessibility. We look at other things as well, like how to design your high quality mark schemes for example, the principal at marking, but accessibility

6:12and inclusion is a really big focus for us. I've actually worked in assessment for a very long time, so, gosh, nearly 17 years now in the different areas, I actually came from assessment research background. And as a Universal Design certainly isn't, it's not a brand new concept, but I think it's definitely getting a lot more attention currently, particularly with the, sort of, the increase in focus on students with their special educational needs and disabilities, for example. And it's just, it's also, like I said, it's really just about good assessment quality. It's something that's been there for a while, but something we're really having an increased focus on. And I think trying to formalise lots of kind of good practice that's been around for a long time and try to really learn more and improve bills upon it, because it's something that you can always make better. You can always learn more about different barriers, different groups of students face. So it was really a lot of work in AQA at the moment to, we've been developing Universal Design principles and embedding and practice and making changes to our papers to make them more inclusive and more accessible by design. Yeah, because I mean, I was going to, going to ask you that, whether universal design,

7:13benefits, particular learner groups, or if it benefits everybody, and what the main aim there is with it. Yes. Yeah, yeah, it's a good question. It's, it's intended to benefit everybody. It, as I said, it sort of makes the assessment quality in better overall, but there will be certain groups that will particularly advantage from having universal design assessments. So, for example, one of the things we look at in a new, the solid design assessments is the format and layouts and the presentation of questions on an exam paper, I try to make them clear and straightforward as possible. And it might be that if you're thinking about a particular piece of knowledge or skills you want to assess, one of the things you might think about then is, or how do you, how do you present this paper question on a paper? You can often have quite complex assessment designs. You think of the sort of questions you might have seen in GCSE and A level papers, for example, they often might involve like a source text or graphs or data, even like students who are asked to engage with. And the way you present those, you can do it in as clean as straightforward a way as possible. So, if you want,

8:17for example, to ask students about data analysis skills and you might try to think, okay, maybe they do need to be able to interpret data from a graph that you might want to try to limit the amount of data they need to process. So you might think, okay, but they'll engage with a particular graph, but you might not want to have loads of extra text and tables and visuals alongside that, because that would give them lots of information to have to process all at once. And that would be, particularly challenging for certain groups of students. So, students with attention difficulties, for example, might particularly struggles try to process so much information all at once, but actually in the stressful context of an exam, all students are going to struggle with that additional cognitive load, the amount of information they have to keep in their mind at any one time. So, whilst particular students might benefit more, it will actually help everybody, making sure it's really clear and focused on what the assessment is intended to measure. Makes perfect sense. And you know, you mentioned this sort of renewed focus on it in light of things like the white paper and other things I'm guessing, but yeah, I mean, I guess

9:24the positive results of this, or I mean, our title for today's show is, how universal design could change assessment. So, I guess, as we work through this conversation, one of the key things might be how this is moving things forward and changing things from perhaps how they used to be or how they were in the past, because maybe it's certain times in the past, questions and exam papers have not necessarily been designed in that way. I don't know if you'd agree with that, but yeah. Yeah, no, I think that's a fair point. I think as I said, there's certainly increased focus on it and a standing assessment could practice, does change your time. I think this sort of core values sort of always been there. So, things we always think about an assessment that's been sort of the crux of it, or things like litterity, so which is really about assessing the right thing, making sure that the assessment measures the things intended to measure, really, so it comes down to. But I think there is a degree of subjectivity involved, there's always more research in the

10:27dairy that can improve things. So, I think historically, for example, there's lots of discussion around how do you make an assessment authentic, which is really important as well, because it needs to reflect for students in powdery, real life later on. There's also how you balance that authenticity with the accessibility, because if you give a huge amount of context, for example, to a question to try to situate it in real world context, then that could be really helpful for lots of students to make it authentic and realistic, but it also adds additional information. So, I think we're perhaps in general, this has an interesting thinking bit more carefully, about when is it appropriate to do that and when is it, when does it add these extra barriers in. So, and I think there's definitely more and more research coming out to support different groups of learners as well, and we're learning more about barriers that particularly in his face. So, there's more research emerging around barriers that neurodivergent learners, for example, face assessment that perhaps there's less of focus on before, because that's the feeling of again, there's increased understanding of in general. Chris, I think you've got a question.

11:32Yeah, hello Louise. I wonder if you might tell us what barriers assessment can unintentionally create for students and why universal design matters in this respect? Yes, thank you. There's lots of different barriers that can always be introduced unintentionally, of course, because it's now something you deliberately put into an assessment, but there can be quite simple changes we can make to an assessment that can really help to minimise these barriers. And often it can be these small changes that add up incrementally to improve the experience of the students overall. A simple example might be, so I mentioned like source-picked source text, for example, something you might see in student types of assessments, like in a history paper or one for a language exam or an English exam. And as students might be presented with a text, they have to read and respond to for different reasons. And some of the small details that actually

12:32creates barriers for some students might be to do with the size of the font or the fonts we use, or how the text is laid out. So things like having a really long dense paragraphs, where there's actually just lots of information to process all at once, might be one area where it could be quite challenging for lots of students. So we would always choose, we have, we tend to use aerial font, the San Serif font in our text, which is something that we know is more accessible for students with a sexier. We try to break up the text where we can so that it's not too dense, so that it can be easily processed by students. But even simple changes, that one of our principles is always to make sure we left lying all of our source text. And if you think about just that our natural reading patterns when you're reading English is a left-to-right language, that's just much more straightforward in terms of visual tracking as you're reading the text. And similarly, we have our line numbering on the left-hand side of the text, so it's naturally at the start of you, but your line of sight as you're reading the text, you're following up that same path. And these are small changes that actually, they don't affect the content at all,

13:33that the text would be the same, but just by presenting it in that way, it makes it more straightforward for students. Another quite simple example is having, or they have one single column or multiple columns of text, so like in an easy paratical, for example, if you have two columns, then the text is broken up with smaller lines, you're more likely to have hyphenated words, there's just lots of more having to move your eyes through it, you're going through multiple lines, but also for STEM students who have sort of visual processing challenges, for example, or attention difficulty again, they can see that second column in their line of sight as they're reading it, so it's a destruction, it's kind of pulling their attention away. So again, it's that small detail that actually, just by putting it in one column rather than two, can make it more acceptable for lots of students to make it easier for them to process that information as they're reading it before they respond to the questions. Fantastic, and in those subjects, then, with their looking at lengthy chunks of text, so English literature and language and history spring to mind, what kind of

14:34processes do you go through to ensure that, say in English or a history exam, isn't testing students on general knowledge rather than the content of the subject as it appears in your specifications? Yeah, so that's a really good question, and that's absolutely never consideration is thinking about, sort of like cultural knowledge, for example, that there's, there's been tests on the assessment, should be, should always reflect what's the specification, the teaching and learning that's led up to that, not bringing an extra kind of knowledge into that. So there's lots of different ways we do it. So, as I said, we have these universal design principles, we have all this guidance that shared with everybody who works with the assessments, which both internally with at our staff, but also, you know, building training to support all of our assessment writers on that as well. And there's lots of different checks and processes throughout the whole whole paper production process. I think, I think one of the things that surprised me most, and I saw a two-book in assessment was just finding out how long it takes to write an exam paper, you know, every exam paper, it tends to be at 12 to 18 months of work, because it goes into it, it goes through so many checks, so many reviews, it's seen by subject matter experts and assessment

15:35specialists and cultural sensitivity specialists, proofreaders, type-setters, it goes through so many different checks, and there's always kind of principles that embed it, so that these are exactly the kinds of things that people are thinking about as they go and through it. So we have subject matter experts who will know the specifications really well, and they'll be able to comment on, is this actually, you know, does this reflect the kind of learning that's happening in the classroom? We've got assessment specialists who'll make sure we only testing it in the right way, we've got type-setters who will be responsible for doing the formatting and layout and making sure that matches the house style, so there's a really actually a very rigorous process that goes into it, which certainly surprised me when I came into this world. Yeah, I think some, one of those particular checks you mentioned there is a new one to me, which is the idea of cultural sensitivity checking. How does that work? That's an interesting idea that I don't expect many of our colleagues would know about. Yeah, I think it's something that it's, or this is what it wouldn't just be one check, it does exist as a check, but I think it's something that we would hope, lots of the sorts of different members involved in that process are aware of, so as I said, this is

16:39all embedded into our unit design principles, so we have a whole section in there about not just cultural knowledge, but any kind of knowledge that may be available to certain groups of students, not others, so cultural knowledge is certainly one of them, so we give examples, part of our guidelines on the sorts of things to think about, so and often it might be thinking about our assumptions of what is general knowledge versus what is actually quite cultural-specific knowledge, so trying to think of an example, so for example, it might be regional variations of language, so I'm a Wales for example, where the word kuch is very commonly used to remain cuddle or hug, but it's probably not commonly used in other areas of the country, so we would think of things like regional variation, variations of language, and either you would not use that particular term or you add a footnote to make sure that it was accessible to everybody, but you also think about cultural knowledge in terms of what students expect to know as part of the specification, so for example, if they're doing an English literature exam and they study the math, and that's obviously, they would be familiar with Shakespeare without particular purpose, but you wouldn't

17:39want to have a reference to Shakespeare in a history exam if students weren't studying that's part of the course, because that would be, yeah, that sort of cultural knowledge that certain students use maybe, maybe advantage some students over others, so these are all the kinds of things that people think about, thinking of the diverse linguistic and cultural context that students might bring with them to the assessment and what they can reasonably expect to know, and for that particular subject. Thanks Louis, so is there a learner need or experience then that has particularly stayed with you as you've moved through this series of assessment posts? Yeah, that's a really good question, and I think that there are probably many. I think the ones that tend to stay with me are the ones that have any kind of, she doing Pat Tom's students emotional well-being, I think, to their self-esteem, sense itself worth and identity and that sort of thing, because I think it's really important for us always to be aware of the human behind, that the numbers, especially when we're working in assessment and we're a bit further away, further moved from the classroom, but I saw one example, there are many I think that I think

18:42want to bring us to mind is one from our student advisory group in AQA, we have a wonderful group of students who have applied to be involved with AQA and they share their feedback and experiences, and we shared all of our universal side principles with them and asked them for their thoughts on it, and it's quite a diverse group as well, and so one of the students who commented on complex language, particularly long sentences, and it happened to be a student who has visual impairments, they have reader to help them in an exam, so we'll read up the questions to them, and they commented it's not just about the language that some students might see on a page and struggle to access, but actually when a student has that kind of access arrangement, they have to listen and process that information in a different way, and they commented that if you know, the question's really long, and they're trying to listen to it and process the information to the speech they're hearing, they'll just switch off and disengage after a while, so it's just completely overwhelming to have all that information provided or really to them, so I think that really stuck struck me as well, that again, it's another group of students that are affected by

19:42complex language in a way I hadn't really considered before, and the fact that that's just that simple design choice, they're having a really long complex sentence, but rather breaking it down to chance could lead to students disengage, you feel overwhelmed straight away, just because the volume of it, I think that really struck me. That all makes perfect sense, Louise, and I want to ask this phrase, sort of fair assessment, you'll see this phrase a lot actually, on social media, etc. when people are talking about assessment, this idea of fairness in assessment, now there'll be some people out there, guests will say, all right, are we making things easier for students, unnecessarily easy, but there'll be other people saying, probably the majority saying, why not do this, because it's better for everybody, and it's not, it's almost that idea of testing on the question rather than the answer, do you understand the question

20:45rather than understanding the answer if you see what I mean? But there'll be some out there who may be a bit more sort of, how would we classify fair? What does that mean to you personally, when someone says, what is fair assessment? Yeah, it's a good question. I think it's a deceptively challenging one, actually, people have written huge papers and books about exactly what fairness is an assessment, but for me, I think it's really important what you flagged there about the, making assessment fairer and more accessible is not about making an assessment easier. It's about giving access to students, and for me, I think the important thing is, avoiding any possible risk of systematic bias being introduced against any student, 20 groups of students because of their particular characteristics, I think for me, that's what's for me, that's the priority in universal design and how fair as it's approached in that way. It doesn't make the content easier, we're testing the same skills, the same knowledge, all it's doing is getting rid of the sort of extraneous noise around it, I suppose, finding these barriers that are not relevant to what the thing we're interested in assessing.

21:47So, you know, the examples I mentioned earlier on, like about the history case study, if we're presenting the text in a single column and using decent sized font and using aerial font and afterlining it, that doesn't change the content at all. We're still interested in the skills at historical investigation, and for that, that's exactly the same for the students point of view, we're just given them the best possible opportunities to demonstrate those skills and that there's knowledge and knowledge. Perfect. Before I hand back over to Chris, just thanks to everybody who's been leaving comments, after the show, we'll try and get to as many as we can and I'm sure Louise will answer them on LinkedIn and so forth. And thanks to everybody as well, these commented are over on YouTube as well. If you are watching and enjoying the conversation, don't forget to drop a like on it and do share it on if you feel it could be useful for somebody. Chris. Fantastic Louise, it's been really interesting to hear what you've said so far about fairness and it sounds to me like you're going a long way to reducing unfairness and exams which may take us closer step by step to the

22:47ultimately fair exam, if ever, such a thing can exist. You've said a little bit already about your universal design practice, but would you be able to talk us through a bit more how these principles work and which specific principles are trying to address which specific problems that you see as currently existing with assessment practice? Yeah, absolutely. So the principles are really sort of this, we have a set of standards I suppose that are sort of overarching approach and then there's a whole load of detail beneath them. So I'll just go through the overarching standards kind of the ideals. So essentially it comes down to first of all asking the question of what is the thing that we're trying to assess. So you're really like clearly defining what it is we're interested in assessing in a particular assessment. Once you know what that is it's much easier to identify the barriers they can get in the way of it. So then once we clear about the purpose of the assessment then we are interested in sort of a little bit about what we talked about already

23:48with the cultural sensitivity of you were thinking about having fair, accessible and biased tasks or thinking about all the different groups of students who've been how-trying to try and make sure that we're not introducing anything that would be reduced by as he is anyone of those groups. The next thing we think about is clarity of instructions making sure that it's really clear straightforward language that we could clearly signpost throughout the assessment what students need to do the kind of format that the others need to take if they need to answer pride more than one example of something making sure that's really really clear and laid out for them. And similarly we want to make sure that we use really clear language in the question themselves. And again this is a bit specific detail as we thought all things are things like use the active voice rather than the passive voice use concise sentences don't use figurative language and don't use a dramatic language always kind of stuff because underneath it. And as well as the clarity of the language we also want to think of the clarity of the visual elements. So you know as I sort of

24:48touched on a new one we do often have to use visual features of assessment whether that's graphs or tables or actually your photographs or images or whatever it might be or maps in geography. So when we have to include visual images the first question is do we need it at all? Is it necessary for assessment because as soon as you have a visual element then that creates a potential barrier for a student with visual impairment for example but if it is needed if it's a valid part of the assessment it's necessary for that subject and how do you present it in a way that is as clear and accessible as possible. So this might include things like thinking about color contrast. So students with color vision deficiency for example you know some color blindness you sphere referred to will struggles differentiate between certain colors like red and green together for example in the not being see the difference. So if we're going to have a graph with bars in it rather than having differentiation by color we might think about texture having black and white but then lines on some some lines some bars and dots on others they can do so any student differentiate between them. And then our final one really is around thinking ahead to those reasonable

25:53adjustments later on and so can we design anything we design in the assessment can we make sure that it is as amenable as possible to reasonable adjustments. So knowing that as we give we provide assessments that are delivered on such a large scale and taken by thousands of students every year there will be students who will need braille versions for example in exam paper. So how do we lay them out in a way that can be adapted as easily as possible into a braille version of paper or a large print version of a paper. And again often it is thinking about how you lay out those those actual elements and those tables and those graphs but like and clearly laying them out so that when it comes to being adapted into another version we can try to ensure that as far as possible a student who's taking an adapted version of a paper will have as comparable experience as possible to students taking the standard version of the paper. So that's kind of the collection of standards. But again as I said there's a lot of detail beneath all of them. Teachers has a student ever asked you something about AI and you've wondered how to answer.

26:55With experience AI you don't need to be a computer science teacher to help your learners understand AI. Are free award-winning resources help you explore how AI works where it's used and how to use it safely and responsibly whatever subject you teach. No prior background in computer science is needed because AI literacy belongs in every classroom. Explore Experience AI today at rpf.io-slash-experient-ai-piphant-teacher-talks. What if one lesson could help students build the confidence, creativity and problem-solving skills they need for the future. Samsung Sol for tomorrow does exactly that. It's a free award-winning innovation programme and competition opened all 11 to 18 year olds across the UK and Ireland. It can run as one lesson or six. No expertise needed. Plus we're talking a chance to win the latest Samsung tech prizes, mentoring and work experience for students. And that's not all. Winning ideas

27:59will be showcased on the iconic Piccadilly circus screens in London. 82% of teachers say it builds essential skills for life and careers. Entries close 18th January 2027. Register now at solfortomorrowuk.com. Teachers have always played a vital role in shaping AQA qualifications. As assessment reform approaches the questions being asked about the future of education matter more than ever and so do the voices helping to answer them. That's why AQA has launched the AQA Teacher Insight programme. Over the next 12 months the programme will bring together teachers, views and experiences through research, engagement and AQA's partnership with Teacher Tap, building a comprehensive picture of classroom experience to help inform the future of qualifications, assessment and education. Search AQA Assessment reform and have your say today.

29:00This show is brought to you in partnership with Hashtit Learning, publishers of professional development books and resources that support great teaching and learning in schools around the world. Level up your professional development today and visit hashtitlearning.com to explore the full range of titles. I'm particularly interested in experience of students who are wholly blind actually and will be using exam papers. How does your system allow for images say on descriptive writing prompts, on language papers or source sections from history papers? What solutions do you find to that very specific kind of problem? Yeah, so we have firms of Braille Specialists who will work with that and as well as Braille offered in terms of the text, they are tactile papers and students will be able to feel the images like I mentioned with the graphs for example, I might be able to feel the difference in the textures. So we work with our Braille providers to get feedback on the kinds of things that they find challenging to produce and kind of and the issues that come up with papers and they'll

30:02feed back to us and again those recommendations will go into our design principles to help to ensure clarity of the layout. But it is things like if you have an image trying to avoid like photographs for example, much more challenging to produce compared to a sketch. Try to avoid 3D having 2D instead because again that's easier to provide in a Braille version. Having an introductory statement to describe what the image is going to be, all these kinds of things add up together to make the Braille adaptations easier to implement later on. Brilliant. I just wondered Louise, how does AQA identify the barriers that need to be removed? I make sure that the changes they're making will genuinely help learners. Yeah, it's a good question. We have a research department who does lots of work in this area. They'll do lots of desk-based research. They'll look at all the kind of academic literature that's out there already and try to collate that and organise that into recommendations for us.

31:02But we also talk to teachers, we talk to students, we ask student advisory group, but we'll research team will go and talk to Sencos and exams officers and then from their experiences and the challenges they face. I was in a school today talking to a head of department there who's overseeing lots of work with students, special education honours and hearing from them what their experiences are in that area. All of that kind of work feeds into developing the recommendations that go into our guidelines of principles. But as I said, it's an evolving area. There's always more work that can be done. And we know that there are certain groups that might be underrepresented in the research. That's also the research team always looking into what we're able to say can build upon that. It also wants to be when we look at new assessments. There'll be things like attesting them out in schools and getting feedback directly from teachers and schools and how those assessments have worked in practice. There's lots of different ways, uni. What are the sort of challenges that teachers in schools are saying to you in terms of

32:04what are they saying to you that they're seeing? Yes, it's lots of things we've come towards about already. So, and that's very complex language and questions is one that comes up quite a lot. That's something that we're really working to focus on. Another one that's come up quite a lot is consistency across subjects, which I think is quite an interesting one because of course students don't just take one subject isolation. They take a whole step to GCRA levels. So one thing that we've looked at is how we present questions consistently, not just within a single subject, but across all of them so that if a student sees the same style of question in biology compared to the one foreign language, for example, it should look forwardly the same as the idea. So, for example, we did a big load of work around multiple choice questions and we found that they were often presented differently so that students say, you're going to question choices and answers and sometimes they might be asked to tick a box, sometimes they're asked to circle right response or near to or write the word out, until there's lots of different ways to do it. So, and that's in the feedback we were getting about

33:04the inconsistency between them. So, that's actually a change we've just made and from next summer, hopefully all multiple choice questions pretty much will be tick a box. And so they're going to just align it so that it'll be more consistent so that students go between their assessments. They'll see the same kind of style across them to be more familiar, more consistent. So, yeah, again, more accessible for them. So, yeah. Yeah, I mean, when we we had a little chat, didn't we? Last week, and we looked at some of the stuff that you've been doing. And one of the things that stood out to me, we don't normally do this, but hey, I mean, it was kind of interesting. So, this is what you sort of shared with me, just this particular example. Do you want to talk me through it? Because I found this really interesting. Sure. Yeah, so this is obviously not a real question. And this is when we mop stuff. We don't we don't have an assessment on potion making. But we just we put an example together of just really bad question that breaks loads of our design prints, show exactly how making calls lots of problems. So, first of all, you can see here that the question

34:08itself is really, really long. It's a really massively long complicated sentence. It doesn't actually tell students what they want to do. It kind of gives them a choice to pick how they want to respond when they want to circle or underline something. There's a picture with red and green, which as I said, is first of all, the picture is not necessarily at all. And then even students who can access it, there's the red and green contrast to be really problematic for lots of students. You've got a table there that doesn't need to be there. That's all, and what it's got after me. That's another really challenging one. Lots of students, by even the guy at the moment, they're actually quite sure that the million is at the bottom of the distance. So, it's going to be an absolute obstacle. And yeah, it's just a lot of students nation to process. There's the table, there's the image, there's the text. Again, there's lots of students who have to have lots of information in their working memory in that moment. And that's a particular challenge for lots of neurodividend students. There's just a lot of cognitive demand for any student who has to process all of that on David's am. And that's the cognitive really bad

35:13question style that we will try to avoid. Because we know that it would add, first of all, in the time constraints environment, it takes students to longer, it's have to access a lot of information. They don't need most of it. So, they're probably worried about how they should try to implement it in the answer. They don't actually need, it's not really relevant. So, adds to their stress and their anxiety on the day. And there are particular groups of students that they would bring you to subvantage. Oh, yes, these, the, I got to mention the, sorry, I'm on my phone, so it's really convenient to have more. And yeah, there's also got some cultural knowledge in there, so reference to Bonfineite, which is something that students who maybe didn't have a British background wouldn't be familiar with. And we've got idiomatic language there, going off the bank, all kinds, all this kind of language that's really complex to access lots of students. So, yeah, that's an example of breaks lots of our principles. Yeah, anyone listening to this, by the way, in the podcast version, go over to YouTube and have a look if you would like at this particular example that we're discussing just so you can see what it sort of looked like. Back to what I wanted to ask you, Louise, and this is in two parts really. Are there any examples of where a

36:18school or a centre have said, look, we want this and you did it. And also, are there any examples where the feedback that's been given to you has ever challenged an assumption or, or completely changed the direction of the work that you've been doing? Yeah, yeah, it's a really good question. So, I guess the, the multiple choice question example, I mentioned everyone, is one that we've done already. So, that's going to be coming out in next summer, that we'll see that change. I think there's other, I think the consistency one as well, we're looking further into bringing that out for other question types, so on the future to have more consistent layout, and sort of more like a templating approach really, so that against that consistency across all the papers, that's definitely something we've worked on. I think the second part of that question is really interesting though, because it is accessibility isn't straightforward. There are cases when you find that you try to improve accessibility for one group, then you might accidentally introduce barriers for a different group, so that things do come as tension and that can be quite

37:18challenging. So, in terms of feedback we've had from centres where it's all challenged expectations, one that springs to mind for me actually is one about, so talk a little bit about visuals, how we try to avoid using visuals wherever possible, because we know they're challenging for many students, visual impairments, extra-executive processing, all that sort of stuff. But then we had a really interesting conversation with some teachers who were from the CSC, you know, teaching language therapists, and they were talking to us in particular about some students, we had language disorders, sort of developmental language disorders, and how actually these are students who have the understanding and the knowledge and skills, but sometimes struggle with language in terms of word retrieval, for example, being able to find the words that help them articulate their particular ideas, and equally in recognising and processing textual language as well. So, they often, one of them main supports is having more multimodal kind of texts and visual things to give them prompts or cues to be able to identify different words,

38:20and they talk a bit about, I command words as example they used actually, so you know, the couple of kind of, that verb you have at the beginning question, tell you what to do, whether it's explain or compare or analyse those kind of command words that are so familiar in exam questions, and they were saying that lots of the students in these language related disorders really benefit from having an icon or a picture, they associate with that word to remind them of what the meaning is, that they know how to do it, so they just need that visual prompt to get a helping access to question. And from our side, that's really challenging, because that's also introduces a different kind of barrier for different group of students, so how do we go about improving on that? So, that's something we're still wrestling with, and thinking about it might be that in the future, with on screen exams, for example, there might be other options, or you can give like hover over options on command words or something like that, but for now, we do have to recognise that there are tensions sometimes, so we have to keep working through it, to keep getting feedback, keep looking at research and testing different ideas out to make sure they're the assessments are as accessible as possible.

39:21I suppose there's also this question then, Louise, about you've mentioned this idea about balance and keeping things in balance, which is an essential fundamental principle really of assessment design, so I wonder whether you might be able to say a little bit more about how you make sure you balance accessibility with demand, so how do you balance accessibility and fairness with this need to make sure that you're still setting rigorous assessments, the test students deep knowledge in a range of different fields? Yes, so that's a really good question, that is one that comes up a lot of accessibility, I think we've touched on it a little bit, I rise you, but I think it's really important to revisit, because it's really important to remember that accessibility is not about changing the level of demand, it's not about making things easier, it's just about removing barriers outside of the, it's essentially a lot of constructs, it's a constructive interest, essentially the knowledge or skills or the attribute, the thing that

40:22we're interested in assessing, and I think it's, again, it's just emphasising that it doesn't change the regga, it doesn't change the level of difficulty, it doesn't change standards at all, it's really just fundamentally about good assessment design, because actually there's barriers shouldn't really be there by design, if we're interested in assessing scientific knowledge, then actually language comprehension isn't part of that, so it shouldn't be overly complicated, but having said that, if we're assessing language, if we're looking at, I don't know, if we're conjugations in a French exam, then that's absolutely relevant, and so we wouldn't want to change the levels of demand of language there, because it's part of what we're assessing, it's the same with English, for example, if we're interested in English literature, in skills and literary analysis, then students do have to be able to engage with figurative language, they have to be able to understand metaphors and similes, and be able to identify them and talk about their effects in the region, all that kind of stuff, so it's not about improving access, but also very much within that subject,

41:23thinking about what is the skill or knowledge in this subject area that we're going to assess, what's the level of demand that is appropriate in this particular context, and then what's outside of that, so what the barriers aren't relevant to this particular knowledge of skills, there aren't relevant to the level of demand we're looking at here, and removing those, so it's absolutely not about changing, lowering standards in any way, it's about just giving all students their access to be able to demonstrate what they can do. Would there be a situation though, I wonder if your assessment design is thinking about this in particular, where you remove some of those barriers, say for instance a very wordy maths question on the maths paper, and the effect of removing the wordiness of that maths question means that the student has more time to think about solving the problem as opposed to sitting there processing the text, how do you deal with that issue, which could potentially arise one would suspect? Yeah, that's a really interesting question actually, and I suppose it would be,

42:26we probably look at it case by case, so as I said, these are the kinds of things that are tested, so we'll be looking at, I think time exams are really interesting when actually, because again, and necessary speediness, I suppose you might call it, is something that isn't, again, generally part of the assessment in that we don't want students to have to be forced to try to answer lots of questions when there isn't sufficient time, so you hope that that situation shouldn't be there to begin with, because in that situation it might be that if an assessment had been designed that with more wordy questions, then when it'd been tested in schools, then the appropriate time limit would have been put on that for that particular design, when you, if you redesign something, and the language is made more straightforward, then it may be that you tested again and it's fine that you don't need as much time, for example, so it's, these decisions aren't made, just completely nice solution, they're always looked at in a border context, they looked out with them subject matter input, and they looked out with testing in schools, and we look at always different elements that the sum of the questions, the question type, the content, the length of time, it takes how many queries and students have and how that all adds up together, so it's not

43:32question, it's not something that's taken in isolation, and it would always be backed by evidence of research. So are there implications then for AQA centres in how they produce, say mock exam papers, as students move towards their GCSE exams and A level exams, should they be taking into account some of the same kinds of issues that you're sharing with us here tonight? Yes, a good question, so we are communicating to schools that we're doing, so the changes so far, I think communication's actually went out last week to centres, and we gave a did update past papers to go to show them what they would look like, so they wouldn't be any surprises in those subjects. The changes are actually relatively small, really, that we've made so far, as I said, it's kind of having that consistent approach to multiple choice questions, having the left in line of text, and all that sort of thing, so it's really about consistency and alignment to making sure it's more approachable in that sense, so there's nothing hugely new, but everything's been communicated to schools, and they've been given those kind of past papers to be able to reference. Any kind of

44:33bigger changes, we would always communicate to schools, we'd always give them the resources, they need to be able to support them, but I think in terms of what's happening in the classroom, what I would hope is it would have a positive impact, because by trying to streamline these things, making instructions clearer, consistent, having the language that I'm big US, and all that sort of thing, the idea is that teachers can really focus on what really matters in the classroom, on teaching and learning, that this should really support good teaching and learning, I think that's what good assessment design should do, it should be, you know, assessment and learning is all two sides of the same coin really, and it should be something that's all integrated, it shouldn't just be an add-on at the end, so it's something that we're always thinking about the impact on teaching and learning, which is why we have those conversations with teachers, and we're connected centers to make sure that, and that, yeah, all those pieces of the puzzle fit together. I mean, just building on what Chris said, as a teacher, you know, as a history teacher, you know, I've created lots of assessments that I still do, and a lot of teachers out there now working with booklets, booklet curriculums, so they're creating assessments within these booklets,

45:34right? What would you say to a teacher now who's designing an assessment before they start, that's a common mistake that you think you, in your work at AQA, would almost see a second nature to universal design and assessment design, and whenever you come up with what you're doing, that is like a common thing that teachers or middle leaders or senior leaders in schools don't consider or make an obvious mistake with every time. Oh, it's a good question, miss if I can. There might be nothing, but. I mean, I think it's a good question, because I think actually all the kind of principles we're applying to high-six exams, they do apply to classroom assessments as well, so it's the same kind of considerations we'd think about when we're designing things here, and it could be really simple things like if there is a question that goes across two pages, making sure that the page is a facing one another, so students don't have to pick flip back and forth on the day of the exam, something like turning over. It can sometimes be these really small details.

46:36Why is that one, why would just, what's the key reason for that one? Well, first of all, some students won't miss it by just having to turn it over, but also just in terms of processing it, you need to see the information side by side to feel to refer to each other, and if you have to turn it over and turn it back again, again, it's relying on working memory and not being able to reference the material in front of you. These little changes actually add up to make quite a big difference, the students experience on the day. You're saying don't skimp on the photocopying. The double-sided exams. That's a coffee, man. Yeah, no, it makes perfect sense, and I think I've probably been guilty of a lot of the lots of the things we've discussed today when it comes to assessment, you know, design and universal design. I've been very, yeah. I don't think, I think you said, I don't, and I don't mean this, I think I don't think teachers necessarily have the headspace, all the time, all the expertise to think about these things when they're designing the assessments

47:40to put the thought into it, which leads really nicely to my question about your process, because you mentioned earlier to Chris about a 12 to 18 month process. So I would love it if you could take us behind the scenes on the 2028 exams, right? I was hoping you could lead me through just a broad stroke, sort of each step or iteration that an exam paper might go through, because I was quite, well, you probably saw my face, like I was quite, I mean, I, you know, I guess back in the day, your job probably wouldn't have existed. I mean, it might have, but it might have just fallen under something else, but I think a heck of a lot of thoughts going into this. So I'm really intrigued just to know, right, 12 to 18 months, what are the steps that are going on? Yes, I was, I will be able to give you a very accurate view of this, because it's my thing, and different team, but I can give you a high level. So I know that my colleagues in assessment management is the team overseas, all of that. And I mean, I'll start by looking at

48:44the performance in the previous year. They'll look at, they'll be lots of lessons learned from the previous exams, and they'll be like a review meeting and trying to get feedback from all of that. And they'll be planning ahead. They'll be looking at the, they'll have planning meetings with sort of, like, to my students associates to make sure that they're planning all the different kind of content areas they want to cover, the exam type question types. They'll be, so they have subject specialists feeding into it, then they'll have reviewers who'll give feedback on it to make sure that they're covering the right kind of areas. They're asked, they've got the right kind of numbers of marks, the right kind of question types, the right kind of layout. They'll be type setters, who are the ones who are specialists and actually formatting and layouting, and how it's sort of specialist training and publishing software to be able to lay it out in the right kind of ways. They'll be proofreaders who will review the contents, make sure that everything reads well. There's a role called a scrutinyer who basically sits the exam as if they were a student, to see if everything works in the same way that see if the time is right, the questions are intuitive, and answerable, and they'll cross-reference that against them, the mark schemes. So there's, yeah, and there's just so many different constant checks and reviews throughout that

49:47whole process. So it's, yeah, I haven't done a justice at all, but so hopefully gives a little bit of a sense of the number of different reviews and content reviews and formatting reviews and layout reviews and language reviews that go on throughout it all. And there's also, I think, feedback from, so for example, inclusion specialists like, that's all which I think is the bridges association with teachers of deaf children who will feedback on like linguistic elements and things like that as well. So there's lots of different, yeah, different people look at it, different stages for different purposes, and that's why it kind of all adds up to this really long process. That's, that's incredible really. I mean, what do you think would, which part do you think would surprise people? I know Chris was surprised earlier with, what was it, Chris? Was it the cultural sensitivities? I can't remember. But you mentioned that. Yeah, that was an interesting interesting angle to say, couldn't it? How does that all work? Yeah, I mean, it's again, it would be somewhat as a specialist for thinking of that, the kind of things we talked about already really, like what's the kind of kind of cultural assumptions might there be that we maybe don't,

50:49we might think of as general knowledge that actually it might be quite a cultural, cultural specific term, it might be a turn of phrase, it might be a particular language choice that might be quite reasonably specific. So, you know, it's, it's interesting. It might pass like, I witnessed it, international assessments for a long time. I found it really fascinating how the same phrases might be understood completely differently in different contexts. So, I know one that springs to mind for me, as I remember there was a text once that referred to somebody sleeping rough. So, everyone in the UK has seen that meant sleeping on the street, but I had colleagues from Australia and America who thought they just meant had a bad night's sleep, they just, they've been trying to name phrases in a completely different way. So, it's these kinds of things that really recognize when it's for our own context. We have to have different perspectives and views on it to be able to see some of those barriers again that we may not be aware of ourselves, which is important to have so many different perspectives and review points within the whole process. I mean, is there anything else you think would surprise people if they knew? You know, is there anything else? The part, any part of this, this process or this work?

51:52I think it's just the, it's the amount of time that goes into it, but I mean, I've literally, I've had sat in meetings before and I've heard debates for like an hour over one single word in a question because they're worried about as students will interpret it in a particular way. Like, I think it's just the amount of thought and care that goes into it is actually really astoundingly see behind the scene. And genuinely, everybody I've worked with the assessment really, really cares about getting it right for students. So, every single element of the paper is a specific design choice, whether it's the word, whether it's the layout, whether it's the, the question type, whether it's, you know, a multiple choice question or a short answer question or an essay question. These things were all thought about really, really carefully and justified. So it's, isn't it? Yeah, I think just the thought and rigor that goes into it, I found quite astonishing assessment. And what, what, what do you think, I mean, you've sort of touched on this, I guess, but what would make you proud of your work in that? Because I can see the passion you've got. I mean, we've spoken several times and I like, I would put your exam, I would put my exam life in your hands, Louise.

52:58I would jump, I would jump off the top depth of the Titanic holding my exam paper that you've produced, I mean, feeling quite satisfied that I got off the ship. Well, thank you. That's about the compliment. Very reassuring. But I mean, what would, what would make you proud about the work you do and the work and the team you work with the AQI and what you do? Yes, absolutely. The team I work with, honestly, it's said, it's just everybody I work with is genuinely, really, really cares about the students. And you know, like AQA says, till spent, never, never let it learn it down. And it's not just a tagline, it's absolutely something people live a youth. And but I think genuinely people come into this area of work because they really believe in it. But I think the other thing is it's just that people do really think about the impact as well. They think about the individual student, but again, there's more and more thinking about the connection between assessment and learning and wanting them to be closer together and work kind of designing assessments and worth teaching to that support, good teaching and learning aren't just under the sun, add on at the end. So yeah, I think that I think it's probably just being proud of

53:58the team I work with. They genuinely do work really, really incredibly hard to try to do that, give learners the best experience possible. And what do you hope teachers and students will feel when they see all this work come together and the paper lands on the desk and the exam room? Not dread. I'm thinking of me and exams now. I mean, you know, within the context of exams, we've all been through it haven't we? I mean, that they're hard, they're stressful. We've all been through GCSEs, nail levels and the rest of it. And we know what it's like. But I think Chris would relish it though. I think just feeling Chris would be like, I can't wait to like dump my knowledge on this page and I've been waiting for weeks, whereas I'm like, this is the worst moment ever. I'm a wrong Chris. I think I might struggle with all the time being going on news days. No, you'd be loving it. Sorry, the reason go ahead.

55:00Yeah, I guess from my side, I, I totally feel reassurance and just to feel a bit just knowing that we're doing the best we can to make sure there is accessible as possible. And I think from students, one of you just knowing not having to think too much about how to navigate the assessment but fit to be intuitive and then to be sort of taken through it naturally without having to stop and think about how to interpret this particular instruction or the rest of it, just really to be able to focus on what's important. So I said to that, when you almost wouldn't notice. The thank you a have a metric for what success would look like from the exam series when these go live. Are you expecting to see particular response issues and students? That's a good question. We have lots of metrics. The majority of the quality of assessment, lots of different ways. We have a whole team dedicated just to that really. So I think from our point of view, we did a huge accessibility audit on our previous exam papers and tried to look at exactly how well we managed to meet all of our criteria. So I think we'd like to repeat that

56:04and be able to hopefully see improvements across the board. So that's something from what we could be doing internally. But then feedback from schools and teachers as well. I would hope that we can want to hear really is that they're seeing what they want to happen in practice, that they're seeing their feedback being realised as far as the ideal. Someone said over on LinkedIn, huge kudos to AQA for having so many different people involved and this is kept in-house because you must be, you know, you're thinking to not to be able to keep for 18 months to be able to keep that completely tight ship and keep it all jammed. I mean, I think that's a fair comment. That is a fair point. So obviously, security is a huge considerate as well with high-six assessments. So it's an impressive operation. So look at looking further ahead. If we get universal design right, what could assessment look

57:07like in say three to five years time? That's a really good question. I think it's, I don't think universal design is one fixed endpoint. I think it's something that we can always improve upon. We can always learn more, right? But I suppose in three to five years, I would hope that we'd start to see, what are you doing this work actually, looking at how universal design applies to on-screen assessments and digital exams? Because that's kind of something that's going to be in the future and we're already working on that and testing it out. And so in on-screen exams, it looks a little bit, lots of the principles are very similar, but you have to think about other things like navigating things on-screen and embed, so there's some really wonderful benefits, obviously. You can embed some of the accessibility features into on-screen exams, like having colour overlays, different colour contrasts, students can zoom in and see the font size and any size they want and that sort of thing. But you'd also have to think about what does by digital literacy skills and what does that mean? Are there any barriers there that might be introduced by mistakes? So it's thinking about universal design in a slightly different context,

58:08but the basic principles I think would remain the same. Spot on. I mean, what do you think still needs to change? I think, literally, again, from a conversation I was having with the teacher today, I think there are certain groups of students that are still underrepresented in the research, as I mentioned earlier on. We've been involved in the past and we know that there are certain groups of new individuals who still tend to be represented in the assessment research literature, so dyscalcula is calcula. I can't remember that's a plan of sense too. And development and coordination disorder, for example, those are groups that tend to be underrepresented. I mentioned about the language disorders early on as well. So I think there's lots more work that can be done in those areas to understand how those particular dinners may face barriers, but also we have to really remember that students aren't a marginal group, even to students who may seemingly same profile on paper. I've completely different lived experiences and may face different barriers. So the other thing we're really, the research is moving more

59:10and more towards, which is hard from a large scale assessment context, is it needs based approach to look at the individual student and what helps them best, which obviously when you're working on large scale and delivering assessments to thousands of students is really challenging. So we have to look at kind of patterns and themes across all of the students. Again, it's a constant feedback loop, but listening to teachers and students and hearing about their experiences so we can improve all of our assessment practices really. It makes sense. We've had hundreds of people sort of tune in and quite a number of people who've stuck with us through from start to now. So thank you very much. If you've enjoyed the conversation, do drop a like on the video. Do you give it a share as well? If you've enjoyed it. And thank you very much for spending a bit of time with us this evening to talk about this. I got one last question for you, Louise. Question is, if there was one thing that you'd like teachers watching or listening to this to remember about universal design and acuease work in this area, what would it be? I think it's really the thread that's been running through all of it really. It's just when it comes

1:00:10to assessment, just think about what is it that you really want to measure, what's the thing that you really want to assess. Anything that gets in the way of that, think of those barriers and try to remove them because that's the best way to ensure as far as possible that we can be as far as possible to all students. Makes perfect sense. Louise, it's been such a pleasure and I've learned actually a lot about the whole process actually with some surprising bits thrown in as well. So been really, really interesting. Chris, I'm sure you share that sentiment as well. Yeah, I think really powerful information is given us here this evening, Louise. An acuease clearly at the forefront of new movements in assessment. What kind of conversations are you in with other examples to make this universal design more universal? Well, I mean, there are certain different connections. There's certain forums where different examples get together. So there's the JCQs during qualification council for examples where

1:01:13they look at access arrangements jointly, for example. But also, lots of it is off-poil, which is the exams regulated to provide accessibility guidance, which is used by all examples as well. So there are certainly different forums and there are different examples also tend to go to similar assessment research conferences with lots of information and research studies are shared. So there are certain ways that these themes are going across to different examples as well, because of course students do experience these exams from different examples as well. I mean, we appreciate that. But yeah, there are definitely ways that information is shared. Brilliant. Louise, it's been great and we've had lots of great comments as well throughout, just saying it's really helpful. So that's great. And I'm sure Louise, you will go to LinkedIn. You will run off after this show to LinkedIn to answer some of the comments that have been made amazing. Thank you very much everybody for tuning in. We have been teachers taught radio at Ion Tom Rogers. That's Christopher Viles and Louise Badden. Thank you very much indeed for your time. And we'll see you all again very, very soon. Cheers everybody and take care and good night.

1:02:18You've been listening to Teachers Talk Radio. Tune in live and listen back at ttradio.org. We look forward to hearing from you next time on Teachers Talk Radio.