this research was conducted before the pandemic, in 2019, as part of our regular technical assessment.
Functional skills mathematics teachers estimated how difficult individual questions would be for their students, based on both previous assessments and sample assessments for the reformed qualifications, using a comparative judgment technique. From this, we obtained measures of expected difficulty for all questions.
We found only slight differences in expected difficulty between the sample assessments offered by each awardee that could be considered when setting pass thresholds. We also found that the expected difficulties between the legacy and reformed qualifications are very similar.
More recently, some representative education and training bodies have said they believe the 2019 reforms to functional qualifications have made it more difficult for them. Ofqual’s chief regulator, Dr Jo Saxton, heard the concerns expressed by people in the sector. Ofqual will continue to monitor and assess these qualifications and listen to the views of the sector.
Reformed versions of functional skills qualifications have been available for first training since September 2019. The reform of these qualifications follows decisions taken by the Department for Education to ensure that qualifications better meet employer needs in terms of the knowledge and skills gained by learners, without changing the requirements for qualifications. Some of the major changes to the Maths Functional Skills qualification were that the non-calculator paper focused on core skills related to fundamental mathematical knowledge. These changes aim to place greater emphasis on the skills needed by both learners and employers, and to increase the recognition and credibility of qualifications with employers.
Ofqual regulations require awarding organizations (AOs) to ensure comparability over time and with other AOs’ qualifications. The work reported here was carried out in 2019 as part of our technical evaluation of the assessment materials for these new qualifications. The study aimed to assess the complexity of selective assessment of functional skills between AOs and compared to the legacy functional skills qualifications proposed before the reforms. This was done using a comparative judgment methodology. We used item expert assessment to obtain estimates of the expected difficulty of the mathematical functional skills items from the sample assessments for the reformed qualifications together with items from the old assessments.
We found that the expected difficulty of items in the heritage assessment and the reformed sample assessment was very similar. The high similarity of both the mean (median) and the overall distribution of item difficulty suggests that AOs remained consistent in setting the difficulty of their ratings after the reform. This is encouraging as the reforms aimed to introduce improved new-style content, but not to increase demand, which this research suggests has been achieved.
Considering only the reformed qualifications, each AO’s assessment sets had a clear division of difficulty between Level 1 and Level 2 qualifications. Overall, the assessments were of comparable difficulty across AOs. The results of this study suggest that, using sample assessments as a guide, AOs will be able to administer tests of appropriate difficulty and with appropriate separation between levels. Moderate adjustments to the cutoffs should be sufficient to ensure comparability of math functional skills tests between and within AOs.
Recommend0 recommendationsPublished in