Artificial intelligence (AI) has been rapidly advancing in recent years, bringing significant changes to various sectors and industries. As we continue to witness the evolution of AI, it is increasingly important to understand what this technology can and cannot do. The OECD recognises the need to systematically assess the capabilities of AI in relation to human skills, particularly in skill domains of key importance for employment and education. Such an assessment can help policy makers and educators better anticipate the impact of technological change on the workforce and prepare individuals for the demands of the future.

Advances in artificial intelligence (AI) are ushering in a large and rapid technological transformation. Understanding how the capabilities of AI relate to human skills and how they develop over time is crucial for understanding this ongoing process. Knowing what AI can do compared to humans can help predict which skills may become obsolete and which skills may become more significant in the years ahead. This knowledge base can help policy makers reshape education systems in ways that best prepare students for the future and provide opportunities to adult learners to renew their skills.

This chapter introduces the study and situates it in a broader context of related research. The study assesses artificial intelligence (AI) capabilities by collecting expert judgements on whether AI can carry out tests from the OECD’s Survey of Adult Skills of the Programme for International Assessment of Adult Competencies (PIAAC). It follows up an earlier study from 2016 to track changes in AI capabilities with regard to PIAAC over time. The chapter first provides an overview of past studies that assess computer capabilities and their impact on the economy. Against this background, it presents the objectives of this study and discusses potential strengths and weaknesses of the methodological approach the study uses to assess AI. The chapter concludes with an outline of the structure of this report.

This chapter offers an overview of changes in human skills and computer capabilities in the domains of literacy and numeracy over time. It first analyses changes in the skill levels of adults aged 16 to 65, working adults and students aged 15 using data from the Programme for International Student Assessment (PISA), the Survey of Adult Skills (PIAAC), the International Adult Literacy Survey (IALS) and the Adult Literacy and Life Skills Survey (ALL). The chapter then describes recent trends in the fields of natural language processing and mathematical reasoning of artificial intelligence (AI). These technological developments are relevant for the potential performance of AI on the PIAAC test. By showing that technological progress develops much faster than human skills in key skill domains, the chapter highlights the need for periodically and systematically monitoring the evolution of AI capabilities and comparing them to human skills.

This chapter describes the methodology of assessing computers’ capabilities to solve the questions of the Survey of Adult Skills of the Programme for International Assessment of Adult Competencies (PIAAC). It first provides an overview of the PIAAC test, the skills it measures and the test questions used to measure them. The chapter then describes the methods used to select experts, to collect expert judgement, to develop the questionnaire and to construct aggregate measures of artificial intelligence (AI) capabilities in literacy and numeracy. The focus is on the methodological improvements on the assessment approach used in the pilot study. The chapter concludes with a summary of the methodological challenges encountered in the study and the attempts to solve them.

This chapter describes the results of the follow-up assessment of computer capabilities with the Survey of Adult Skills (PIAAC). It first presents the results of the literacy assessment and then the results for numeracy. The chapter studies AI performance by question difficulty by exploring different ways of aggregating experts’ ratings. It then shows the average evaluations of the individual experts and analyses disagreement and uncertainty among them. Subsequently, the chapter provides a comparison of artificial intelligence (AI) and adults’ performance. Finally, the expert discussion of the rating exercise is summarised to illustrate challenges that experts faced in assessing AI with PIAAC.

The chapter analyses changes in assessed literacy and numeracy capabilities in artificial intelligence (AI) between 2016 and 2021. To that end, it compares the majority responses of the expert groups that completed the pilot and the follow-up assessments. In addition, it looks at how the AI evaluations of experts who participated in both studies changed over the period. The chapter also studies the level of experts’ agreement and the prevalence of uncertain answers in both assessments to compare the quality of group ratings obtained in 2016 and 2021. Subsequently, it analyses experts’ projections of how AI capabilities will evolve by 2026 to obtain information on the likely direction of AI progress in the near future.

This chapter summarises the results of the assessment of artificial intelligence (AI) capabilities in literacy and numeracy and discusses their implications for policy. The chapter first considers likely impacts of developing computer capabilities on employment. For this purpose, it analyses the use of literacy and numeracy at work and the proficiency of workers who use these skills on a daily basis. It then discusses the education implications of AI advancements. In particular, it highlights the need for developing skills in the population that are beyond those of AI technology. The chapter also touches upon the importance of supplying workers with diverse skills, including digital skills, to help them cope with occupational changes resulting from the use of technology.