Reviews of National Policies for Education

Rethinking Quality Assurance for Higher Education in Brazil

Please cite this publication as:
OECD (2018), Rethinking Quality Assurance for Higher Education in Brazil, Reviews of National Policies for Education, OECD Publishing, Paris.
https://doi.org/10.1787/9789264309050-en

Metadata, Legal and Rights

ISBN: 978-92-64-30904-3 (print) - 978-92-64-30905-0 (pdf) - 978-92-64-31305-7 (HTML) - 978-92-64-31304-0 (epub)

DOI: https://doi.org/10.1787/9789264309050-en

Reviews of National Policies for Education

ISSN: 1563-4914 (print) - 1990-0198 (online)

This work is published under the responsibility of the Secretary-General of the OECD. The opinions expressed and arguments employed herein do not necessarily reflect the official views of OECD member countries.

This document, as well as any data and any map included herein, are without prejudice to the status of or sovereignty over any territory, to the delimitation of international frontiers and boundaries and to the name of any territory, city or area.

The statistical data for Israel are supplied by and under the responsibility of the relevant Israeli authorities. The use of such data by the OECD is without prejudice to the status of the Golan Heights, East Jerusalem and Israeli settlements in the West Bank under the terms of international law.

Photo credits: Cover © eabff/Shutterstock.com.

Corrigenda to OECD publications may be found on line at: www.oecd.org/about/publishing/corrigenda.htm.

© OECD 2018

You can copy, download or print OECD content for your own use, and you can include excerpts from OECD publications, databases and multimedia products in your own documents, presentations, blogs, websites and teaching materials, provided that suitable acknowledgement of OECD as source and copyright owner is given. All requests for public or commercial use and translation rights should be submitted to rights@oecd.org. Requests for permission to photocopy portions of this material for public or commercial use shall be addressed directly to the Copyright Clearance Center (CCC) at info@copyright.com or the Centre français d’exploitation du droit de copie (CFC) at contact@cfcopies.com.

Foreword

Brazil has well-established systems to assess, monitor and assure the quality of learning and teaching in private and public higher education providers in the federal higher education system. Participation in the external quality assurance system is mandatory for private and federal public higher education institutions, which together account for over 90% of undergraduate enrolment in Brazil. The procedures used to assess institutions and undergraduate programmes are wide-ranging, encompassing institutional self-evaluation, peer review visits, programme-level quality indicators and large-scale student performance assessment. A separate, comprehensive system of external peer review is used to assure the quality of academic postgraduate education throughout the country.

At the undergraduate level, the National System of Higher Education Evaluation (SINAES) and related regulatory and supervisory systems have been operating, with some modifications, since 2004. SINAES has become a fixture of the Brazilian higher education landscape, but has, itself, never been subject to a systematic assessment of its relevance, effectiveness and efficiency. The system for evaluating academic postgraduate programmes, run by the Foundation for the Coordination of Improvement of Higher Education Personnel (CAPES) has been operating since 1998. It is well respected in the academic community in Brazil, but as postgraduate education continues to expand, there are questions about the sustainability of its intensive peer-review processes.

Against this backdrop, the Ministry of Education (MEC), the National Commission for Evaluation of Higher Education (CONAES) and CAPES asked the OECD to undertake an independent review of federal quality assurance policies for higher education. The terms of reference for this review called on the review team, composed of international experts and OECD staff, to assess the relevance, effectiveness and efficiency of the procedures in place. The team was asked to consider the effectiveness of the systems in ensuring minimum quality standards, providing differentiated measurement of quality and promoting improvement of quality and quality-oriented practices in higher education institutions. The team’s findings and recommendations are presented in this report.

I hope this report will support Brazil in its efforts to promote quality in its higher education system through effective systems of quality assurance. The OECD stands ready to help Brazil in this effort.

Andreas Schleicher

Director for Education and Skills and Special Advisor on Education Policy to the Secretary General OECD

Acknowledgements

This review has been undertaken as part of the OECD’s efforts to support countries to enhance the relevance, effective and efficiency of their education policies, coordinated by the Policy Advice and Implementation Division within the Directorate for Education and Skills. The review was coordinated by Thomas Weko (Senior Analyst, OECD), and preparation of the report was coordinated by Simon Roy (Analyst, OECD). The review team consisted of María José Lemaitre (Executive Director, Centro Interuniversitario de Desarrollo, Santiago, Chile); Pedro Teixeira (Associate Professor, University of Porto, and Director of the Centre of Research on Higher Education Policy); Thomas Weko (Senior Analyst, OECD) and Simon Roy (Analyst, OECD). Simon Roy and Thomas Weko authored the report, with the support of OECD consultant Manuela Fitzpatrick, drawing upon input of Pedro Teixeira and María José Lemaitre.

The reivew was initiated by the National Commission for Evaluation of Higher Education (CONAES) and the Ministry of Education (MEC). It benefitted from the support of CONAES members Professor Simon Schwartzman and Professor Renato Hyuda Pedrosa. Professor Elizabeth Balbachevsky, from the University of São Paulo, developed a country background report to inform the review team. Within MEC, the review was supported by the leadership of Maria Helena Guimarães Castro (Vice-Minister), Henrique Sartori de Almeida Prado (Secretary for Regulation and Supervision of Higher Education, SERES) and Maria Inês Fini (President, INEP). Both INEP and MEC provided data and documentation to inform the analysis, the former under the leadership of Mariângela Abrão (Director of Higher Education Evaluation, INEP). The review team is indebted to Ricardo Correa Coelho, who acted as the principle point of contact for the review in MEC and to Leandro Gomes Cardoso and Manoela Vilela Araújo Resende, who provided invaluable logistical support.

The team met with a wide range of government officials and higher education stakeholders during a review visit to Brazil in March 2018, and is grateful for their time and input. They included members of MEC staff responsible for higher education; key staff from INEP; federal, state, and private bodies responsible for supporting postgraduate education and administrators, academic staff, and students from private and public higher education institutions located in Brasília, Recife, and São Paulo (see Annex 2).

Key planning and operational support for the project was provided by Manuela Fitzpatrick (OECD, consultant), and administrative support was provided by Jonathan Wright (OECD, Project Assistant), who also prepared the report for publication. The review team is grateful for the feedback and support received from Andreas Schleicher (Director, Directorate for Education and Skills, OECD) and Paulo Santiago (Division Head, Policy Advice and Implementation Division, EDU, OECD).

While the OECD review team benefited greatly from many discussions with a wide range of Brazilian stakeholders, as well as documents, data, and a country background report provided by MEC, any errors or misinterpretations in this report are the responsibility of the review team.

Abbreviations and Acronyms

APCN

Avaliação de Propostas de Cursos Novos (CAPES Evaluation of Proposals for New Courses)

CAPES

Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (Foundation for the Coordination of Improvement of Higher Education Personnel)

CC

Conceito de Curso (Course Score)

CENEVAL

Centro Nacional de Evaluación para la Educación Superior (National Centre for Higher Education Evaluation, Mexico)

CI

Conceito Institucional (Institutional Score)

CNE

Conselho Nacional de Educação (National Education Council)

CNPq

Conselho Nacional de Desenvolvimento Científico e Tecnológico (National Council for Scientific and Technological Development)

CONAES

Comissão Nacional de Avaliação da Educação Superior (National Commission for Evaluation of Higher Education)

CPA

Comissão Própria de Avaliação (Internal Evaluation Commission)

CPC

Conceito Preliminar de Curso (Preliminary Course Score)

CPF

número do Cadastro de Pessoas Físicas (Population Register Number)

CST

Curso superior de tecnologia (Advanced technology programme)

CTC-ES

Conselho Técnico e Científico da Educação Superior (Technical and Scientific Council for Higher Education)

DAES

Diretoria de Avaliação da Educação Superior (Directorate for Evaluation of Higher Education – part of INEP)

DCN

Diretrizes Curriculares Nacionais (National Curriculum Guidelines)

EGEL

Exámenes Generales de Egreso de la Licenciatura (General Examination for Graduating Bachelor’s Students)

ENADE

Exame Nacional de Desempenho de Estudantes (National Examination of Student Performance)

ENEM

Exame Nacional do Ensino Médio (National Examination of Upper Secondary Education)

HEI

Higher Education Institution

INSAES

Instituto Nacional de Supervisão e Avaliação da Educação Superior (National Institute for Supervision and Evaluation of Higher Education – proposed, but never created)

ISCED

International Standard Classification of Education

IDD

Indicador de Diferença entre os Desempenhos Observado e Esperado (Indicator of difference between observed and expected performance)

IGC

Índice Geral de Cursos (General Course Index)

INEP

Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira (Anísio Teixeira National Institute for Educational Studies and Research)

MBA

Master of Business Administration

MEC

Ministério da Educação (Ministry of Education)

NDE

Núcleo Docente Estruturante (Structuring Teaching Core: group of teaching staff responsible for designing and maintaining programmes)

OAB

Ordem dos Advogados do Brasil (Brazilian Bar Association)

OECD

Organisation for Economic Cooperation and Development

PDI

Plano de Desenvolvimento Institucional (Institutional Development Plan)

PhD

Philosophiae doctor (Doctor of Philosophy)

PNE

Plano Nacional de Educação (National Education Plan)

PNPC

Programa Nacional de Posgrados de Calidad (National Programme for Quality Postgraduate education, Mexico)

PPC

Projeto Pedagógico do Curso (Programme Pedagogical Project)

SERES

Secretaria de Regulação e Supervisão da Educação Superior (Secretariat for Regulation and Supervision of Higher Education)

SESu

Secretaria de Educação Superior (Secretariat for Higher Education)

SETEC

Secretaria de Educação Profissional e Tecnológica (Secretariat for Professional and Technological Education)

SINAES

Sistema Nacional de Avaliação da Educação Superior (National System of Higher Education Evaluation)

SNPG

Sistema Nacional de Pós-graduação (National System of Postgraduate Education)

Executive Summary

Brazil has well-established systems to assess, monitor and assure the quality of learning and teaching in private and public higher education providers in the federal higher education system. The Ministry of Education (MEC) makes regulatory decisions about accreditation of institutions and authorisation and recognition of undergraduate programmes based on the results of evaluations coordinated by the Anísio Teixeira National Institute for Educational Studies and Research (INEP). The Foundation for the Coordination of Improvement of Higher Education Personnel (CAPES) implements a separate system of quality assurance for academic postgraduate programmes.

The external quality processes for higher education institutions (HEIs) and undergraduate programmes are mandatory and apply to private and federal public institutions. These account for 90% of the over 2 400 HEIs in Brazil and enrol 91% of undergraduate students in the country. Three quarters of undergraduate enrolment in Brazil is in the private sector. The remaining 9% of enrolment is in public state and municipal institutions, which are subject to regulation and quality assurance by state governments. CAPES evaluation applies to all academic postgraduate education in the country.

This review assesses the relevance, effectiveness and efficiency of the quality assurance procedures in place. The OECD review team was asked to consider the effectiveness and efficiency of the systems in ensuring minimum quality standards, providing differentiated measurement of quality and promoting improvement of quality and quality-oriented practices in HEIs, providing recommendations for improvement.

Regulating “market entry”: ensuring providers of higher education meet high standards, while streamlining quality assurance procedures

In contrast to some other countries in the Latin America and Caribbean region, compliance with Brazil’s system of accreditation for private higher education institutions and official recognition of undergraduate programmes is nearly universal. The requirements of institutional accreditation are sufficiently rigorous to limit fraudulent or grossly unqualified private institutions from entering the higher education marketplace. The formal requirement for all new undergraduate programmes to obtain official recognition in the early stages of their operation provides a basic guarantee of quality.

Nevertheless, Brazilian authorities could improve the relevance and effectiveness of the regulatory and evaluation processes that govern the “market entry” of new private higher education providers and undergraduate programmes. First, current quality assurance systems and frameworks can be adapted to make them more effective. In the short-term, measures should include creating a user-friendly online platform to provide students and families with reliable information on the accreditation status and quality of higher education programmes; developing more sophisticated indicators to assess the quality of distance education and monitor its expansion; improving the selection and preparation of peer review commissions; and taking greater account of pedagogical processes and initial results in on-site reviews of recently created programmes.

Second, there is scope to target the finite resources available for external evaluation of higher education on institutions and programmes that present the greatest quality risks for students and society. HEIs with demonstrated capacity in internal quality assurance could be permitted to “self-accredit” their own programmes, following rigorous institutional reviews. For institutions that remain subject to programme-level reviews, Brazilian authorities could consider allocating certain tasks to a professional inspectorate, allowing academic peer reviewers to focus on evaluating core aspects of the learning process.

Ongoing quality assurance of undergraduate programmes: improving measurement of quality and better targeting of resources

Each year, students graduating from undergraduate programmes registered in particular disciplines take a mandatory competency assessment: the National Examination of Student Performance (ENADE). The formal objective of ENADE is to assess students’ acquisition of knowledge and skills specified in National Curriculum Guidelines (DCN) and their understanding of unspecified broader societal themes. These objectives are too broad for a test with the scope of ENADE and the general knowledge component of the tests is unrelated to the content of the programmes it is supposed to evaluate.

There are also weaknesses in the way ENADE is designed and implemented, which hinder its ability to generate reliable information on student performance and programme quality. It is a low stakes exam, which reduces the motivation of students; test items are not standardised, meaning tests are not of equivalent difficulty between years and subjects and there are no explicit quality thresholds to indicate what good performance is. Results for students on each programme are standardised to generate a score on a scale of one to five, but is a relative measure of average student performance, not a clear indication the level of their knowledge and skills. Brazilian authorities should undertake a thorough assessment of the objectives, costs and benefits of large-scale student testing to identify how weaknesses can be addressed and how, in contrast to the current situation, ENADE could be made into a useful tool and feedback mechanism for teachers and institutions.

The results of ENADE feed into a composite indicator of quality for each programme: the Preliminary Course Score (CPC). This also includes scores for the profile of the teaching staff, student feedback and an indicator of assumed learning gain (IDD). The IDD is based on a number of bold assumptions about the influence of programmes on student performance, which make it hard to justify its weight (35%) in the CPC. INEP should move to monitoring programme performance using an “indicator dashboard”, with a broader range of disaggregated indicators, including measures of student drop-out and, ultimately, graduate employment outcomes.

Site visits for established undergraduate programmes, which are currently used only when programmes perform poorly in relation to the CPC, use a review template and scoring system that do not focus on identifying the causes of poor observed performance and do not consider graduation rates and graduate destinations. Site visits should be retargeted to focus on the root causes of poor performance highlighted by indicators, while peer reviewers should also visit good programmes to help them understand the factors that affect good performance.

Quality in postgraduate programmes: fine-tuning existing practice and planning for the future

The system of external quality assurance for academic postgraduate education in Brazil sets a comparatively high bar for academic postgraduate training to enter the system. However, there is scope for the peer review committees that evaluate new programme proposals to focus more on the relevance of programmes to expanding knowledge fields and on the design of the training provided to postgraduate students. The four-year periodic reviews involve resource-intensive review of staff outputs, alongside consideration of other factors, but neglect training conditions, student output and graduate destinations. CAPES should rebalance the evaluation criteria to focus more on student outputs and outcomes. The reliance on peer review will make the system harder to scale as postgraduate education expands, while inbreeding creates risks for objectivity and quality. It is important to involve international peers in assessment of the top-rated programmes and plan for the future, notably though evaluating the role of academic Master’s programmes and the real costs of the current peer review processes.

Quality higher education institutions: increasing focus on internal quality assurance and the role of external institutional reviews

Although private and federal public institutions are subject to periodic re-accreditation, with institutional reviews, “de-accreditation” is rare for private institutions and effectively impossible for public institutions. The General Course Index (IGC) – another composite indicator used by INEP - provides limited signals about institutional quality. On-site re-accreditation reviews pay limited attention to evidence of institutional performance, internal quality processes and their practical implementation. Brazilian authorities should reduce the period of accreditation for universities and university centres (currently eight or ten years) and, in more robust institutional reviews, increase the focus on outputs, outcomes and internal quality assurance procedures. A greater focus on these issues would allow Brazil to move to a system where institutions with demonstrably strong internal quality assurance capacity and a proven record of delivering quality can accredit (authorise and recognise) their own programmes.

Governance of the quality assurance system: ensuring greater transparency, improved steering and engagement of sector organisations

The basic legitimacy of external quality assurance in the federal higher education system is not questioned and INEP has developed significant experience and capacity in evaluation. However, challenges in system governance include a potential conflict of interest in MEC, which both steers and regulates federal public institutions. CONAES – the National Commission for Evaluation of Higher Education - lacks resources and the capacity to oversee the quality assurance system, while higher education intermediary organisations have weak capacity in quality-related matters. Drawing inspiration from international examples, Brazil should explore ways to establish an independent quality assurance agency to take on roles currently in MEC and INEP, considering the option of combining a professional inspectorate and academic peer review. CONAES should receive dedicated resources and sector associations in higher education should be incentivised to promote quality across the system. In cooperation with state governments and the higher education sector, federal authorities should explore how a reformed external quality assurance system could also apply to state and municipal institutions.

1.

Assessment and recommendations

This chapter presents on overview of the main findings and recommendations resulting from the OECD review of the external quality assurance system in the federal higher education system in Brazil. The analysis has assessed the relevance, effectiveness and efficiency of the external quality assurance procedures applicable to undergraduate and postgraduate programmes and higher education institutions (HEIs) in the federal higher education system. The chapter summarises findings and recommendations in relation to the different components of the external quality assurance systems: a) procedures to regulate the “market entry” of new HEIs and new undergraduate programmes; b) procedures for the ongoing monitoring and evaluation of existing undergraduate programmes and related feedback and corrective measures; c) external quality procedures governing academic postgraduate programmes; d) ongoing monitoring and evaluation of higher education institutions (institutional evaluation) and; e) governance and administrative bodies and arrangements that have been created to implement and oversee the processes above.

1.1. Focus of this chapter

A review of procedures for external quality assurance in the federal higher education system in Brazil

This chapter presents on overview of the main findings and recommendations resulting from the OECD review of the procedures for external quality assurance in the federal higher education system in Brazil. In line with the terms of reference agreed with the Brazilian authorities at the start of the project, the analysis has assessed the relevance, effectiveness and efficiency of the external quality assurance procedures applicable to undergraduate and postgraduate programmes and higher education institutions (HEIs) in the federal higher education system in Brazil.

A focus on relevance, effectiveness and efficiency

Specially, the terms of reference ask the team to analyse the effectiveness and efficiency of the different aspects of existing quality assurance systems in a) ensuring minimum quality standards in educational provision; b) providing differentiated measurement of quality (between types of provision and levels of quality offered) and; c) promoting improvement of quality and quality-oriented practices in HEIs (quality enhancement). On the basis of the analysis, the OECD review team was invited to provide recommendations for improving the system.

An analysis structured around the main components of the system

In light of the organisation of external quality assurance in the federal higher education system, the review has analysed the different functions of the existing quality system in turn:

1. First, procedures in place to regulate the “market entry” of new HEIs and new undergraduate programmes.

2. Second, the procedures for the ongoing monitoring and evaluation of existing undergraduate programmes and related feedback and corrective measures.

3. Third, the external quality procedures governing the “market entry” and periodic evaluation of academic postgraduate programmes, coordinated by the Foundation for the Coordination of Improvement of Higher Education Personnel (CAPES).

4. Fourth, the ongoing monitoring and evaluation of higher education institutions (institutional evaluation) and related feedback and corrective measures.

5. Finally, the governance and administrative bodies and arrangements that have been created to implement and oversee the above processes.

1.2. External quality assurance in Brazil: evaluation, regulation and supervision

Established systems of external quality assurance for undergraduate programmes, postgraduate provision and higher education institutions

Brazil has well-established systems in place at national level to regulate the operation of public and private higher education providers and assess and monitor the quality of their teaching and learning activities. There are distinct external quality assurance systems for the undergraduate (ISCED 6) and postgraduate (ISCED 7-8) levels of education. Both systems involve the external evaluation of individual higher education programmes, while the evaluation of higher education institutions (HEIs) is coupled with the processes for quality review for undergraduate provision.

Most higher education in Brazil falls under the regulatory responsibility of the federal government and a large proportion of enrolment is in the private sector

The external quality processes for HEIs and undergraduate programmes apply to higher education providers that are legally considered to be part of the “federal higher education system”. This comprises federal public HEIs and all private HEIs in the country. Of the roughly 2 400 HEIs in Brazil, 92% (federal public and private) fall under the regulatory responsibility of the federal government and these institutions, together, account for 91% of undergraduate enrolment. 75% of total undergraduate enrolment in Brazil is in the private sector. The remaining 9% of undergraduate enrolment is in public state and municipal institutions, which are subject to regulation by state governments, not the federal authorities. The system of quality assurance for academic postgraduate programmes, implemented by CAPES, applies to all higher education providers in the country, including state and municipal institutions.

The quality assurance system for higher education institutions and undergraduate programmes involves regulation, evaluation and supervision

For HEIs and undergraduate programmes, the current system of quality assurance was established by legislation adopted in 2004 (Presidência da República, 2004[1]). This establishes a system based on the legal regulation of institutions and undergraduate courses, currently undertaken by the Secretariat for Regulation and Supervision of Higher Education (SERES) within the Ministry of Education (MEC). SERES grants official accreditation to HEIs and authorisation and recognition to individual undergraduate programmes. It does so on the basis of the results of quality evaluations of institutions and programmes, which form part of the National System of Higher Education Evaluation (SINAES) and are coordinated by the Anísio Teixeira National Institute for Educational Studies and Research (INEP), a semi-autonomous agency under the responsibility of MEC. INEP uses a combination of on-site peer reviews, results from the National Examination of Student Performance (ENADE) and programme-level quality indicators to evaluate HEIs and undergraduate programmes. SERES also uses the results of ongoing evaluation and monitoring of HEIs and programmes by INEP to inform its supervision of the federal higher education system and, in some cases, impose remedial measures and sanctions on HEIs that perform poorly.

CAPES evaluates academic postgraduate provision through a system of peer review, the results of which have far-reaching consequences

The evaluation of academic postgraduate programmes (Master’s, Professional Master’s and Doctoral programmes), undertaken by CAPES, involves an elaborate system of peer review, organised by 49 discipline-specific field committees. The field committees approve all new academic postgraduate programmes and undertake evaluations of existing programmes on a four-year cycle. The results of the CAPES evaluations are fundamental for the approval or new academic postgraduate programmes, the continued operation of existing programmes and the allocation of public money for researcher training. Programmes which fail to achieve minimum standards in CAPES evaluations lose funding and the official validity of their qualifications, and are forced to close.

1.3. Regulating market entry: new institutions and undergraduate programmes

Main findings

A system of institutional accreditation and programme-level recognition is used to regulate market entry for HEIs and undergraduate programmes, mostly affecting the private sector

Private higher education institutions are required to obtain formal external institutional accreditation (credenciamento) from MEC to allow them to begin operation and may only initially be established as teaching institutions, with the formal status of “college” (faculdade). Once established, private institutions may transition to the status of “university centre” (larger teaching institutions) or a fully-fledged “university” (institutions with teaching and research activity). This is possible if the institutions meet certain criteria related to number of programmes; the qualification and employment status of staff; and for universities, research; and if they successfully complete a process of re-accreditation. In 2017, MEC processed over 200 requests for institutional accreditation from private institutions.

Public HEIs may be established with any institutional form and are accredited automatically by their acts of establishment. The establishment of new federal public institutions is rare, but, from a legal perspective, such institutions can be created without any requirement to undergo an initial external evaluation.

As a general rule, colleges in the federal higher education system, which are almost exclusively private, are required to obtain formal authorisation from SERES (autorização) to start new undergraduate programmes. As discussed below, colleges with adequate institutional quality ratings may be exempted from the authorisation process in certain circumstances. Institutions with the status of university centre and university have a greater degree of autonomy and are not generally required to obtain authorisation in advance to start new undergraduate programmes, but must notify SERES of the creation of all new programmes. Universities and university centres do require prior authorisation to start new programmes in medical fields and law. In 2017, MEC processed nearly 1 600 requests for authorisation for new classroom-based and distance undergraduate programmes.

All HEIs in the federal system, whatever their legal form, are required to submit new programmes to an external quality assurance process called “recognition” (reconhecimento), once half of total teaching hours for the first student cohort have been completed (in the second or third year, for example). All new programmes offered by HEIs need to complete the recognition process successfully for the degrees they award to be valid nationally in Brazil. In practice, most of the programmes undergoing initial recognition are also in new or expanding private sector institutions. Federal public universities, as well-established institutions, already active in a broad range of study fields, create fewer new programmes and are thus less involved in processes of programme recognition.

Decisions about accreditation, authorisation and recognition are informed by the results of on-site peer evaluations coordinated by INEP

SERES makes decisions regarding the accreditation of institutions, and authorisation and recognition of programmes, based on the results of on-site peer evaluations of the institutions and programmes in question. Two or three external evaluators undertake review visits, using evaluation criteria and scoring detailed in evaluation template (“instruments”) specific to each process.

For the accreditation process, which applies only to new private institutions, the evaluation template is organised around five thematic axes, and assesses the proposed institution against 45 qualitative indicators, each of which is evaluated on a five-point scale. The principal foci are the Institutional Development Plan (30%), planned academic policies (20%), planned management policies (20%) and infrastructure (20%). The final score generated by this evaluation, on a scale of one to five, is referred to as the “institutional score” or Conceito Institucional (CI). Institutions need to score at least three to receive accreditation from SERES. Institutional accreditation for colleges is valid for three to five years, depending on the CI score they receive. After this period, colleges must formally undergo a process of re-accreditation (see discussion of institutional re-accreditation below).

For the processes of programme-level authorisation and recognition, the on-site evaluation instruments used by external reviewers establish nearly identical review templates. The evaluation template for authorisation focuses on planned inputs (teaching staff, infrastructure, etc.) and activities (pedagogical processes, support to students etc.) linked to the programme. There are around 50 qualitative indicators in the template. These are assessed by the external evaluation team appointed by INEP taking into account programme documents, discussions with proposed staff and visits to the facilities planned for the programme. The criteria in the template for recognition focus on the real inputs and activities involved in the new programme, once the first cohort of students has completed half of their study hours.

SERES can impose sanctions on HEIs if recently established programmes receive negative evaluation results in the process of recognition

If the result of the on-site evaluation at the stage of recognition is negative (a score of two or less), SERES requires the HEI to draw up a “Commitment Protocol”, which sets out how the quality problems detected will be addressed within a 12-month timeframe. If it considers there is an immediate risk for students, SERES can legally impose one or more sanctions on the HEI providing the programme, including suspension of the right to recruit new students. This rarely happens in practice. At the end of the period established by the Commitment Protocol, the programme is subject to another on-site inspection by INEP evaluators. If it still fails to meet minimum quality requirements, SERES can launch a “sanctioning procedure”, which may entail the same range of sanctions. For serious cases in private institutions, the relevant legislation allows for the withdrawal of institutional accreditation, which would effectively lead to the closure of the institution. Again, in practice, such cases are rare. Legally, some of the sanctions provided for in the legislation can be applied to public institutions, but the legal status of these institutions as public bodies means they may not have their institutional accreditation withdrawn.

Recent changes have removed the requirement for colleges to seek prior authorisation to start new programmes in specific cases

The authorisation of new programmes proposed by colleges is a risk-adjusted process. Recent changes to the regulatory regime allow colleges to obtain authorisation for new courses under certain circumstances, without undergoing an on-site inspection. Colleges with the minimum institutional quality score (CI) of three can start up to three new programmes a year without on-site reviews, provided they already have officially recognised (i.e. quality assured) programmes in the same disciplinary field. Colleges with institutional quality scores of four and five can create more new programmes a year in fields where they already operate. In these cases, HEIs must still request authorisation, but the procedure is based exclusively on a desk-based analysis by SERES of the programme documents submitted by the HEI.

The procedures for institutional accreditation have been effective in ensuring compliance with basic standards and not hindered expansion of the system

In contrast to some other countries in the Latin America and Caribbean region, compliance with Brazil’s system of institutional accreditation appears to be nearly universal. Private institutions do not frequently operate without institutional accreditation. The requirements of institutional accreditation appear to be sufficiently rigorous to limit fraudulent or grossly unqualified private institutions from entering the higher education marketplace. Moreover, accreditation requirements do not appear to have created excessive barriers to the market entry of private higher education providers. Brazil’s higher education system has grown swiftly over the last decade, and private sector institutions have provided the majority of new study places.

Some cases of fraud do exist, while information on the accreditation status of institutions is not as accessible and transparent as it could be

Nonetheless, there are examples of accredited higher education institutions offering programmes that are not authorised, and organisations that are not accredited higher education institutions offering fraudulent diplomas. While the Ministry’s e-MEC platform provides a single national registry of accredited institutions and authorised programmes, it is primarily an administrative database. Incidents of allegedly fraudulent provision suggest that not all students have ready access to information that allows them to confirm the validity of the institutions and programmes in which they plan to study. While the layout and functionality of the e-MEC site are not designed to be used by students and their families, the information contained in the system could easily be exploited as part of a more user-friendly information service.

Despite some strengths, there are concerns about the rules governing distance education providers and programmes

Distance education now accounts for almost 20% of total enrolment in Brazil, with over 90% provided by the private sector. Private distance education institutions and the programmes they provide are subject to the same procedures for institutional accreditation and programme-level authorisation and recognition as providers of traditional classroom-based higher education. A limited number of qualitative indicators relating specifically to distance education have been incorporated into the evaluation templates used for accreditation, authorisation and recognition, covering pedagogical approaches, digital technologies and infrastructure. For example, evaluators are called on to consider the capacity of teaching staff and assistant tutors to support and mentor the number of students proposed for each programme (the proposed study places). Brazilian legislation requires distance education programmes to respect the requirements of national curriculum guidelines (DCNs), for fields where these exist, and distance programmes have hitherto mostly been blended programmes, with some face-to-face instruction and assessments, often conducted in decentralised distance education learning centres (referred to as “poles”). Internationally, blended programmes have been shown to be more effective than fully online programmes (Escueta et al., 2017[2]).

Recent legislative changes have made it easier for private higher education providers to establish large numbers of distance education “poles” (up to 250 a year), in multiple locations, without the need for the facilities in each location to be inspected by INEP evaluators. Some stakeholders in Brazil are concerned that this will promote the uncontrolled expansion of distance education, without adequate quality guarantees. Furthermore, the specific evaluation criteria for distance education institutions and programmes used currently are few in number and underdeveloped in light of the risks associated with this kind of provision (limited staff-student interaction, the risk students are isolated, the challenges of organising fair and rigorous assessments and examinations, etc.).

The system for programme-level authorisation and recognition creates additional guarantees of minimum quality standards

The formal requirement for all courses to obtain official recognition in the early stages of their operation provides a basic guarantee of the quality of programmes. The procedures in place force higher education providers to reflect seriously about the design of the programmes they are providing and put in place a range of documents and processes – described in the Programme Pedagogical Project (PPC) – that should contribute positively to the delivery of relevant programmes meeting minimum quality criteria. Nevertheless, the factors verified through the on-site evaluation at the stage of recognition are all conditions for the delivery of quality programmes, rather than indicators of the initial performance of the programmes in question (student progression and performance, for example). Moreover, the processes used to evaluate the quality of new and recently created programmes are subject to various lines of criticism.

There have been concerns about the profile and objectivity of the review commissions undertaking on-site reviews

Representatives of private institutions consulted by the OECD review team complained that the composition of the reviewer pool used to implement on-site reviews is often skewed towards public universities, while institutional representatives more generally argued that those who are called upon to carry out reviews sometimes lack expertise concerning the programme under review. There is also concern about the subjectivity or unreliability of qualitative assessments. The revised process of on-site review for programme authorisation and recognition, as amended in late 2017, asks reviewers to make qualitative judgments on a five-point Likert scale, using pre-formulated judgement criteria. Despite the attempts by INEP to formulate the judgement criteria clearly, these scales still leave considerable room for interpretation. They call upon reviewers to make distinctions that are likely to be inconsistent between individuals. The OECD review team was told by campus officials that the same programme offered in different campuses with otherwise near-identical supply conditions received different marks from on-site reviewers.

The set of indicators used in the evaluation templates and the weight accorded to different topics are not optimal

The on-site evaluation instrument used by external reviewers for the process of programme recognition assigns 40% of its weight to assessment of the teaching staff attached to the programme. This reflects the fact that the staff are working at the time of recognition, so the composition of the teaching workforce can be judged more accurately than during the previous process of authorisation, where this is used. However, the judgement criteria reward the presence of full-time staff with doctoral degrees and attach little value to professional experience, thus disadvantaging professionally oriented programmes. At the same time, relatively little weight is attached to assessment of the pedagogical and didactic approaches implemented by the programme, despite their crucial role in supporting students to acquire relevant learning outcomes.

The on-site evaluation templates now make special provision for the authorisation and recognition of distance education courses. However, 45 out of 55 indicators in the templates are applicable to both classroom-based and distance programmes. The specific indicators of programme quality related to curriculum, instruction, learning support, and assessment in distance programmes are less developed than those used in accreditation systems in other OECD and partner countries, including the United States. Developing appropriate measures of quality that reflect the specific characteristics of distance education is, however, a challenge shared by many higher education systems.

Finally, on-site visits carried out for programme recognition permit higher education institutions to award degrees without providing evidence about the initial performance of the programme, such as rates of attrition among its students in the first years of operation. Additionally, the process of recognition does not systematically elicit information from the students whom the programmes serve, or external stakeholders who have experience of working with the programme and its students, such as public sector employers and private firms which provide internships.

The processes of programme-level authorisation and recognition are administratively burdensome for HEIs and INEP

The OECD review team heard frequent criticisms from institutional representatives of the delay and burden associated with the on-site review process for authorisation and recognition. INEP and SERES argue that the situation has improved in the last two years. In particular, they point to the fact that HEIs that have received adequate quality scores (a CI of three or above) are exempted from on-site reviews at the stage of authorisation for programmes in fields where they already have courses (within certain limits). They argue that the most recent regulatory changes in Decree 9 235/2017 reduce burden for institutions with an established quality record, allowing them to create additional study places more easily, for example.

While there has indeed been a shift in the regulatory approach, the market entry process for new undergraduate programmes in the federal higher education system remains administratively burdensome for HEIs and the evaluation agency (INEP), when compared to equivalent processes in many OECD countries. In Brazil, despite the recent changes, all new programmes are required to go through the recognition process, with on-site reviews that depend on peer review and are logistically complex to organise.

This contrasts with the situation in many OECD and partner countries, where HEIs can create new programmes and issue valid diplomas without prior programme-level authorisation. In these and other systems, authorities also often link quality review procedures more closely to risk of poor quality than is the case in the Brazilian system, with less complex procedures in place for institutions that can demonstrate they present a lower quality risk. Although the large private higher education sector in Brazil creates specific risks, which are not found in all higher education systems, there is certainly scope for Brazil to draw on risk management practice in other quality assurance systems.

Key recommendations

1. Improve the reliability and visibility of information about institutions’ accreditation status to ensure students and families are well informed

Although MEC, with the support of evaluations coordinated by INEP, regulates the entry of new institutions into the Brazilian higher education marketplace more comprehensively than in other systems undergoing rapid expansion, the quality assurance system is not fully effective in preventing fraudulent and unauthorised provision. The first line of defence against unaccredited higher education providers is students themselves. Informed students understand which institutions are accredited and not, and why this matters to them, and are able to identify and avoid unaccredited institutions. In principle, comprehensive information about accredited institutions and recognised programmes is available through the online e-MEC. However, e-MEC is not a user-friendly source of accreditation information. More accessible public Internet resources found in other higher education systems could serve as references for the Brazilian authorities in this regard. In the medium-term, the aim should be to develop a comprehensive online portal providing students and prospective students not only with programme-level information on quality assurance results, but also on issues such as graduation rates and graduate employment outcomes (see discussion on programme indicators below).

2. Over time, increase the focus on institutions as units of evaluation in the external quality assurance system to reduce burden, while maintaining effectiveness

Despite attempts to address concerns about the composition of review commissions and reduce requirements for authorisation in some cases, the Brazilian system of programme review at market entry remains complex and burdensome and may not represent the best use of the country’s resources. Programme-focused regulatory decisions – for new and existing programmes - account for more than 10 000 of the 12 000 acts that SERES handles annually. The Brazilian system of quality assurance currently focuses proportionally more efforts on the programme-level than on the institutional level as a unit of evaluation and monitoring. Permitting HEIs with demonstrated capacity to assume responsibility for the quality of the programmes that they offer and to become “self-accrediting institutions”, following rigorous institutional reviews, could significantly reduce the burden of programme approval through authorisation and recognition. It would also allow attention to be focused on programmes that present greater quality risks in institutions not granted self-accrediting status. Quality guarantees could be maintained across the system by an enhanced system of programme-level monitoring indicators and more rigorous and comprehensive process of institutional re-accreditation.

3. In the near term, take steps to improve the evaluation process for programmes that remain subject to programme-level authorisation and recognition

The OECD review team sees a clear case for maintaining programme-level authorisation and strict market entry requirements at programme level for HEIs that lack a strong track record of good quality provision and are not able to demonstrate adequate capacity to self-accredit their own programmes. It is thus important to increase the effectiveness of these processes in promoting quality practices for institutions that remain subject to programme-level authorisation and/or recognition. Priorities for improving current practice in the short-term include:

Further improving the criteria used to select and assign peer reviewers for on-site reviews to increase the fit between reviewer expertise and programme review responsibilities.

Continuing and increasing efforts to improve the training of peer reviewers, with a view to improving the reliability and impartiality of scoring.

Increasing the weight attached to the organisation and implementation of teaching and learning in the evaluation instrument for recognition, reflecting the importance of these factors for students.

In cooperation with international peers, refining and expanding the specific indicators used for the evaluation of distance education programmes, so that these address the particular risks associated with this type of provision. This should consider how best to evaluate decentralised distance education centres (“poles”).

Using the recently introduced process of feedback about the performance of peer reviewers to monitor and revise selection and training.

4. In the longer term, take steps to reduce further the burden and improve the effectiveness of quality assurance processes for programmes outside self-accrediting institutions

In the longer term, two issues should be considered in particular. First, the procedures for on-site visits could be fundamentally reformed. Responsibility for reviewing institutional infrastructure and basic institutional policies could be assigned to a well-trained and professionalised inspectorate. The expert judgment of academic peers (who currently review all aspects of institutions and programmes) could then be applied to a more limited set of indicators than at present, focused on core teaching and learning activities. A sequenced process of accreditation and authorisation could be implemented in which a professional inspectorate initially carried out its work, and academic peers would be engaged only for institutions and programmes that have passed a first stage of review. Second, it will be important to identify ways in which the more extensive, quantitative, and comparable information about intermediate programme performance can be incorporated into the process of programme recognition. Examples include student attrition from programmes, and student feedback concerning the teaching and learning environment.

1.4. Assuring and promoting quality for existing undergraduate programmes

Main findings

Student testing, programme indicators and on-site reviews all play a part in the ongoing quality assurance of undergraduate programmes

Once undergraduate programmes have been recognised, they are subject to an ongoing cycle of evaluation coordinated by INEP on behalf of the Ministry of Education. As currently designed, this cycle involves the collection and collation, by INEP, of programme-level data, including the results of the programme’s students in a national assessment of learning outcomes, to create a composite indicator of programme performance every three years. As a general rule, in cases where programmes score low ratings in relation to this composite indicator, a new on-site programme review is undertaken by external evaluators. The results of a programme in relation to the composite indicator and, where used, the on-site review determine whether or not SERES renews its official recognition, thus guaranteeing that the diplomas awarded by the programme retain national validity.

ENADE is a set of tests used to measure the performance of students in undergraduate programmes

Each year, students graduating from undergraduate programmes registered in a particular set of disciplinary fields are required to take a mandatory competence assessment – the National Examination of Student Performance (ENADE). Disciplines are assigned to three broad groups, with disciplines in group I evaluated one year, group II the year after and group III the year after that, meaning each discipline is subject to ENADE every three years. The ENADE tests contain a general competence assessment, common to exams in all fields in a single year, and a discipline-specific component. There are currently separate ENADE tests for 88 study fields, with discipline-specific questions for each test developed by multiple academics selected by INEP. In addition, all students participating in ENADE are required to complete a student feedback questionnaire providing biographical information and a personal assessment of their programme. The average results obtained by students in each programme are used to calculate an “ENADE score” on a five-point scale for that programme. These scores are published and also feed into the composite indicator of programme quality discussed below.

The objectives established for ENADE by the legislator are unrealistic

The objectives of ENADE, as currently formulated, are unrealistic. The legislation establishing SINAES requires ENADE to measure students’ performance in relation to the content of relevant national curriculum guidelines, their ability to analyse new information and their wider understanding of themes outside the scope of their programme. The requirement to measure understanding of unspecified “themes outside the scope of the programme” – which has given rise to the general competence assessment in ENADE - is inherently problematic because it is so general and the knowledge and skills assessed, by definition, are not part of the programme’s core intended learning outcomes. It is thus unclear how those running programmes could be expected to equip students with such a range of unspecified knowledge and skills or why they should be held accountable for students’ not having these competences at the end of their studies.

As ENADE is a written examination with a restricted duration, it is also impossible to measure the full range of learning outcomes that any adequately formulated curriculum guidelines should contain. Moreover, by implying that ENADE sets out to measure students’ learning outcomes in relation to the National Curriculum Guidelines for undergraduate programmes, there is a risk that the content of ENADE (in a given year or over several years) comes to be seen to define what is important in the National Curriculum Guidelines. If ENADE is to be maintained, Brazil’s legislators and quality assurance authorities need to provide a more credible account of what it can realistically achieve and how risks for innovation and responsiveness can be mitigated.

There are significant weaknesses in the way ENADE is currently designed and implemented, which undermine its ability to generate reliable information on student performance and programme quality

The OECD review team considers that there are at least five principal weaknesses in the way ENADE is currently designed and implemented:

1. The first problem relates to the participation of students and their motivation to make an effort in the test. A proportion of the students who should be taking the test each year are not doing so. Across years, between 10-15% of students registered to take the test each year do not turn up on the day. Moreover, there are concerns among stakeholders in Brazil that some HEIs seek to avoid registering a proportion of students for ENADE. At the same time, ENADE is a high stakes exam for HEIs (as it is used in the quality rating of their programmes), but a low stakes exam for students. Although attendance is compulsory, ENADE scores have no effect on students’ academic record and there is evidence that a significant proportion of students do not complete large parts of the test. Evidence from other OECD and partner countries suggests that if the results of tests have no real consequences for students, this impacts negatively on student motivation and performance. Low student motivation is likely to have negative implications for the validity of ENADE results as an accurate reflection of the learning outcomes of students.

2. A second concern relates to the development, selection and use of test items for each ENADE test. At present, there is no robust methodology to ensure that the difficulty of each test item is taken into account in the composition of the test and thus that a) tests in the same field are of equivalent difficulty between ENADE cycles and b) tests in different fields are of a broadly similar level of complexity. This means that it is not possible to compare the raw results or the Conceito ENADE between years or between disciplines. A related question is whether the number of discipline-specific items included in ENADE (30) is adequate to generate a reliable indication of students’ learning outcomes from an undergraduate programme. The answer almost certainly depends on what the exam is for. A robustly designed examination with only 30 items may be able to provide a general indication of a students’ level of knowledge and competencies in a specific disciplinary field. However, such a test is unlikely to provide reliable evidence of students’ performance in specific sub-fields or aspects of the curriculum, which limits its usefulness as a tool to help teaching staff and institutions improve the design of their programmes.

3. A third problem is that no explicit quality thresholds or expected minimum levels of performance are set for ENADE tests. Without tests of a comparable standard of difficulty and without defined quality thresholds (pass, good, excellent, etc.), ENADE scores are simply numbers, reported according to their position in relation to all other scores in the same test. It is thus impossible to know if students in programmes that achieve 50% or 60% in ENADE are performing well or poorly.

4. A fourth problem relates specifically to the design of the general competencies (formação geral) component of ENADE. This is currently composed of general knowledge questions regarding current affairs and social issues, including two questions that call for short discursive answers. However, unless all undergraduate programmes have knowledge of current affairs and social issues as explicit intended learning outcomes – which is not the case – it is unreasonable to judge individual programmes on students’ performance in these areas.

5. A final issue is that the standardisation of ENADE scores compounds the lack of transparency about what ENADE results really mean. Raw marks are attributed to a five-point scale based on the standard distribution of scores in a single subject in a given year. As tests may vary in difficulty and students obtain very different distributions of scores, where a programme falls on a standard distribution of the scores for all programmes says little about the actual quality of the programme in question.

These reliability issues and the limited use of ENADE results by HEIs for quality improvement call into question the resources dedicated to the exam

The results of ENADE are used by INEP and SERES for regulatory purposes, as discussed below. However, institutions consulted by the OECD review team report that they did not use of ENADE results in efforts to improve the design and content of programmes. Representatives of institutions consistently indicated that they did not see ENADE as providing useful feedback to help them improve their programmes. Although the OECD review team does not have access to a detailed breakdown of the costs of implementing ENADE, these account for a substantial part of INEP’s budget for evaluation of higher education, which amounted to over 118 million reals (USD 30.7 million) in 2017. It is questionable whether the quality and usefulness of the results achieved with the exam as currently configured justify the investment of public resources committed.

The Preliminary Course Score (CPC) is used as a composite indicator of programme quality

To monitor programme performance, INEP currently uses a set of indicators comprising a) measures of student performance and assumed learning gain (based on ENADE test results); b) the profile of the teaching staff associated with the programme and; c) feedback from students about teaching and learning, infrastructure and other factors from the questionnaires they complete in advance of taking the ENADE test. When new ENADE results are available for each programme, after each three-year cycle of testing, INEP calculates a programme score – the Preliminary Course Score (CPC). Programmes that score below three out of five on the CPC are systematically subject to on-site inspections by external review commissions, with a positive evaluation score (a CC of three or above) a pre-requisite for renewal of their official recognition. Courses that score three or above on the CPC generally have their programme recognition renewed automatically by SERES, without having to undergo an on-site inspection.

The CPC is an unreliable measure of quality, lacking transparency

The CPC attributes 35% of its total weight to an “Indicator of difference between observed and expected performance” or IDD. This is calculated by comparing each student’s actual results in ENADE with the performance that would be expected given their previous performance in the national high school leaving exam, ENEM. The combination of the boldness of the underlying assumptions about the predictive value of ENEM results for the performance of undergraduate students; the poor reliability of ENADE results in the first place; and the potential influence of factors outside the control of the programme on student performance mean that the IDD provides only limited information on programme quality.

Moreover, it is widely accepted that the weightings attributed to the different indicators in the CPC are arbitrary, with no discernible scientific basis (TCU, 2018[3]). This further compounds the lack of transparency about what the scores attributed to courses really mean in practice for students, families and society at large. It is positive that the CPC sets out to include indicators of the teaching process (through the imperfect proxy of teaching staff status); qualitative feedback from students (the main beneficiaries of the system) and measures of student learning outcomes. It does not, however, contain a measure of the attrition rate of students (what proportion of students entering a programme complete it) or the subsequent employment outcomes of students.

The principle of using indicators to identify “at risk” programmes and target finite resources for on-site inspections makes sense, especially in a system as large as Brazil’s. However, the CPC does not provide a reliable mechanism to identify poorly performing courses. The absence of quality thresholds in ENADE and the standardisation processes used to create the ENADE score, combined with the weaknesses of the IDD, mean it is far from clear whether a CPC score of three represents an adequate standard of quality or not. A reform of the monitoring indicators used and the way they are combined is necessary.

Site visits are undertaken for programmes that perform poorly on the CPC

When programmes are identified through the CPC as performing poorly – often meaning they have poor relative performance in ENADE – they are subject to an on-site inspection by external evaluators, coordinated by INEP. The evaluators assess the supply conditions for the programme using the same evaluation instrument that was already used for programme recognition (reconhecimento). The results of the new on-site inspections are used by SERES as a basis for decisions for the renewal programmes’ official recognition. The evaluation attributes a new quality score – an updated Conceito de Curso (CC) – that effectively replaces the CC attributed at the time of initial recognition and exists alongside the CPC score in the e-MEC system.

These site visits use a review template and scoring system that do not focus on identifying the causes of poor performance in the CPC and do not consider graduation rates and graduate destinations

The on-site visits for renewal of recognition, as currently organised, (re)check compliance with basic standards that was already checked through the initial on-site visit for the recognition of the programme. The evaluation instrument for recognition and renewal of recognition places a 40% weighting on the category “teaching staff” and 30% on “infrastructure”, with just 30% attributed to teaching and learning policies and practices. The indicators and judgement criteria relating to teaching staff mostly focus on the qualifications and experience of the individuals in question, with only three indicators dealing with the activities (atuação) of staff or their interaction with each other.

As such, the renewal of recognition on-site reviews do not focus strongly on the teaching and learning-related factors that might be expected to have greatest influence on student performance and quality. Frequently, it appears that programmes which score poorly on the CPC measure subsequently achieve a higher score on the CC (TCU, 2018[3]). As such, these programmes nominally recover the higher quality score. It is understandable that the CPC and the inspections leading to the CC can generate different values, as they measure almost entirely different things. A greater focus on teaching activities and the greater attention to outputs and outcomes (attrition rates, learning outcomes, graduation rates and employment outcomes) would make the evaluation instrument more effective in identifying the real causes of poor performance.

More generally, the objective of targeting on-site inspections on weakly performing programmes has advantages, as the systematic use of periodic on-site inspections for all existing programmes in Brazil would almost certainly be unfeasible for logistical and financial reasons. However, it also means programme-level site visits at this stage in the evaluative process always have a punitive character and that peer reviewers are not exposed to good practice in well-established programmes, which could inform their judgements about, and recommendations to, poorly performing programmes.

Key recommendations

1. Undertake a thorough assessment of the objectives, costs and benefits of large-scale student testing as part of the quality assurance system

Officially ENADE currently seeks to assess students’ acquisition of knowledge and skills specified in the relevant National Curriculum Guidelines (DCN) or the equivalent documents for Advanced Technology Programmes, as well as their understanding of unspecified “themes outside the specific scope” of their programme. This is an unrealistic objective and no standardised test could achieve this. Moreover, as discussed in the preceding analysis, the current design and implementation of the ENADE tests are characterised by significant weaknesses. At present, ENADE results are used extensively as a basis for regulatory decisions (renewal of programme recognition), but are not used by institutions and teachers to identify areas where they programmes need to be strengthened.

The OECD team believes that, in its current form, ENADE does not represent an effective use of public resources. As such, as a basis for decisions on the future of the system, a thorough reflection is needed about the objectives of large-scale student testing in Brazilian higher education and the costs and benefits of different approaches to implementing it. The main questions to answer are:

1. Can an improved version of ENADE, addressing the current design and implementation weaknesses noted in this report, be implemented and generate reliable information about the quality of undergraduate programmes?

2. Could the information about the quality of programmes generated by a revised ENADE be provided by other, potentially more readily available, indicators? What is the specific and unique added value of ENADE results?

3. If a revised version of ENADE does indeed have the potential to generate valuable information that cannot be obtained from other sources, does the value of this information justify the costs of implementing ENADE? How can the costs of implementation be minimised, while still allowing ENADE to generate reliable and useful results?

The OECD team believes two factors should be considered in particular. First, for ENADE to have the greatest possible added value, it needs to be able to provide reliable information that can help teachers and institutions to identify areas of weakness in their programmes (in terms of knowledge coverage or skills development). ENADE results cannot simply be a blunt indicator used to inform the regulatory process, as other indicators, such as graduation rates or employment outcomes could be used for this purpose. Second, the current requirement to apply the ENADE test to all programmes every three years increases the fixed cost of implementing the system. It is important to consider whether sampling techniques could be deployed to reduce costs, while maintaining reliability.

2. If a reformed version of ENADE is retained, ensure the objectives set for the exam are more realistic

If the decision is taken to maintain a revised version of ENADE, it is crucial to ensure the objectives set for it in the relevant legislation and implementing decisions are realistic and clearly formulated. The objective of a reformed ENADE could be to provide:

An indication – rather than a comprehensive picture - of the performance level of students in relation to intended learning outcomes, as one indicator, alongside others, in a comprehensive system of external quality evaluation and;

Data on student performance that can be used directly by teachers and institutions in identifying weaknesses in their programmes as a basis for improvement (quality enhancement).

To achieve these objectives, the test should focus on measuring knowledge and skills that programmes explicitly set out to develop in their students. This means abandoning claims to measure abstract general knowledge with no direct link to the programme and focusing on a) selected discipline-specific knowledge and skills and b) generic competencies that can realistically be developed in an undergraduate programme. The latter category might include critical thinking and problem-solving. These can theoretically be tested for using discipline-specific test items.

3. Improve the design of ENADE tests to ensure they generate more reliable information on learning outcomes that can also be used by teachers and HEIs

If maintained, ENADE tests should be designed in a more rigorous way to ensure that they are of comparable levels of difficulty within subjects from one year to the next and that tests for different disciplines are of equivalent difficulty for equivalent qualifications (bachelor’s, advanced technology programme, etc.). This may require a shift from classic test theory to item-response theory. As part of this process, performance thresholds and grades should be established clearly in advance. The objective should be to provide students and programmes with easily understood and usable grade-point averages and grade distributions. The approaches to both test design and performance thresholds used by CENEVAL in Mexico or testing organisations in the United States might provide valuable inspiration on how a revised form of the ENADE tests could be developed. It is important for INEP to draw on the expertise of other organisations involved in standardised testing internationally in the development of new approaches and test formats, to ensure it benefits from a wide range of expertise.

4. Explore ways to make the results of ENADE matter for students

If maintained, ENADE needs to be made into a higher stakes exam for students, so that they make an effort to actually demonstrate the level of knowledge and skills they possess. Currently, it is difficult to make ENADE results count towards individuals’ degree scores, not only because of institutional autonomy, but because only every third cohort has to take ENADE and including ENADE in degree results may be perceived as unfair to students in a year where the test exists. As a minimum, the ENADE score could be included in the student’s diploma supplement. Alternatively, ENADE could be made into a curriculum component for the years in which, or – in the case of sampling - for the students to whom, it is administered, with the requirement that an equivalent test for students in other years be administered by institutions. It is not yet clear if this would be possible legally.

5. Introduce a new indicator dashboard, with a broader range of measures, to monitor programme performance and identify “at risk” programmes

The use of the Preliminary Course Score (CPC) cannot be justified in its current form for the reasons discussed above. However, systematic programme level data are a crucial tool for monitoring a system as diverse and variable in quality as Brazil’s. The most promising option would be to include a broader set of more transparent indicators in an ongoing monitoring system, with thresholds established to indicate “at risk” performances on different indicators. This information could then be used to inform regulatory decisions and feed into subsequent evaluation steps (such as on-site reviews). The system should apply to all programmes, with data obtained from institutions and other sources, as appropriate, and consolidated in a renewed version of e-MEC.

Such a system could use a more diverse set of indicators of teaching staff, real (not standardised) ENADE results (based on established performance thresholds), an indicator of drop-out rates and, when possible through linking data sources using the national identity number (CPF), information on employment rates and earnings. Indicators of the socio-economic profile of students could be included in the system, with higher tolerances for issues like drop-out or ENADE performance for programmes with intakes from lower socio-economic groups. Such variation in tolerances should be limited, as all students should be expected to reach minimum standards and all programmes maintain a certain proportion of their students. A revised form of the IDD could potentially be maintained alongside the other indicators in the indicator dashboard, provided its status as a proxy for expected performance and its limitations are made clear, and its weight in the overall monitoring system is reduced.

The OECD review team understands that INEP is already planning (October 2018) to “disaggregate” the components of the CPC and complement these with additional indicators to inform the regulatory process. Hopefully, this recommendation will support this process.

6. As part of a new system of institutional accreditation, exclude institutions with demonstrated internal quality assurance capacity from on-site programme reviews for the duration of their accreditation period

There is scope to exempt institutions from systematic ongoing programme-level review that have a track record of good performance and that can demonstrate a high level of internal quality assurance capacity. As discussed below, this would require existing systems for institutional accreditation and re-accreditation to be strengthened. If problems were identified through programme indicators in the indicator dashboard, such institutions would be responsible for addressing these issues internally. Addressing poor quality would become a key focus of institutional review and poor performance or failure to address problems adequately could lead to institutions losing self-accrediting status in the subsequent round of institutional review. This move would further reduce some of the burden of external programme-level reviews for renewal of recognition (as well as the initial recognition process).

7. Maintain programme-level supervision for other institutions, with targeted on-site reviews for poorly performing programmes and randomly selected highly performing programmes.

For the remaining institutions, programme-level review would be maintained. The new programme-level indicator dashboard (which would cover all programmes, including in self-accrediting institutions) would allow poor programmes to be identified and replace the current CPC system. If annually collected data on completion rates and employment outcomes were included in the dashboard, alongside input indicators and periodic results from a reformed ENADE, this would allow more effective continuous monitoring of programmes. Problematic programmes could first be called upon to submit an improvement plan that could be assessed remotely, largely in line with current supervision procedures. SERES, or a future quality assurance agency (see below), could decide on timeframes for improvement and whether and when an on-site visit would be required. It is crucial that SERES, or a successor agency, have the capacity to close poor programmes rapidly if programme indicators fail to improve without clear justification and evaluators give a negative assessment following an on-site inspection.

However, while targeting of resources is important, the risk of evaluators only being exposed to poor quality programmes – and thus lacking good reference points – needs to be addressed. As such, it is recommended that reviewers also take part in reviews of randomly selected programmes that obtain good scores in relation to monitoring indicators – potentially including programmes in “self-accrediting” institutions - to allow them to gain more insights into the range of practices and performance that exists in their field in the country.

8. Develop a separate evaluation instrument for on-site reviews of established programmes

The current process for on-site reviews of established undergraduate programmes uses the same evaluation and judgement criteria as the instrument for programme recognition (which occurs when the first student cohort has completed between half and three-quarters of the programme). This instrument pays insufficient attention to programme outputs and outcomes (notably the results of (a revised) ENADE, attrition and graduation rates and employment outcomes) and to the teaching and student support practices that would be expected to have the greatest influence on these outputs and outcomes. A new instrument should thus be developed for on-site reviews of established programmes, which places most emphasis on these factors. The earlier suggestion for an inspectorate to examine infrastructure and basic institutional policies would mean that site visits by peer reviewers should focus exclusively on the learning environment and possible causes of poor outputs and outcomes.

1.5. Assuring the quality of postgraduate education

Main findings

A dedicated system for the evaluation of academic postgraduate programmes

The system of external quality assurance for academic postgraduate education in Brazil began in its current form in 1998. It evaluates and regulates academic (stricto sensu) Master’s programmes and doctorates. In Brazil, stricto sensu Master’s courses – including so-called “Professional Master’s” – are widely understood as the first stage in an academic or research career – a situation that is largely a reflection of the relatively recent expansion of doctoral education in the country. In many other OECD higher education systems, Master’s programmes, where they exist, are seen primarily as an extension and deepening of undergraduate education, preparing graduates for a wide range of high-skill jobs in the economy.

The CAPES evaluation system includes a specific approval process for new courses (APCN), designed to ensure only academic teams with demonstrated expertise, a proven track record of quality research and adequate facilities are authorised to provide academic postgraduate education. Course proposals are assessed by a field committee composed of academic peers from the field in which the course seeks to operate. Following a standard assessment and validation process, new courses are formally approved if they score at least three on a nominal scale of one to five. Every four years, CAPES implements a comprehensive evaluation of all academic postgraduate programmes that have already been accredited and been in operation sufficiently long for students to have produced academic results. The results of this evaluation – attributed through scores on a scale of one to seven - allow programmes to continue operating or, in case of poor performance, lead to withdrawal of funds and recognition for the diplomas they award. This effectively means programmes that fail the CAPES evaluation are forced to close.

The system for approval of new programmes sets a high bar for entry to the system, but there is scope to review the balance of quality indicators used

The APCN process consciously sets a comparatively high bar for entry into the system of academic postgraduate training and for the creation of doctoral training provision in programmes that already operate at Master’s level. In so doing, it seeks to maintain high minimum standards for postgraduate education, protect students against poor quality provision and ensure efficient targeting of public funding. During the review visits, the OECD team noted a high degree of support for the principle of maintaining a high threshold for entry into the academic postgraduate education system.

The criteria examined in the process for approval of new courses cover a wide range of the variables that might reasonably be expected in an ex-ante assessment of a proposed postgraduate programme. However, the current evaluation system attaches comparatively limited attention to the relevance of new courses to national or regional needs and developing knowledge areas; to the design of the training programme; and to support and personal development opportunities offered to students. Although the coherence of the proposed course with the Institutional Development Plan (PDI) of the host institution is assessed, there is no explicit assessment of the relevance of course to the needs of Brazil, in terms of knowledge development and highly qualified human resources. Similarly, there is little obvious room in the evaluation templates currently used to assess how the training programme will help to develop students’ knowledge and skills and monitor their progress.

The four-year periodic reviews involve resource-intensive review of staff outputs, while neglecting training conditions, student output and graduate destinations

Every four years, the field committees draw on information on staff, students, graduates and details of scientific outputs reported by each postgraduate programme through the online Sucupira platform as a basis for their assessment of each programme. The quality of student publications and the quality of the academic output of staff in academic journals are assessed using a standard classification of publication “vehicles”, recorded in an online database called Qualis. The assessment of books and book chapters is undertaken by physically reviewing a sample of publications for each programme in depth, but represents one of the largest calls of the time of members of some field committees (notably in the humanities, social sciences and some of the hard sciences).

The set of indicators used in the CAPES four-year evaluations covers many of the key variables that would widely be assumed to contribute to high quality postgraduate provision. It is positive that the evaluation grid, under different headings, takes into account factors such as staff-to-student ratios, time to graduation and cooperation networks with external research and non-academic organisations, for example.

However, the most striking feature of the four-year reviews is the strong focus on the scientific output of the academic staff involved in the programmes being evaluated. The CAPES evaluation is – nominally at least – an evaluation of postgraduate training programmes, not a research performance evaluation. As such, it is questionable why the system does not allocate less weight and fewer resources to assessing the performance of staff and more to assessing the performance of students and outcomes of graduates. Although there is also some attempt in the current CAPES system to assess the destinations of graduates from programmes, this aspect of programme performance is not currently addressed adequately.

The reliance on peer review will make the system harder to scale as postgraduate education expands, while inbreeding creates risks for objectivity and quality

Despite the strengths of the current division of responsibilities within the CAPES evaluation system, the evaluation system relies heavily on the voluntary contribution of academic staff organised in discipline-specific field committees. Although academics involved in the CAPES evaluation process consulted by the OECD review team felt the time and effort required of them for the current system for approval of new programmes remained reasonable, they highlighted that the CAPES system as a whole is becoming unmanageable for field committees, as the number of postgraduate programmes increases.

The Review team understands that no assessment of the value of the time dedicated to evaluation of courses by academic staff in the field committees – and thus also the cost to their home institutions - is currently available. Given the comparatively rapid rate of expansion of postgraduate provision in Brazil in recent years and the related increase in the number of proposals for new courses, it will be important to develop a better understanding of the number of person-hours used in the evaluation process and the associated costs.

The reliance on disciplinary committees composed exclusively of Brazilian academics also risks creating an excessively narrow academic focus in evaluations. While scientific excellence and traditional measures of academic output remain the basis for postgraduate education, it is important to complement this with perspectives from outside academia, to ensure that the development of postgraduate education responds to broader national and regional needs. Moreover, the current process for the evaluation of new courses involves limited or no direct interaction between those proposing the new courses and those evaluating the proposals.

A second key issue with the staffing of CAPES evaluation processes is the risk of endogamy (inbreeding). Even in a country size of Brazil – particularly given the relatively small size of its postgraduate training system – the number of established academics in a given field of study is limited. The number working in very high-quality departments and programmes at an international level is even smaller. As such, there is the risk that the people making judgements on whether or not a given programme is of international standard have close connections with the programmes they are judging. Moreover, the comparatively small pool of evaluators and their background may lead the evaluation process to reward programmes that reproduce existing models of education, rather than innovate.

Key recommendations

1. Adjust the weighting of evaluation criteria in assessment of new courses to focus more on relevance, training and continuous improvement

The OECD review team considers the current evaluation process for new courses could be improved by adopting the following modifications:

Revise the structure of the evaluation fiche for new courses to create a more transparent structure that follows the intervention logic for postgraduate training programmes, moving from inputs to outputs with a clearly explained rationale for each indicator used.

Include a separate section in the evaluation fiche on the relevance of the programme to national development needs, taking into consideration the development of new scientific areas and the knowledge and skills required for the further development of the private and public sectors in the country, including in natural sciences, social sciences and the arts.

Increase the weight attached in the evaluation of new courses to the training dimension of programmes and support provided to students, with an assessment of the likely capacity of the programme to equip students with relevant research and transversal skills.

Include a more explicit requirement for a programme development plan for all new programmes approved, setting out specific and measurable goals over time.

2. Bring additional perspectives into the evaluation of new programmes

To bring a broader range of perspectives to the process and potentially promote innovation and inter-disciplinary cooperation, CAPES should involve one or more academics from other academic fields in the field committees undertaking the assessment of new courses. In addition, to bring in expertise and perspectives from outside the academic community, CAPES should consider appointing specialists in economic development and the evolution of skills and knowledge requirements, as well as representatives of the private economy and the wider public sector to the Scientific and Technical Council (CTC-ES). If implemented effectively, this could ensure that final decisions on programme approval take into account broader national needs and developments.

3. Maintain programme-level accreditation in the medium-term, but consider the long-term desirability of transitioning to institutional self-accreditation for established institutions and programmes

Brazil’s postgraduate education system has grown rapidly in recent years and might still be considered to be in a phase of consolidation, when compared to postgraduate education systems in many other OECD and partner countries. In the medium term, it therefore makes sense to maintain course-level accreditation, to maintain oversight of the continued development of the system and ensure the promotion of quality. In the longer term, it could be possible to move to a system of institutional self-accreditation linked to a strengthened model of institutional accreditation. This would allow universities to start academic postgraduate programmes if they met certain criteria in terms of staff and profile and had been judged to have strong institutional quality systems in an institutional quality review (see below). The provision of publicly funded scholarships and additional programme funding should certainly remain dependent on positive external evaluation of the programme, in line with practice in many OECD systems.

4. Clarify the objectives of periodic evaluations and rebalance the focus of evaluation criteria to include greater focus on student outputs and outcomes

The periodic (four-year) evaluations of post-graduate programmes currently devote disproportionate attention and resources to assessing the outputs of academic staff. CAPES evaluations should focus on assessing the conditions for and performance of postgraduate training, not the research output of academic departments. The OECD review team therefore recommends increasing the weight attributed to educational processes, student outputs and employment outcomes, and reducing the weight attributed to staff outputs. This would make it possible to reduce the time and resources allocated to assessment of staff output, by assessing only a limited sample of research output. The Qualis system for journal rankings should also be reviewed, to introduce more uniformity in the classification of journals between knowledge fields. CAPES should also consider whether it is feasibly systematically to include interviews with course and programme coordinators as part of the periodic assessment of courses and programmes, to gain addition insights into the operation and performance of the programme and answer questions arising from documentary evidence.

5. Ensure those judging whether programmes are of international standing really have an international perspective

Given Brazil’s aspiration to develop a world-class postgraduate training system, it would be valuable to gain an international perspective on the programmes judged nationally to be among the best in the country. The OECD review team therefore recommends that CAPES systematically involve non-Brazilian academics in the assessment of programmes pre-selected by field committees as candidates for being programmes of international quality or excellence. In light of the number of programmes involved, it is likely to be most feasible to concentrate this international involvement on programmes proposed for the top score of seven. It may be possible to organise international peer review committees, who are able to review synthesised information about the programmes under review in English or Spanish, and potentially conduct group interviews remotely or in person with programme coordinators.

6. Undertake evaluations of specific components of the CAPES system and aspects of academic postgraduate provision as inputs to future policy

The OECD review team identified two specific issues where further information and analysis appears to be required in order to plan future policy for academic postgraduate education in Brazil, and its external quality assurance:

First, the full costs associated with the current system of external peer review are a “black box”. Peer review is inherently time-consuming and therefore expensive. The time academic staff spend involved in peer review is time they are not dedicating to their core activities of teaching, research and engagement with society. In order to help plan the future development of the system of peer review, CAPES should undertake an assessment of the cost of the time used by members of the field committees in the evaluation process, including the unit cost per programme evaluation.

Second, there is a wider question relating to the future of academic (stricto sensu) Master’s programmes. It would be valuable to undertake a systematic evaluation of the role of Master’s education in Brazil, including a specific focus on the profile and effectiveness of the professional Master’s programmes created in recent years. This evaluation should consider, in particular, the destinations of previous graduates from these programmes and the views of the academic community and private and public sector employers on the relevance and future role for Master’s level education in Brazil.

1.6. Assuring the quality of higher education institutions

Main findings

HEIs are also subject to monitoring and periodic re-accreditation

Legally, both private and federal public institutions are subject to periodic re-accreditation (recredenciamento), based on on-site reviews coordinated by INEP. For private institutions, successful re-accreditation is a prerequisite for their continued operation (although “de-accreditation” is rare). For federal public institutions, the process is essentially no more than a formality, as they cannot have their accreditation removed. The period for which (re-)accreditation is valid varies depending on the organisational status of the institution and institutional quality score (CI) already awarded to the institution. Universities and university centres are only re-accredited every eight to ten years, while colleges must be re-accredited at least every five years. In addition, institutions are subject to annual monitoring, based on their average performance of their programmes in relation to SINAES programme-level indicators and the results of CAPES evaluations for stricto sensu postgraduate programmes. The weighted averages of the Preliminary Course Score (CPC), and, where applicable, the scores attributed by CAPES for new and existing postgraduate programmes, are used to produce an overall score for each institution called the “General Course Index” (IGC).

But institutional review plays a far less prominent role than in many other systems of external quality assurance

While the letter of the law governing quality assurance in higher education in Brazil accords a central role to institutional autonomy and self-evaluation, the practical implementation of the SINAES imposes a complex system external programme-level scrutiny on a three-year cycle. For institutions that perform poorly in ENADE and on the CPC, this leads to regular programme-level inspections, using prescriptive processes that limit the room for manoeuvre for institutions. For institutions that tend to perform well in relation to ENADE and the CPC, particularly universities and university centres that are only subject to institutional review every eight to ten years, on-site evaluations by external reviewers are comparatively infrequent occurrences.

There are few incentives for institutions in this position to develop strong internal quality assurance systems that go beyond the minimum requirements imposed by the legislation, or to promote quality enhancement internally on a continual basis. Interviews conducted by the OECD review team in several institutions suggest that Internal Evaluation Commissions (CPAs) focus primarily on ensuring compliance with SINAES rules and delivering data to INEP, rather than developing internal quality systems tailored to institutional needs or promoting innovation and quality improvements. This contrasts with the situation in many European countries and in the United States, where institutional review and evaluation of internal quality procedures form the core of external quality assurance practices.

The General Course Index (IGC) provides limited signals about institutional quality

The IGC score – also calculated on scale of one to five - is used by external bodies and the media in reporting about the quality of higher education in Brazil. The IGC is widely perceived as a visible public signal of institutional quality that that institutions themselves feature in advertising. The real signal value of the IGC as a quality indicator for consumers is limited, however. While IGC scores range, in principle, from one to five, scores of one are virtually unknown, and nearly all scores cluster at values of three and four. In 2016, 93% of universities and 96% of university centres received scores of three or four. Setting aside the validity or reliability of the IGC, it is clear that its discriminating power for non-college institutions is low. Although the reputational effects of IGC can be important, it is not an indicator that is likely to have an impact of how institutions understand and manage the quality of the education that they provide. The IGC does not introduce new performance information for institutional leaders.

On-site re-accreditation reviews do not consider evidence of institutional performance and attach little weight to the quality of internal quality processes and their practical implementation

Like the other on-site review processes (such as recognition), the re-accreditation review process coordinated by INEP (potentially only every ten years) is focused on input and process, rather than outputs or performance, and reviewers are responsible for scoring qualitative indicators on a five-point scale. Given that the process of re-accreditation necessarily focuses on institutions that are already operating, with graduating students and graduates, there is scope to include greater consideration of outputs (graduates and evidence of the learning outcomes) and outcomes (graduate destinations) in the institutional assessments at this stage. The current evaluation instrument for institutional re-accreditation devotes comparatively little attention (in terms the number of indicators and judgement criteria) or weight to assessment of the internal evaluation capacity of institutions.

Owing to the schedules for re-accreditation, the institutional quality score awarded through re-accreditation processes (CI) is not calculated and reported on an annual basis, but rather with a periodicity that may range from three to ten years. In light of its infrequency, and perhaps because it is not linked to student outcomes as observed in ENADE, the CI score appears to function solely as a regulatory input, and not as a public signal of institutional quality.

Key recommendations

1. Reduce the period of re-accreditation for universities and university centres

Universities in some of the best-regarded higher education systems in the world must undergo external institutional reviews every four, five or six years. This is the case in the United Kingdom, the Netherlands and Sweden, for example. The current eight or ten-year accreditation periods for universities and university centres mean these institutions have few incentives to develop robust institutional quality mechanisms and problems in institutional quality management may go undetected for long periods. Instead of the current system, institutions with demonstrated internal quality capacity could be rewarded through dispensation from some or all aspects of programme-level review, subject to successful reaccreditation on a five or six-year cycle (see below).

2. Reduce the weight attached in institutional re-accreditation reviews to input and process indicators that measure basic supply conditions for higher education

There is scope to rebalance the weights attributed to the evaluation indicators used at the stage of institutional re-accreditation, away from inputs and towards processes and outputs. A first aspect of this is to remove indicators relating that measure basic supply conditions for higher education, such as infrastructure and equipment and general management policies. The availability of suitable infrastructure to supply each undergraduate programme is verified through the programme-level recognition and renewal of recognition processes, while some of the most general institutional policies are unlikely to change – or need to change - considerably over time. It is therefore wasteful to devote resources to re-evaluating and re-scoring these kinds of variable through the re-accreditation review. The inclusion of these indicators also reduces the proportional weight attributed to factors that are important to verify in re-accreditation such as educational results and institutional performance.

3. Increase the weight attributed to outputs and outcomes

Evidence about educational results and institutional performance is neglected in the current system of institutional re-accreditation. While processes of accreditation cannot take into account programmatic and institutional performance, re-accreditation can – but does not. Institutions should be able to graduate most students who begin their studies, and they should do so in a timely way. Those who graduate should be able to find employment, preferably in fields related to their area of study – and most certainly so if their studies have a career orientation – whether accounting, civil engineering, or nursing.

Quantitative programme and institutional indicators should ideally focus on the outputs and outcomes of higher education, while on-site reviews conducted by peers would helpfully focus on the inputs and processes that generate the outputs and outcomes observed in indicators. For example, indicators focused on outputs or outcomes, such as graduation rates, would be complemented by an on-site review process that examines the conditions that affect variation in these rates. These conditions include student advice and mentoring processes; how institutions identify students at risk of falling behind or dropping out; and the social or psychological, and academic support services provided to students at risk.

4. Increase incentives for institutions to take a strategic view of quality

The processes of institutional quality assurance do not encourage institutions to take a truly strategic and institution-wide view of quality. The IGC generates a score that is an aggregation of programme-level results. However, it does not generate a score that has been demonstrated to be useful in differentiating different levels of institutional performance or providing actionable feedback to institutions. Institutional Development Plans (PDIs), in their current form do not appear to provide an opportunity for institutions to take a comprehensive and strategic view of their institution, its profile, and quality of its educational programmes. It would be valuable to provide incentives to institutions to develop more meaningful PDIs with a stronger focus on how quality across a range of dimensions can be maintained and enhanced. One way to do this is to make assessment of internal quality policies and practice a much bigger part of the re-accreditation process, through greater weighting in the relevant evaluation instrument for on-site reviews.

5. Move to a system where institutions that can demonstrate strong internal quality assurance capacity and a proven record of delivering quality can accredit (authorise and recognise) their own programmes

Finally, processes for demonstrating institutional quality do not permit higher education institution to demonstrate that they have the capacity to take care of quality, and should be authorised to act as self-accrediting organisations, and should be permitted to create, revise, and eliminate programmes on their own initiative – as happens in other higher education systems in the world. The process of re-accreditation – specifically, the resulting CI score – changes the periodicity of institutional reviews, but it does not alter the level of responsibility that institutions are permitted to exercise. If account for institutional quality is to be joined up to institutional responsibility for the quality of programmes, it will need to be a very different and more robust process than at present. Examples of such differentiated models – where some institutions are subject to programme-level review and others are accorded self-accrediting status on the basis of rigorous institutional review - exist in other systems of higher education and could serve as inspiration for Brazil.

1.7. Governance of external quality assurance

Main findings

The current governance landscape for quality assurance, involving SERES, CONAES, INEP and CAPES has some strengths

There are important strengths to the governance and implementation of quality assurance in Brazil. For example, INEP is recognised internationally as a leading public agency for educational assessment. Its wide experience with large-scale assessment and its capacity to manage data collection systems provides the nation’s higher education quality assurance system with a high level of competence. CONAES has succeeded in attracting experts to its council, and through them has been able to mobilise higher education research from across the nation to inform the further development of SINAES.

The basic legitimacy of external quality assurance is not questioned and the system has developed significant experience and capacity in evaluation

The basic legitimacy and integrity of the quality assurance system is widely accepted across the higher education system, by public and private institutions alike, and by representatives of academic staff and the administrators and owners of higher education institutions. In the course of its implementation, SINAES has used a range of evaluation techniques – including self-assessment, peer review, and external review grounded in student assessment - that has been widely welcomed. Moreover, some higher education institutions in Brazil now closely monitor the experience of their students and their readiness to participate in external assessments. Others are making efforts to use compulsory self-assessment and peer review processes as opportunities for improvement, and to engage broadly their university community in the assurance of quality.

…but the current system of governance faces three main challenges

There are three fundamental challenges facing the institutions of quality assurance that merit attention and improvement.

1. First, the design of quality assurance institutions creates conflicting responsibilities for the Ministry of Education. MEC establishes, funds, and steers the federal university system, through its Secretariat for Higher Education (SESu). At the same time, it is responsible, through SERES and, indirectly, INEP, for evaluating their performance and for regulatory actions concerning the programmes they offer. These conflicting responsibilities lead the nation’s higher education institutions, especially its private institutions, to view the Ministry as a champion of one sector, rather than a neutral arbiter among all.

2. Second, while CONAES is responsible for providing guidance and feedback on the functioning of SINAES, it is not properly resourced and organised to do so. CONAES does not have its own professional staff or a dedicated budget, and lacks the capacity to undertake the sort of detailed and sustained analytical work that is needed to evaluate how SINAES is working. Instead, it depends upon the input of implementing bodies whose work it is to supervise and guide, most especially INEP. This dependence is exacerbated by the participation of the implementing bodies on the council itself. It lacks sufficiently wide input – from professional bodies, employer associations, and other centres of government - to take into account the broader social responsibilities of higher education.

3. Finally, in most higher education systems, responsibility for promoting and sharing quality improvement practices lies with bodies outside of government - with associations that represent sub-sectors (such as research, confessional, or polytechnic universities), and with bodies that represent professional groups within higher education institutions, including institutional research, curriculum design, assessment, and quality assurance. The review found few examples of the engagement of equivalent bodies in Brazil in research, advocacy, and training in support of quality improvement, and little attention on the part of public authorities to their potentially important role.

The federal system of quality assurance does not apply to all higher education providers in Brazil

As noted earlier, the systems for external quality assurance of HEIs and undergraduate programmes analysed in this report apply only to private HEIs and federal public HEIs. State and municipal public institutions – which account for almost 10% of enrolment - are not subject to SINAES, but rather to state-level regulatory and quality assurance rules. Although this situation reflects the constitutional distribution of competences in the Brazilian state, which allow considerable autonomy to states and municipalities, it leads to a fragmented system and means there is no single national benchmark of higher education quality. A single quality reference framework would make external quality assurance for higher education more transparent and understandable for students and their families.

Key recommendations

1. Create an independent quality assurance agency

To address the conflicting responsibilities of MEC – or indeed any future ministry responsible for higher education - Brazilian authorities should consider creating an independent quality assurance body that stands outside the Ministry, in line with practice in many OECD and partner countries. This agency would take the lead in implementing the reformed system of quality assurance proposed in this report. Good international models of bodies with strong legal, financial, and administrative independence exist. In systems with a similar legal tradition to Brazil, such agencies include, for example, Portugal’s Agency for the Assessment and Accreditation of Higher Education (A3ES).

The work to design and create any new agency for quality assurance in Brazil will need to address some key questions:

Which existing functions should be transferred to the new agency? In principle, the new agency would combine the evaluation functions coordinated by INEP’s higher education evaluation directorate (DAES) and the regulatory and supervisory roles of SERES. The changes to the overall model of regulation, evaluation and supervision proposed in this report – such as increased focus on institutional review, reduced numbers of programme-level reviews, a reformed ENADE and a new indicator dashboard - will affect requirements for staff in different roles. The advantages and disadvantages of creating specific evaluation units for different sets of disciplines (natural sciences, social sciences etc.) should be considered. Such units, integrated within the agency, could potentially allow evaluation to be better tailored to individual disciplines and work more closely with the discipline-specific CAPES evaluations.

Should some tasks be devolved to decentralised offices in the states? The current system of quality assurance in the federal higher education system is highly centralised, with all evaluation and regulation activities coordinated from Brasilía. Devolving responsibility to regional departments might theoretically allow a more differentiated approach to quality assurance, with better consideration of the large regional differences in Brazil. However, in the view of the OECD team, distinct quality assurance procedures in different parts of the country would risk creating a two- (or multi-)tier system and undermining national recognition of quality standards. It could be possible, however, to establish regional offices to house professional inspectorates to undertake inspection of infrastructure and institutional management, freeing academic peer reviewers to focus on assessment of academic performance, potentially remotely (see above). The costs of the current system of peer review and the potential costs of a permanent inspectorate would need to be assessed in detail.

How should the new agency be funded? The current system of external quality assurance in Brazil is funded by a combination of public resources (paying the salaries of public servants, for example) and fees paid by institutions for evaluation activities. Quality assurance agencies in a number of systems, including the Portuguese example mentioned above, are funded primarily through fees from institutions. To ensure efficient use of public resources, this should be the long-term aim in Brazil. A thorough analysis will be required to determine the costs of a new agency and the level of fees needed to finance its operation.

The OECD team recognises that there is an existing proposal to create a National Institute for the Supervision and Evaluation of Higher Education (INSAES), that was introduced as a draft bill in Congress in 2012 (Congreso Nacional, 2012[4]), but not pursued. This initiative effectively also proposed a merger of the functions of SERES and INEP, but was criticised for its potential cost and limited added value. The OECD team believes that a new agency would be the most effective way to implement a reformed system of external quality assurance. The reforms proposed in this report are vital to improve the effectiveness and efficiency of the system and any future agency must be designed to operate as efficiently as possible and with limited direct public subsidy.

2. Strengthen CONAES

To ensure that the quality assurance agency has an advisory council that brings a wide social vision to its work, CONAES could take on this responsibility, after substantial modification. CONAES would be a council with members holding fixed and staggered terms to ensure their independence of government, and encompass balanced representation from students, public and private sector employers, instructors from public and private higher education institutions, higher education administrators, leading researchers, and the senior policy official in MEC with responsibility for taking a comprehensive view of higher education.

3. Restructure the government departments that are responsible for higher education

MEC – or any future ministry responsible for higher education - can support the improvement of quality assurance by restructuring its responsibilities for higher education. This could entail creating a post for a principal policy officer who takes a comprehensive and strategic view of the entire Brazilian higher education system – which the Ministry presently lacks. Units organised along sectoral lines, for example, could support the work of a senior official. These might include groups responsible for (a) federal universities; (b) private universities; (c) technical higher education; and (d) coordination with state and municipal higher education institutions. This scheme of organisation would benefit the nation’s quality assurance system by supporting a strategic and comprehensive vision for the higher education system, by clarifying the role of private provision within the system, and by encouraging continued differentiation of institutions and policies.

4. Incentivise the development of expertise in quality assurance in sector organisations

In monitoring and evaluating the nation’s quality assurance system, a reconstituted quality assurance agency and advisory council (i.e. CONAES) should focus on supporting the development of quality enhancing organisations outside of government. For example, it could support collaboration among state and national bodies of institutional evaluation offices (CPAs), so they share experiences of quality management and improvement practices with one another.

5. Explore how a reformed external quality assurance system could also apply to state and municipal institutions

A single system of external quality assurance applying to all higher education institutions in the country would be more transparent for students and the public than that current co-existence of a large federal system and individual systems for state and municipal institutions in each state. The federal and state authorities, working with the higher education sector, should explore how – and under what conditions - a reformed federal system of quality assurance could be applied to state and municipal institutions, while respecting the distribution of competences enshrined in the constitution of the Union.

References

[4] Congreso Nacional (2012), Projeto de lei: cria o Instituto Nacional de Supervisão e Avaliação da Educação Superior - INSAES, e dá outras providências., Secretaria de Assuntos Parlamentares, http://www.planalto.gov.br/ccivil_03/Projetos/PL/2012/msg398-31agosto2012.htm (accessed on 30 November 2018).

[2] Escueta, M. et al. (2017), “Education Technology: An Evidence-Based Review”, NBER Working Paper Series, No. 23744, National Bureau of Economic Research, Cambridge, http://www.nber.org/papers/w23744 (accessed on 23 November 2018).

[1] Presidência da República (2004), Lei no 10 861 de 14 de Abril 2004, Institui o Sistema Nacional de Avaliação da Educação Superior – SINAES e dá outras providências (Law 10 861 establishing the National System for the Evaluation of Higher Education - SINAES and other measures), http://www.planalto.gov.br/ccivil_03/_ato2004-2006/2004/lei/l10.861.htm (accessed on 10 November 2018).

[3] TCU (2018), RA 01047120170 Relatório de auditoria operacional. Atuação da Secretaria de Regulação e Supervisão da Educação Superior do Ministério da Educação - Seres/Mec e do Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira - Inep nos processos de regulação, supervisão e avaliação dos cursos superiores de graduação no país. (Operational audit report: Activities of SERES/MEC and INEP related to regulation, supervision and evaluation of higher education programmes in the country), Tribunal de Contas da União, Brasília, https://tcu.jusbrasil.com.br/jurisprudencia/582915007/relatorio-de-auditoria-ra-ra-1047120170/relatorio-582915239?ref=juris-tabs (accessed on 13 November 2018).

2.

Scope of the review and analytical framework

This chapter provides a brief overview of the main aspects of the external procedures in place in Brazil to assure the quality of the federal higher education system – the subject of this review - before setting out the framework that the review team has used to structure and guide its assessment of the relevance, effectiveness and efficiency of these procedures. To contextualise the analysis in the review and the analytical framework used, the chapter also provides a brief review of some of the main developments and challenges faced by higher education quality assurance systems internationally.

2.1. Focus of this chapter

This chapter provides a brief overview of the main aspects of the external procedures in place in Brazil to assure the quality of the federal higher education system, before setting out the framework that the review team has used to structure and guide its assessment of the relevance, effectiveness and efficiency of these procedures.

As with other OECD education policy reviews, this review provides a qualitative assessment of the specific policies under scrutiny. It takes into account the objectives established by national authorities for the policies in question and bases its judgements on documentary evidence and stakeholder opinion regarding the implementation of the policies in Brazil; lessons from international standards and practice in other OECD and partner countries; and the experience and expert opinion of the review team members.

The review recognises that systems for the external quality assurance of higher education, like other types of public policy, need to be tailored to the specific situation in the jurisdictions where they are applied. As in other policy fields, there is no single set of “best practices” in the external quality assurance of higher education that can be applied uniformly to all higher education systems. This review therefore presents international standards and guidelines – to the extent that these exist – and practice examples from other countries as reference points and potential sources of inspiration for Brazil, rather ready-made models that could or should be applied in the Brazilian context.

The simple analytical framework used to guide the review – outlined more fully later in this chapter - is based on standard policy evaluation criteria. The most important criteria are the relevance of the objectives of different parts of the external quality assurance system to the challenges and requirements of the Brazilian context; the effectiveness of the different aspects of the system in achieving their objectives; and the efficiency (and cost-effectiveness) with which they do this. In judging the relevance, effectiveness and efficiency of different aspects of the quality assurance system for higher education in Brazil, the review team has taken into account domestic criteria (for example, does the system fulfil the objectives established by Brazilian legislators? Is the system considered to be effective by Brazilian stakeholders?); and international criteria (for example, does the Brazilian system promote minimum quality standards, differentiated assessment of quality or quality enhancement as well as systems in other jurisdictions?).

Before setting out in more detail the analytical framework and the evaluative questions that structure the rest of the analysis, the following sections, first, provide an overview of main components of Brazil’s external quality assurance systems for higher education and, second, briefly examine the development of external quality assurance systems in higher education more generally, and common challenges faced internationally.

2.2. External quality assurance of higher education in Brazil

The federal government regulates most higher education in Brazil

Brazil has well-established systems in place at national level to regulate the operation of public and private higher education providers in the federal higher education system and assess and monitor the quality of their teaching and learning activities.

Within Brazil’s federal governance structure, responsibility for providing and regulating higher education is formally shared between the federal government, the 27 federative units (the 26 states and the federal district of Brasília) and the municipalities. The federation, many states and a small proportion of (large) municipalities all have public higher education institutions (HEIs) falling under their responsibility. All private higher education providers in the country legally fall under the regulatory responsibility of the federal government. The “federal higher education system” thus comprises federal public and all private HEIs. Of the roughly 2 400 HEIs in Brazil, 92% (federal public and private) fall under the regulatory responsibility of the federal government and these institutions, together, account for 91% of undergraduate enrolment. 75% total undergraduate enrolment in Brazil is in the private sector (see Section 3.4 for an overview of the Brazilian higher education landscape).

Quality is assured through related processes of regulation, evaluation and supervision

The federal authorities assure the quality of higher education institutions and undergraduate education¹ in the federal system through a combination of distinct, but closely related, processes referred to as regulation, evaluation and supervision and currently coordinated by the Ministry of Education (MEC):

The regulation of higher education, undertaken by Secretariat for Regulation and Supervision of Higher Education (SERES), a division of MEC, involves issuing formal approval (in the form of regulatory acts) for the operation of higher education institutions and individual undergraduate programmes. All public and private HEIs in the federal system formally require accredited status to operate² (and periodic re-accreditation) and official recognition of the undergraduate programmes they provide. Programme recognition must also be renewed periodically, based on the results of quality evaluations. As discussed in Chapter 4 depending on their level of institutional autonomy, institutions may also need prior authorisation from MEC to start new undergraduate programmes.

SERES makes its decisions regarding the accreditation and re-accreditation of HEIs and the authorisation, recognition and renewal of recognition of undergraduate programmes taking into account the results of institutional and programme evaluations, undertaken by the evaluation division of the Anísio Teixeira National Institute for Educational Studies and Research (INEP). These evaluation activities encompass external assessment of institutions and programmes and assessment of student learning outcomes through the National Examination of Student Performance (ENADE). Collectively, these three types of evaluation (institutional, programme and student) form the National System of Higher Education Evaluation (SINAES), which was established in its current form in 2004 (Presidência da República, 2004[1]). The evaluation of institutions and programmes (discussed in more depth in the following chapters) is based on on-site inspections by external review panels, results obtained by graduates in ENADE and, at present, a limited number of other quantitative indicators intended to measure programme quality.

Alongside its regulatory duties (the issuing of regulatory acts), SERES is also tasked with the supervision of quality in the federal higher education system. In practice, this means ongoing monitoring of quality levels in the federal higher education system, using the results produced by the evaluation work coordinated by INEP, and taking preventive and corrective measures when quality problems are identified in individual programmes and institutions (Presidência da República, 2017, pp. 1, art.2[2]). SERES can require institutions to take steps to address quality problems identified in evaluations, impose sanctions or proactively require intensified monitoring and evaluation of particular programmes or institutions.

Figure 2.1 illustrates the inter-relationship between regulation, evaluation and supervision, whereby the evaluation activities, coordinated by INEP, that make up the SINAES feed into the regulatory and supervisory activities undertaken by SERES.

Figure 2.1. Regulation, evaluation and supervision in Brazilian higher education

Source: OECD Secretariat based on (Presidência da República, 2004[1]; Presidência da República, 2017[2])

Separate procedures exist to assure the quality of academic postgraduate programmes

Postgraduate education in Brazil³ takes the form of either purely vocational “specialisation” programmes, referred to in the Brazilian system as lato sensu programmes (which include MBAs), or academic Master’s, Professional Master’s and Doctoral programmes, which are classed as stricto sensu postgraduate programmes. Accredited HEIs in the federal education system may provide lato sensu programmes without regulatory authorisation or recognition from MEC, provided they operate at least one formally recognised undergraduate programme or at least one approved stricto sensu postgraduate programme (Presidência da República, 2017, p. 29.2[2]). The provision of academic, stricto sensu, postgraduate programmes in conditioned on prior evaluation and approval by the Foundation for the Coordination and Improvement of Higher Level Personnel (CAPES), a decentralised agency of the Ministry of Education.

CAPES has been responsible for assuring the quality of academic postgraduate educational programmes since the mid-1970s. Since 1998, it has operated a quality assurance system that requires all proposed new academic postgraduate programmes to achieve a positive evaluation score in a peer review process and, from then on, to receive a positive evaluation in periodic reviews (currently every four years). CAPES is responsible for the quality assurance of all academic postgraduate programmes in Brazil, including in state and municipal institutions, which are not subject to SINAES. In practice, it fulfils the roles of evaluation, regulation and supervision for academic postgraduate programmes – functions which are split between INEP and SERES for undergraduate provision in the federal system.

However, CAPES only evaluates individual postgraduate programmes. It requires that all programmes within its remit be provided in formally accredited institutions and thus relies on SERES and state education authorities to undertake institutional accreditation (for institutions in the federal and state systems, respectively). Moreover, the results of CAPES evaluations are taken into account in a composite indicator of institutional quality used by INEP (see Chapter 7).

The normative framework for SINAES has recently been updated

The legal basis governing higher education in Brazil is provided in the federal Constitution and by the 1996 Education Act (Presidência da República, 1996[3]), which establishes basic principles concerning the role of higher education, the division of competences between the Union and the states and the role of the federal government in quality assurance. SINAES was established by a specific law passed by Congress in 2004 (Presidência da República, 2004[1]). This sets out the basic principles of the SINAES, including the requirement for institutional, programme and student assessments, the role of on-site inspections and ENADE, and the relationship between evaluation, regulation and supervision. The current regime for evaluation of stricto sensu postgraduate provision was established through ordinances (portarias) issued by MEC and CAPES from 1998 onwards and which have been updated periodically.

The detailed implementation rules for SINAES (evaluation) and the processes of regulation and supervision, reflecting the 2004 law, were initially established in a 2006 presidential decree (Presidência da República, 2006[4]) and supplemented by ordinances issued directly by MEC. This decree was replaced at the end of a 2017 by a new decree setting out the processes of evaluation, regulation and supervision in greater detail than had previously been the case (Presidência da República, 2017[2]). The changes brought about by this new decree are discussed in the relevant sections of this report. In broad terms, however, it seeks to simplify the administrative processes related to authorisation of undergraduate programmes and modifications to programmes, rationalise procedures for on-site inspections and provide more clarity about the preventive and corrective measures SERES may take in its supervisory role (MEC, 2017[5]).

These recent changes have been reflected to the extent possible in the review findings and taken into account in the formulation of recommendations. However, as the changes have only recently started to be reflected in the practices of SERES and INEP, it has not been possible to seek views from stakeholders about their practical impact or to judge their effectiveness in practice.

2.3. Quality assurance in higher education internationally: developments and challenges

External quality assurance in higher education has developed in recent decades

The development of quality assurance systems in higher education is a comparatively recent phenomenon. Historically, the quality of learning and teaching in higher education was generally – and largely implicitly - assumed to be guaranteed by the presence of academics with an established record of scholarship. A high degree of individual academic autonomy in universities meant that, in many countries, neither public officials nor university and faculty management frequently intervened in the teaching activities of staff members. Although governments may have granted or withdrawn permission for universities to operate, and may have provided funding, owned buildings and even regulated staff conditions and salaries in public institutions, in much of the world they rarely or never engaged with institutions’ day-to-day teaching and research activities. This pattern of strongly independent public or private not-for-profit institutions is the traditional model of higher education institutions in most of Latin America.

Over the last three decades, this situation has evolved, as governments across the world have introduced external quality assurance systems for higher education and higher education institutions have increasingly developed formalised internal quality procedures for learning and teaching. Key factors driving increased government intervention have included the expansion of higher education and the break-down of historical trust relationships among small elites; the expansion of demand-absorbing private sector provision – particularly in Latin America and Eastern and Southern Europe; and a broader trend for governments and society to demand greater evidence of performance and value for money from publicly supported institutions and services (OECD, 2008[6]; Brunner and Miranda, 2016[7]).

Quality assurance systems have different objectives

Literature on quality assurance in higher education often distinguishes between the related goals of accountability and quality enhancement (ESG, 2015[8]; CHEA, 2016[9]). Accountability refers to the aim of providing information to assure the public (including students and their families) of the quality of higher education institutions’ activities. This is a goal common to virtually all external quality assurance systems in higher education. Quality enhancement involves providing advice and recommendations on how higher education providers might improve what they are doing, and is a less well-established aspect of many external quality assurance systems.

Some external ‘quality assurance’ systems are essentially little more than licensing systems, where higher education providers are authorised to operate by public authorities if they meet minimal operating requirements. Such systems provide only a minimum level of accountability, which is essentially limited to providing the public with information on whether a provider is legally registered or not. Other systems make authorisation and accreditation of higher education activities (or the allocation of public funds) dependent on positive results from a more in-depth evaluation of indicators of quality. Depending on the relevance and quality of the criteria and data sources used by HEIs and external evaluation agencies, such systems might be expected to offer a greater guarantee of minimum quality standards and, as a result, a better degree of accountability. In addition to providing these kinds of accountability guarantees, the most advanced external quality systems also seek to promote enhancement of quality and continuous improvement. Such systems generally seek to move beyond external regulation and control to promote a quality culture in all areas of higher education activity, in partnership with HEIs and academic staff.

Defining and measuring quality is challenging

Arriving at a shared understanding of what quality in higher education is and how it should be measured has proved challenging for those involved in developing and running quality assurance systems (CHEA, 2016[9]). There are at least three main reasons for this:

1. As noted in the 2015 Standards and Guidelines for Quality Assurance in the European Higher Education Area ‘Higher education aims to fulfil multiple purposes’ and ‘stakeholders, who may prioritise different purposes, can view quality in higher education differently’. The importance attached to different aspects of the educational process or different kinds of learning outcome acquired by students (for example, theoretical knowledge as opposed to practical skills) can vary between individuals and groups within a single higher education system and internationally. The co-existence of higher education institutions with diverse missions and types of programme is explicitly acknowledged in the Brazilian legislation establishing the SINAES (Presidência da República, 2004, p. art.3[1]).

2. The expansion of higher education has led to more diverse modes of provision and student populations, and to demands for a more diverse set of programmes and institutional profiles. This means that quality in higher education now comes in more diverse forms and needs to be measured in more diverse ways.

3. Even when there is agreement on the components of quality in different types of higher education context, it may be conceptually and technically difficult to measure these components in a reliable way. If what is important may not always be measurable, what is measurable may not always be important. In the absence of reliable or feasible direct measures of different aspects of quality, quality assurance systems frequently resort to the use of proxy measures, which themselves can become the subject of disagreement.

Quality education is education that is fit for purpose

Notwithstanding these challenges, recent international efforts to develop a shared understanding of quality in higher education (CHEA, 2016, p. 48[9]) argue that quality education is best conceived of in terms of fitness for purpose. In other words, good quality higher education is education that:

1. Sets out to deliver the right kinds of learning outcomes for students – where the right kinds of learning outcomes are ones that meet the needs of students and society. The concept of learning outcomes encompasses both breadth and depth of knowledge and skills: good quality education programmes establish the right intended learning objectives, at the right level of complexity for their target student population;

2. Creates and uses a learning environment (qualified teachers, teaching methods, learning resources, opportunities to gain practical experience, etc.) suitable for achieving these learning outcomes and;

3. Succeeds in practice in delivering the intended learning outcomes for as many participating students as possible who begin studying.

The first element above is about intentions and, specifically, setting relevant learning objectives. The second element concerns inputs (including teaching staff and resources) and processes (including teaching methods and activities). The third element deals with the output⁴ of the educational process. In addition, those concerned with the quality of higher education may look at the broader outcomes of graduates who have gone through the educational process and in particular their entry into and progression within the labour market. Outcomes in this sense are influenced by the educational process, but also by a range of other external factors.

Quality systems focus on inputs and processes, less frequently on outputs and outcomes

External quality assurance systems in higher education initially focused to a large extent on measuring inputs, such as the level of qualification of teaching staff and the number of books in the library, as a way of gauging the quality of provision. Although input measures are usually readily available and objective, on their own they provide little evidence of quality in practice. The fact that a member of teaching staff has a Master’s degree of a PhD might be considered important and perhaps necessary, but it is not sufficient to assure their ability as a teacher. Funding is another input. Adequate funding might be assumed to be a precondition for creating an effective learning environment. However, defining what level of funding is ‘adequate’ is often challenging and controversial and, once defined, adequate funding is not, in itself, a guarantee of quality.

Processes – in particular learning, teaching and assessment methods – can provide an indication of the likely effectiveness of the teaching and learning experience for students, and are therefore monitored in many quality assurance systems in higher education. For example, procedures for external marking of written assessments and examinations may be taken into account as an indicator of the reliability of assessment processes (and thus the validity of the results and qualifications awarded by an educational programme). However, many aspects of teaching and learning are difficult to capture and assess in a binary (yes/no) or quantitative way, which makes it difficult to collect quantitative data on such processes.

In recent years, there has been an increasing focus on the possibility of using output and outcome information in quality assurance systems. The most direct outputs of the educational process are graduates with increased knowledge and skills (learning outcomes) acquired through their education. Isolating the specific added value of a student’s higher education experience from other factors like the social and cultural background is intrinsically very challenging. However, internationally, standardised tests are used in some jurisdictions to measure the skills and competences of higher education students and graduates in a comparable manner.

In the United States, for example, the CLA and CLA+ tests have been deployed widely to test generic competences (CAE, 2018[10]), while in Mexico, the Exámenes Generales para el Egreso de Licenciatura (EGEL) are used by many institutions to test graduates in specific disciplinary areas (CENEVAL, 2018[11]). In Colombia, all higher education students are required to take a general competence test (Saber Pro) in order to graduate (ICFES, 2018[12]). Although the Saber Pro exams in Colombia are compulsory, the results obtained by students are not used directly to generate quality scores for the institutions they attended. As discussed in Chapter 5, the ENADE examination in Brazil is the only example, in a major higher education system, of large-scale, external examinations that are both compulsory for students and used directly in the quality assurance of programmes and institutions.

Other output or outcome related measures do not look at student learning per se, but rather at indirectly related issues. For outputs, this includes graduation and completion (e.g. proportion of students successfully completing their course). For outcomes, this includes employment outcomes (e.g. proportion of graduates that are employed and in what types of job).

The relationship between output and outcome measures and course quality is not always straightforward. Although a high-quality course may be expected to provide good support to students from different backgrounds to allow them to complete their course successfully, a 100% completion rate may be an indication of low standards, rather than good quality. While it is logical to assume that high quality courses prepare students well to get good jobs, graduate employment outcomes depend on a wide range of factors beyond the quality of the educational programme. Graduates from a prestigious, but objectively poor quality, course may consistently succeed in getting good jobs. Graduates from high quality courses may have difficulty in finding appropriate work if the relationship between the course content and knowledge and skills required in the labour market is weak or general employment conditions are difficult.

Table 2.1. Quality indicators: objectives, inputs, processes, outputs and outcomes

| Example indicators

Objectives of the educational process

(Objective indicators)

Relevance / appropriateness of intended learning outcomes for the educational programme

Transparency / clarity of intended learning outcomes

Intended learning outcomes have been developed taking into account latest developments in the scientific or professional area concerned

Intended learning outcomes have been developed in consultation with relevant external stakeholders (employers, sector associations, unions etc.)

Inputs to the educational process

(Input indicators)

Funding available to ensure stable operation of the educational provider (continuity of service, framework conditions for educational provision)

Funding available for the educational processes (to pay for staff, buildings, equipment etc.)

Ratio of teaching staff to students

Qualifications and research outputs of teaching staff

Surface area of classroom space per student

Accessibility of teaching and learning spaces for students with disabilities

Library facilities (number of books, online resources available, etc.)

Laboratory facilities

The educational process

(Process indicators)

Teaching contact hours

(where relevant) Hours of practical activities built into curriculum (laboratory time etc.)

(where relevant) Work-based learning experience built into curriculum and conditions of this (internships, placements etc.)

Use of online / blended learning in curriculum

Staff time dedicated to academic support outside class

Availability of / time dedicated to pastoral / non-academic support

Marking arrangements for assessments (double marking etc.)

Ongoing quality assurance measures in place (peer review of teaching etc.)

Outputs of the educational process

(Output indicators)

Proportion of students graduating successfully (successful completion rate)

Average time taken to graduate

Knowledge and skills of students (learning outcomes) at end of programme.

Added value: additional knowledge and skills (learning outcomes) gained during the programme (measured by tracking skills over time)

Outcomes of the educational process

(Outcome indicators)

Employment status of graduates

Alignment between qualification type and level of employment

Graduate earnings

Social mobility

Evidence of active citizenship among graduates

Source: OECD, Education and Skills Directorate

Measuring the quality of postgraduate education may present specific challenges. Although the quality of taught Master’s programmes is often assured using the same types of indicators as for undergraduate education, the quality of research-oriented Master’s and doctoral programmes is also influenced to a great extent by the broader research environment in which they take place. Assessment of the quality of postgraduate provision may therefore consider research-related input, process and output indicators. These can include the research performance of the academic department hosting the programme (input), opportunities for postgraduate students to attend conferences or organise and participate in research-related events (process) or the number and quality of research outputs by students (output). In a world where postgraduate students increasingly go on to work outside academia and scientific research, however, the quality indicators used need to adapt to take this into account.

2.4. An analytical framework for the review

The objectives of the review

The terms of reference for this review, agreed with the Ministry of Education at the start of the project, call on the review team to assess the relevance, effectiveness and efficiency of the external quality assurance procedures applicable to undergraduate and postgraduate programmes and HEIs in the federal higher education system in Brazil. Specially, the terms of reference ask the team to consider the effectiveness and efficiency of the systems in a) ensuring minimum quality standards (basic accountability); b) providing differentiated measurement of quality (between types of provision and levels of quality offered) and; c) promoting improvement of quality and quality-oriented practices in HEIs (quality enhancement). The team was invited to provide an analysis in relation to these points and recommendations for improving the systems in place.

Disaggregating the different components of the quality assurance system

As illustrated by the earlier discussion, the procedures for external quality assurance system in the federal higher education system comprise a set of distinct processes for HEIs, undergraduate programmes and stricto sensu postgraduate programmes. For each of these, specific procedures exist governing:

1. “Market entry” for new institutions and programmes, based on ex-ante assessment of the likely quality of the institution or programme proposed (before they begin operation);

2. Ongoing monitoring and evaluation of quality for existing institutions and programmes and;

3. The provision of feedback to programmes and institutions and actions to taken to respond to quality problems detected through the evaluation processes, including sanctions.

These three stages of quality assurance, which bring together the regulatory and supervisory functions of SERES, the evaluation function of INEP and the combined evaluation and supervisory roles of CAPES, are illustrated in Figure 2.2.

Figure 2.2. External quality assurance system for higher education in Brazil

Components of the external quality assurance system for higher education in Brazil

Source: OECD Secretariat

The review set out to analyse all of these components. Taking into account the way the different components are organised in practice and related to each other, they have been grouped and analysed as follows:

1. Market entry for new HEIs and new undergraduate programmes (SINAES evaluation by INEP and regulatory decisions by SERES) – analysed in Chapter 4 of this report.

2. Ongoing monitoring and evaluation of existing undergraduate programmes and related feedback and corrective measures (SINAES evaluation by INEP and regulation and supervision decisions by SERES) – analysed in Chapter 5.

3. Market entry and periodic evaluation of stricto sensu postgraduate programmes and related feedback and corrective measures, coordinated by CAPES – analysed in Chapter 6.

4. Ongoing monitoring and evaluation of HEIs and feedback and corrective measures (SINAES evaluation by INEP and regulation and supervision decisions by SERES) – analysed in Chapter 7.

In addition, Chapter 8 of this report analyses the governance and administrative bodies and arrangements that have been created to implement and oversee the processes outlined above.

An evaluative framework to structure the analysis

The terms of reference for the review specify basic evaluation criteria, which should structure the assessment of the components of the higher education quality assurance systems set out above:

1. The analysis of the relevance of the procedures and governance arrangements needs to consider the extent to which the objectives of these procedures and arrangements respond to the challenges and requirements of the Brazilian context. The objectives of processes and arrangements may be specified explicitly in legislation or be implicit in the approaches taken to implementation.

2. The analysis of the effectiveness of the different aspects of the system focuses on the extent to which they achieve their explicit and implicit objectives in practice.

3. The analysis of the efficiency of the different aspects of the system considers the relationship between the resources committed to the processes and bodies and results and impact achieved.

These are the core criteria used in many policy and programme evaluation frameworks used by national and international bodies, including and UNESCO and the OECD (UNESCO, 2007[13]; OECD/DAC, 2018[14]). The assessment of effectiveness is often complemented by consideration of the wider impact of policies and programmes, in an attempt to capture effects beyond the immediate outputs of the policy or programme in question. A full analysis of the results and impacts of the Brazilian external quality assurance system would be methodologically challenging in the absence of any counter-factual situations (higher education not subject to the quality procedures being examined). In any case, such an impact analysis is beyond the scope of this review. Nevertheless, the review team has sought to consider the wider effects of the quality assurance system, to the extent possible.

As noted, the review assesses the relevance, effectiveness and efficiency (and wider impact) of the higher education quality assurance system in Brazil using two main types of judgement criteria:

1. Domestic criteria, which are based on objectives, viewpoints and evidence encountered in Brazil, including the explicit objectives of the Brazilian legislation and stakeholder perceptions about effectiveness and efficiency and;

2. International criteria, which take into account international standards and guidelines for effective external quality assurance in higher education and examples of effective practice in other jurisdictions.

Many variables can affect the design of quality assurance systems in higher education. National contexts have a strong impact on how quality assurance systems are configured and there is no one-size-fits-all institutional model of good practice. However, the analyses of various international associations working in the field of quality assurance in higher education point to a growing international consensus around a set of principles that can guide the design of effective external quality assurance. Based on existing international guidelines and available literature, some of the key attributes one might expect to see in an effective external quality assurance system, and their relationship to relevance, effective and efficiency, are summarised in Table 2.2 below.

Table 2.2. Factors in effective quality assurance systems

Aspect of the system | Characteristics of effective QA systems

RELEVANCE

1. Objectives of QA processes and the definition of quality

The objectives of each step in external and internal quality assurance processes are clearly formulated and relevant to the challenges faced by the higher education system

An appropriate balance is struck between aiming to eliminate or avoid poor quality provision and improve existing provision, including provision that is already judged to be of good quality

The quality of teaching and learning is conceived of as fitness for purpose, where the primary purpose of the educational experience is to equip students with relevant learning outcomes (a student-centred conception of quality).

The definition of relevant learning outcomes takes into account an appropriate range of knowledge and skills and the level (degree of complexity) of the knowledge and skills acquired [ie transversal skills as well as specific content knowledge]

Definitions of quality are flexible enough to recognise quality in different forms in different types of educational programme

EFFECTIVENESS

2. Measurement of quality

Quality is measured using an appropriately wide range of relevant and reliable indicators, including input, process and output indicators

Where appropriate, differentiated indicators are used for different types of provision to take account of their specific characteristics (e.g. academic vs professional courses)

EFFECTIVENESS

3. Responsibility for quality and for quality assurance

Teaching staff and higher education providers are clearly identified as those with primary responsibility for delivering quality

Subsidiarity: decisions about quality are taken at the lowest level possible while maintaining effectiveness and adequate accountability

Quality assurance agency/ agencies act in the public interest, are adequately resourced and are sufficiently independent from both the higher education sector and government.

Additional government initiatives to promote quality are coordinated with quality assurance systems to ensure consistency

EFFECTIVENESS

4. Use of information about quality

Where appropriate, evidence of poor quality is used to eliminate poor quality provision, with demonstrable results

Elsewhere, information about the quality performance of programmes and institutions is used systematically to improve quality, using an appropriate set of tools, with demonstrable results.

Information about the quality performance of programmes and institutions is made public to ensure transparency, with demonstrable results.

EFFECTIVENESS 5. Adapting to change and innovation | The quality assurance system is able to adapt rapidly and flexibly to take account of changes in the way teaching and learning are offered or could be offered The quality assurance system actively promotes adoption of valuable new course content, new technologies or learning approaches.
EFFICIENCY 6. Cost effectiveness and administrative burden | The system is cost effective for taxpayers and the system overall and reduces the administrative burden on the education providers to a minimum.

Source: Developed by the OECD Education and Skills Directorate, drawing on INQAAHE Guidelines of Good Practices 2016 (INQAAHE, 2016[15]); Standards and Guidelines for Quality Assurance in the European Higher Education Area (ESG, 2015[8]) and; CIQG International Quality Principles: Toward a Shared Understanding of Quality (CHEA, 2016[9])

Key questions will be addressed for each component of the quality assurance system

The domestic and international judgement criteria have been used to inform the analysis and conclusions in the subsequent sections of this report. For each component of the Brazilian external quality assurance system for higher education (grouped above), the report addresses the following key questions:

1. Relevance: Are the objectives of this component of the quality assurance system clear and relevant to needs in the Brazilian context?

2. Effectiveness: Does this component of the system use appropriate indicators (measures) of quality that would allow it to measure quality in line with the Brazilian legislation and international good practice?

3. Effectiveness: Is responsibility for assuring the quality of higher education appropriately distributed between higher education providers and external authorities and between different external authorities in this component of the QA system, taking into account the objective of Brazilian legislation and international experience?

4. Effectiveness: Is the information about quality collected / assembled in this component of the system used well to inform effective decisions about quality, promote quality improvement or increase transparency, in line with Brazilian legislation and international good practice?

5. Effectiveness: Is this component of the system able to adapt flexibility to accommodate change and does it actively promote innovation?

6. Efficiency: Is this component of the system efficient in line of the resource committed and the effects generated. Is it cost effective and does it maintain administrative burden at a minimum?

References

[7] Brunner, J. and D. Miranda (2016), Educación Superior en Iberoamérica - Informe 2016 - Aseguramiento de la Calidad, Centro Interuniversitario de Desarrollo (CINDA), Santiago de Chile, http://www.cinda.cl (accessed on 13 November 2018).

[10] CAE (2018), CLA+ for Higher Education, Council for Aid to Education, https://cae.org/flagship-assessments-cla-cwra/cla/ (accessed on 13 November 2018).

[11] CENEVAL (2018), Exámenes Generales para el Egreso de Licenciatura (EGEL), CENEVAL, http://www.ceneval.edu.mx/examenes-generales-de-egreso (accessed on 13 November 2018).

[9] CHEA (2016), The CIQG International Quality Principles: Toward a Shared Understanding of Quality, Council for Higher Education Accreditation , Washington, DC, http://www.chea..

[8] ESG (2015), Standards and Guidelines for Quality Assurance in the European Higher Education Area (ESG).

[12] ICFES (2018), Información general del examen Saber Pro, Instituto Colombiano para la Evaluación de la Educación, http://www2.icfes.gov.co/estudiantes-y-padres/saber-pro-estudiantes/informacion-general-del-examen (accessed on 13 November 2018).

[15] INQAAHE (2016), INQAAHE Guidelines of Good Practices 2016 - revised edition, International Network for Quality Assurance Agencies in Higher Education (INQAAHE), Barcelona, http://www.inqaahe.org/.

[5] MEC (2017), Decreto desburocratiza e premia instituições pela qualidade (Decree reduces bureaucracy and rewards institutions for quality), Ministério da Educação, http://portal.mec.gov.br/ultimas-noticias/212-educacao-superior-1690610854/58611-decreto-desburocratiza-e-premia-instituicoes-pela-qualidade (accessed on 12 November 2018).

[6] OECD (2008), OECD Reviews of Tertiary Education: Mexico 2008, OECD Reviews of Tertiary Education, OECD Publishing, Paris, http://dx.doi.org/10.1787/9789264039247-en.

[14] OECD/DAC (2018), DAC Criteria for Evaluating Development Assistance, OECD Development Assistance Committee (DAC), http://www.oecd.org/dac/evaluation/daccriteriaforevaluatingdevelopmentassistance.htm (accessed on 14 November 2018).

[2] Presidência da República (2017), Decreto Nº 9.235, de 15 de dezembro de 2017 - Dispõe sobre o exercício das funções de regulação, supervisão e avaliação das instituições de educação superior e dos cursos superiores de graduação e de pós-graduação no sistema federal de ensino. (Decree 9235 of 15 December 2017 - concerning exercise of the functions of regulation, supervision and evaluation of higher education institutions and undergraduate and postgraduate courses in the federal education system), http://www.planalto.gov.br/ccivil_03/_Ato2015-2018/2017/Decreto/D9235.htm (accessed on 10 November 2018).

[4] Presidência da República (2006), Decreto nº 5773 de 9 de maio de 2006, Dispõe sobre o exercício das funções de regulação, supervisão e avaliação de instituições de educação superior e cursos superiores de graduação e seqüenciais no sistema federal de ensino. (Decree 5773 concerning the exercise of the functions of regulation, supervision and evaluation of higher education institutions and undergraduate and short-cycle programmes in the federal education system), http://www.planalto.gov.br/ccivil_03/_Ato2004-2006/2006/Decreto/D5773.htm (accessed on 12 November 2018).

[1] Presidência da República (2004), Lei no 10 861 de 14 de Abril 2004, Institui o Sistema Nacional de Avaliação da Educação Superior – SINAES e dá outras providências (Law 10 861 establishing the National System for the Evaluation of Higher Education - SINAES and other measures), http://www.planalto.gov.br/ccivil_03/_ato2004-2006/2004/lei/l10.861.htm (accessed on 10 November 2018).

[3] Presidência da República (1996), Lei Nº 9.394, de 20 de dezembro de 1996, Estabelece as diretrizes e bases da educação nacional. (Law 9394 establishing basic guidelines for national education), http://www.planalto.gov.br/ccivil_03/LEIS/L9394.htm (accessed on 13 November 2018).

[13] UNESCO (2007), Evaluation Handbook, UNESCO Internal Oversight Service Evaluation Section, Paris, http://unesdoc.unesco.org/images/0015/001557/155748E.pdf (accessed on 13 November 2018).

Notes

 1.  Undergraduate education (graduação) in Brazil encompasses a) 4-6-year Bachelor's degrees (Bacharel), which are academically oriented and account for the majority of enrolment; b) Licentiate's degrees (Licenciado), which are 3-4-year teacher training qualifications and; c) professionally oriented, 2-3-year Advanced Technology Programmes (Cursos Superiores de Tecnologia). All of these qualifications are classified as ISCED 6.

 2.  Public HEIs have de facto accredited status through their acts of establishment, but formally require periodic re-accreditation.

 3.  Postgraduate education in Brazil encompasses a) academic Master’s degrees; b) professional Master’s degrees; and c) doctoral degrees. These qualifications are classified as stricto sensu postgraduate degrees and classified respectively as ISCED 7 (academic and professional Master’s) and ISCED 8 (doctoral degrees). In addition, many higher education institutions in Brazil offer professionally oriented postgraduate qualifications referred to as Specialisation degrees (Cursos de especialização em nível de pós-graduação). These are classified as lato sensu qualifications, which do not give access to doctoral degrees and are not evaluated by CAPES.

 4.  The terms ‘output’ and ‘outcome’ are not used consistently in literature about education systems and educational performance. In this report, qualified graduates with additional knowledge and skills are classed as an output of the educational process. Outcome is used primarily to refer to the subsequent labour market outcomes of graduates.

3.

Higher education in Brazil

This chapter presents a brief overview of the higher education system in Brazil and the underlying socio-economic context in which it operates. It starts with a short review of recent macroeconomic developments and demographic trends, before examining the main legal and administrative governance arrangements for the public and private higher education institutions that make up the federal higher education system and account for the vast majority of institutions and student enrolment in Brazil. The chapter then presents key data relating to the institutional landscape in higher education, types of programme offered, the teaching workforce, enrolment and attainment rates, a discussion of social equity and evidence about the learning and employment outcomes for graduates.

The statistical data for Israel are supplied by and under the responsibility of the relevant Israeli authorities. The use of such data by the OECD is without prejudice to the status of the Golan Heights, East Jerusalem and Israeli settlements in the West Bank under the terms of international law.

3.1. Focus of this chapter

Over the past decade, Brazil has seen rapid growth in participation in higher education, mostly enabled through the expansion of private higher education provision (de Magalhães Castro, 2015[1]). Enrolment in higher education increased from less than six million students in 2009 to over eight million in 2016, with over 75% of students now studying in private institutions (MEC, 2018 (unpublished)[2]). In the decade up to 2017, the tertiary education attainment rate among young adults in Brazil (aged 25-34) increased from 10% to 17%. The average rate of tertiary education attainment for this age group in OECD countries is 43% (OECD, 2018[3]).

The increase in higher education attainment in Brazil mirrors trends seen in other OECD and partner countries in the last decade, albeit from a lower starting base and with current attainment rates remaining low by international standards. Further expansion of higher education enrolment is an explicit objective of Brazil’s current National Education Plan, which aims for a third of 18-24 year olds to be enrolled in higher education by 2024 (MEC, 2014[4]).

This objective is consistent with policies pursued by governments across the world. Increasing levels of higher education attainment have long been associated with economic competitiveness in an increasingly knowledge-driven global economy (OECD, 2018[5]). As Brazil’s economy and labour market expand in more knowledge-intensive sectors, demand for higher-level skills is likely to increase.

However, with expansion, increased concerns about the quality of higher education in Brazil have emerged, particularly in some sections of the system (Salto, 2018[6]). Against this backdrop, Brazilian authorities have also focused considerable efforts on regulating private higher education providers and implementing mechanisms to assure the quality of higher education provision in all parts of the system. This report reviews the systems in place in Brazil to assure the quality of higher education. To provide context for the discussions of quality assurance that follow, this chapter provides a brief overview of the broad socio-economic context in which Brazil’s higher education system operates as well as key characteristics of the system itself.

3.2. The socio-economic context for higher education in Brazil

Economic conditions

Brazil’s economy is gradually emerging from the recession

In the early 2000s, macro-economic stability, positive demographic trends and favourable global economic conditions, including rising commodity prices, allowed the Brazilian economy to grow, leading to high employment rates, wage growth and an expansion of private and public consumption. The strong economic context, coupled with improving access to school education and extensive government transfer programmes have allowed an estimated 25 million Brazilians to escape poverty since 2003 (OECD, 2018[5]).

However, this model of economic growth reached its limits by the middle of the current decade. A rapidly-ageing population, deteriorating trade performance, political instability and rising public debt led to a deep and prolonged recession in 2015 that wiped out almost seven years of growth and doubled unemployment. Poverty levels have stagnated and Brazil remains one of the most unequal countries in the world. Nevertheless, since 2017, growth has resumed and annual inflation and unemployment have started to decline (Figure 3.1.)

Figure 3.1. GDP growth and unemployment in Brazil

Source: (OECD, 2018[5]) OECD Economic Surveys: Brazil 2018, OECD Publishing Paris, http://dx.doi.org/10.1787/eco_surveys-bra-2018-en

Stronger investment and productivity are key for future growth

In this context, the OECD’s most recent economic survey of Brazil (2018[5]) argues that stimulating growth and social progress will require strong investment across the economy. The OECD economists call for policies to reduce administrative burdens, simplify taxes and streamline licensing of economic activities. Raising productivity, which has been largely stagnant over the last 15 years, will also be crucial for future economic development. Improvements in productivity require not only more investment in physical capital, but also in the skills of people (OECD, 2018[5]). Ensuring access to high quality higher education for all will be key for productivity growth.

Demographic trends and social conditions

Brazil has a young population, that is beginning to age

At 25%, the share of young people (aged 15-24) in the working-age population in Brazil is currently high in comparison to most OECD countries. However, by 2050, the proportion of young people in the working-age population in Brazil is forecast to have fallen to below the average of OECD and selected emerging economies (OECD, 2014[7]). This will mean fewer young people are available to generate wealth to help support the rest of society and each individual will need to be more productive just to maintain current living standards. Studies suggest that a highly educated – and thus more productive - population will be key to responding to the challenges of a higher old-age dependency ratio (Dwyer et al., 2016[8]).

High levels of inequality

In contrast to many OECD countries, Brazil has seen a decrease in income inequality over the past decade. However, the Gini coefficient – measuring income inequality - remains higher than in any OECD country (Figure 3.2). Half of the population receives 10% of total household income, while the other half holds 90% (OECD, 2018[5]).

Figure 3.2. Income inequality

Gini coefficient, 0 = complete equality; 1 = complete inequality, 2017 or latest available

Source: (OECD Data, 2018[9]), Income inequality, https://data.oecd.org/inequality/income-inequality.htm

Significant inequalities also exist along geographical and ethnic lines. For example, a 2014 study calculated that the likelihood of a young, black woman living in the Northeast of Brazil being unemployed was 28.6%. This compared to an unemployment rate for white males living in the South of 7.6% (OECD, 2014[7]). Among the richest 1% of the population, less than one in five of are black or mixed race, even though these groups account for over 50% of the total population. Among the poorest 10% of the population, over 70% are black or mixed race (IBGE, 2014[10]).

Crime and corruption are widespread in Brazil

Brazil ranks 96th out of 180 countries in the latest Transparency International corruption index (Transparency International, 2018[11]). Scandals that surfaced in relation to corruption in public procurement, including by state-owned companies, and infrastructure concessions in recent years have created significant political turmoil. The OECD (2018[5]) has highlighted improving transparency and accountability as a key priority to tackle the root causes of corruption.

According to the latest OECD data, Brazil's homicide rate is 27.6 per 100 000 population, more than seven times the OECD average of 3.6 (OECD, 2017[12]). In addition to being a social and criminal justice issue, reports suggest that high levels of violence have a negative impact on economic growth due to the direct costs of crime and as an indirect constraint for business growth (World Bank, 2006[13]).

3.3. Governance arrangements, funding and key public policies for higher education

Governance of higher education

Responsibility for higher education is shared between the Union, states and municipalities

Higher education provision in Brazil is a shared responsibility between the federal government, the 27 federative units (the 26 states and the federal district of Brasília) and the municipalities. The federal authorities, state governments and municipalities are all permitted to create and fund the operation of public higher education institutions. In practice, the involvement of state and municipal authorities in providing higher education varies considerably between states across the country. The federal government, through the National Education Plan, establishes the national strategy for higher education. It has also assumed primary responsibility for funding student aid programmes, is responsible for the external quality assurance of federal public higher education providers and has exclusive responsibility for licensing and assuring the quality of private higher education providers throughout the country. State governments are responsible for regulating and assuring the quality of state and municipal public institutions in their territories.

Ministry of Education steers, regulates and supervises the federal higher education system

The federal Ministry of Education (Ministério da Educação, MEC) is responsible for establishing national education policy at the federal level, as well as co-ordinating the different levels and parts of the education system. The federal higher education system comprises both federal public higher education institutions and all private higher education institutions in Brazil.

In higher education, MEC is currently responsible for establishing, funding, steering and regulating the federal Higher Education system, through its different secretariats (Figure 3.3), which, at the time of writing, are:

The Secretariat for Higher Education (Secretaria de Educação Superior, SESU) establishes, funds, and steers the network of federal universities and is, de facto, responsible for developing the overall strategy of the federal government in higher education.

The Secretariat for Vocational and Technological Education (Secretaria de Educação Professional e Tecnológica, SETEC) co-ordinates, monitors and evaluates vocational education and training in the country, including the (relatively small) network of federal institutes of education, science and technology that provide Advanced Technology Programmes (cursos superiores de tecnologia) alongside non-tertiary professional programmes.

The Secretariat for Regulation and Supervision of Higher Education (Secretaria de Regulação e Supervisão da Educação Superior, SERES) supervises and regulates the system, including the approval of new institutions and programmes. SERES is the main ministry department involved in the Brazilian quality assurance system for undergraduate education and some types of professionally oriented (lato sensu) postgraduate education (the Sistema Nacional de Avaliação da Educação Superior, SINAES).

Figure 3.3. Current governance arrangements for the federal higher education system in Brazil (November 2018)

Note: This figure does not provide a complete overview of the education system in Brazil, nor does it include the different state and municipal level governance systems.

Source: (MEC, 2018[14]), Estrutura Organizacional - Ministério da Educação (Organisational Structure - Ministry of Education), http://portal.mec.gov.br/institucional/estrutura-organizacional

MEC is supported in the regulation, evaluation and supervision of higher education by two main advisory bodies and two specialised implementation agencies with varying degrees of autonomy:

The National Council for Education (Conselho Nacional de Educação, CNE) provides advice for the development and assessment of national educational policy, including the National Plan for Education (Plano Nacional de Educação, PNE). The CNE is composed of 24 members nominated by the President for a four-year mandate. The CNE is composed of two chambers: one for basic education (Câmara de Educação Básica) and one for higher education (Câmara de Educação Superior).

The National Commission for Evaluation of Higher Education (Comissão Nacional de Avaliação da Educação Superior, CONAES), established in 2004, is an advisory body responsible for overseeing the implementation and further development of the national system of evaluation of higher education, SINAES. CONAES is tasked with assessing the mechanisms of institutional, programme and student evaluation, developing proposals for the development of higher education institutions, defining curriculum parameters for each field, articulating with the state education systems, among others.

The Anísio Teixeira National Institute for Educational Studies and Research (Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira, INEP), established in 1937, is a semi-autonomous agency responsible for implementing SINAES. It coordinates the collection of data and is tasked with developing and implementing key elements of quality assessment in higher education, including the National Examination of Student Performance (Exame Nacional de Desempenho de Estudantes, ENADE), programme evaluation and institutional evaluation.

The Foundation for the Coordination of Improvement of Higher Education Personnel (Coordenação de Aperfeiçoamento de Pessoal de Nível Superior, CAPES), established in 1951, is a public foundation under the responsibility of MEC. CAPES has been responsible for assuring the quality of academic (stricto sensu) postgraduate programmes since the mid-1970s. CAPES operates an extensive system of accreditation and quality rating of postgraduate provision based primarily on peer review. In addition, the Foundation provides public funding to research and promotes international scientific co-operation.

The Ministry of Science, Technology, Innovation and Communication (Ministério de Ciência, Tecnologia, Inovações e Comunicações, MCTIC) provides funding for research and innovation in higher education institutions, primarily through its associated executive agency, the National Council for Research and Development (Conselho Nacional de Desenvolvimento Científico e Tecnológico, CNPq).

Funding higher education

A hybrid higher education system, with significant private provision

The Brazilian higher education sector is classified as a “hybrid system” (Ferreyra et al., 2017[15]), with significant public and private higher education sectors and differences between these in terms of funding. As established in the 1988 Constitution, public institutions may not charge tuition fees to students and rely almost exclusively on public funds for their operation. As in other countries, there are opportunities for public institutions to obtain resources from other funding streams (Corbucci and Fonseca Marques, 2003[16]), through donations and by providing services. However, consolidated recent data on these funding streams are not available for public institutions in Brazil.

Institutions in the private sector receive no direct institutional subsidies and depend on income from student fees. Private institutions are free to determine the level of fees charged (Dwyer et al., 2016[8]). In 2017, students in the private sector paid an average monthly fee of BRL 898 (EUR 281) (SEMESP, 2017[17]), although this figure masks considerable variation between institutions and programmes.

Private institutions may benefit from indirect public subsidies, through public grant and loan programmes provided to low-income students attending private institutions (FIES and PROUNI – see below). Students in public institutions do not have access to publicly funded maintenance grant and loan schemes. Private not-for-profit institutions may also benefit from indirect tax and social security exemptions, which were estimated to amount to BRL 9 billion in 2013 (Davies, 2017[18]).

Public spending on higher education is slightly below the OECD average

Total public expenditure in Brazil on educational institutions from primary to tertiary level represents 5% of the country’s GDP. This comparatively high level of public expenditure mostly reflects comparatively high spending on primary to post-secondary non-tertiary education (4% of GDP) (OECD, 2018[3]). However, these data do not take into account private and public spending on student support schemes (as opposed to institutional subsidies), which through assisting students to pay fees, form an indirect subsidy to private higher education institutions. If this expenditure were accounted for, studies suggest that expenditure on tertiary education institutions in Brazil as a percentage of GDP would be higher than the average in OECD countries (Nascimento and Verhine, 2017[19]).

Figure 3.4. Total expenditure on educational institutions as a percentage of GDP (2015)

From public, private and international sources, % of GDP.

Note: Data for Brazil take into account public expenditure to public institutions only (federal, state and municipal-level institutions). Private spending and public spending on student support schemes (as opposed to institutional subsidies), which through assisting students to pay fees, form an indirect subsidy to private higher education institutions are not accounted for.

Source: (OECD, 2018[3]), Education at a Glance 2018: OECD Indicators, OECD Publishing, Paris, http://dx.doi.org/10.1787/eag-2018-en

In 2015, public subsidy per student to public tertiary education institutions in Brazil, including research and development (R&D) activities and adjusted for purchasing power parity (PPP), was USD 14 261. This is close to the level of public subsidy to public institutions in OECD countries such as France (USD 14 386) and the Netherlands (USD 14 369). These data refer exclusively to public spending per student in public institutions, as private institutions do not receive direct public subsidies and data on private spending on public higher education in Brazil are not available. The figures thus reflect spending levels that affect less than 25% of Brazilian students.

Figure 3.5. General government and private expenditure per full-time tertiary student in

In equivalent USD converted using PPPs for GDP, by level of education, based on full-time equivalents

Note: * Data for the United Kingdom refer to institutions that formally have private, not-for-profit, legal status, but which have historically been government-dependent and are considered to be public institutions in national policy documents. Data on private expenditure on public higher education institutions are not available for Brazil, although in the absence of tuition fees, private revenues are likely to be low.

Source: (OECD, 2018[3]), Education at a Glance 2018: OECD Indicators, OECD Publishing, Paris, http://dx.doi.org/10.1787/eag-2018-en

Expenditure per tertiary student declined at the beginning of the current decade and in 2014 was equivalent to 80% of the total value for 2010. In comparison, expenditure per student at primary, secondary and post-secondary non-tertiary level had increased by 58 percentage points since 2010 (OECD, 2017[20]).

Public spending in tertiary education is comparatively centralised

While 92% of final funds for pre-primary and school education (after transfers between levels of government) are managed by regional and local governments in Brazil, around 80% of final public funds for higher education are managed by the federal government. The remaining 20% is managed primarily by state governments (OECD, 2018[3]). The significance of the state and municipal public higher education sectors varies considerably between states in Brazil.

The federal Ministry of Education allocates operating budgets to federal higher education institutions on an annual basis, based on historical patterns for current expenditure and on funding formulas for capital expenditure. While federal institutions have a certain degree of autonomy in resource allocation, the Ministry allocates a specific share for current expenditure and another for capital expenditure (Corbucci and Fonseca Marques, 2003[16])

State-level universities receive funding from the state government. In some cases, the resource allocation for each university is tied to the state level budget. In others, such as the State of São Paulo, universities receive a specific share of state-level taxes. State-level institutions tend to have greater autonomy to allocate such resources internally.

Funding higher education is a politicised topic

There is growing controversy regarding the way higher education is funded in Brazil, as students from more advantaged backgrounds are more likely to access highly selective elite public institutions, which are free, whereas those from poorer backgrounds tend to attend fee-paying private institutions (McCowan, 2007[21]). This is widely seen to exacerbate socio-economic inequalities (OECD, 2014[7]). In 2017, the Brazilian Congress blocked a proposal that would have allowed public higher education institutions to charge tuition fees for specialisation and professional (lato sensu) postgraduate programmes (Portal da Câmara dos Deputados, 2017[22]).

In 2016, the Brazilian government approved an expenditure rule that sets a ceiling on federal expenditure for the next 20 years to stabilise public debt. In practice this will freeze future primary expenditure at 2016 levels, adjusting to inflation (OECD, 2018[5]). Previously the Constitution defined a minimum percentage of revenue to the education and health sectors – 18% of net tax revenue and 15% of net current revenue, respectively. The new rule also protects expenditure to these sectors, but alters the mechanism for doing so, by defining spending floors for education and health, whereby expenditure for these sectors should be no lower than their 2017 expenditure ceiling (IMF, 2017[23]).

Key federal government policies for higher education

Strategy focused on expansion and quality

In recent years, the federal government policy has focused on increasing access to higher education, in particular for disadvantaged socio-economic groups, as well as promoting improvements in quality. In 2014, the federal government adopted the National Plan for Education (PNE) for 2014-24, which sets 20 goals for improving access to education and quality from early childhood to adult education. Three of the goals relate explicitly to higher education (Box 3.1). INEP is tasked with monitoring implementation of the PNE and assessing results at the federal level. State and municipal level governments are responsible for the implementation and monitoring PNE in their jurisdictions.

Box 3.1. Higher education goals in the National Education Plan 2014-2024

Target 12: by 2024, raise gross enrolment in higher education to 50% and net enrolment¹ to 33% of the 18-24 year-old population, ensuring the quality of provision and with at least 40% of new students enrolling in public institutions. Related measures include: infrastructure improvement, increasing the number of study places, raising completion rates, offering at least a third of classes in the evening, expanding student support systems, and adopting affirmative policies. In 2017, the gross enrolment rate was 35% and the net rate was 23%.

Target 13: by 2024, raise the quality of higher education and increase the qualification level of the entire teaching workforce in public and private institutions. Ensure that at least 75% of teaching staff have attained a bac’s degree and 35% a doctoral degree. Relevant measures include: improving the National System of Higher Education Evaluation (SINAES), encouraging institutional self-assessment, expanding the National Examination of Student Performance (ENADE) and ensuring that by 2024, three-quarters of students correctly answer at least 75% of the items on the exam. In 2017, 78% of teaching staff had Master’s degrees and 40% a PhD.

Target 14: by 2024, increase participation in stricto sensu graduate programmes. The goal is to award 60 000 Master’s and 25 000 doctoral degrees per year. Relevant measures include: expanding financial support to stricto sensu graduate programmes, articulating CAPES and state-level R&D support agencies and expanding the offer of stricto sensu graduate programmes. In 2016, 59 600 stricto sensu Master’s degrees were awarded and 20 600 PhDs.

Sources: (Presidência da República, 2014[24])Lei No.13005 - Aprova o Plano Nacional de Educação (PNE) (Law No. 13005 - Approval of the National Education Plan), http://www.planalto.gov.br/CCIVIL_03/_Ato2011-2014/2014/Lei/L13005.htm, (INEP, 2018[25])Relatório do 2º ciclo de monitoramento das metas do plano nacional de educação (Report on the second cycle of monitoring of the goals of the National Education Plan).

Modest expansion of public institutions and student support programmes for students in the private sector

Over the last two decade, federal higher education policy has been characterised by investment in expanding and improving federal higher education institutions and student aid programmes aimed at helping students from low-income backgrounds to access courses in the private sector.

The Programme for Restructuring and Expansion of Federal Universities (Programa de Apoio a Planos de Reestruturação e Expansão das Universidades Federais, REUNI), launched in 2007, provided funds to create additional study places and increase completion rates in federal institutions (Presidência da República, 2007[26]). In parallel, the “University for All” Programme (Programa Universidade para Todos, ProUni), created in 2005, provides tax exemptions to private institutions that offer free or reduced tuition fees to students from low-income families or who attended public upper secondary education. The Higher Education Student Funding Programme (Financiamento Estudantil no Ensino Superior, FIES), created in 1999, is a public student loan system which offers subsidised interest rates and comparatively generous repayment terms to students at private institutions. A reduction in interest rates and an extended reimbursement timeline introduced in 2011 led to a significant increase in demand for loans – from 34 654 in 2009 to more than 700 000 in 2014 (Salto, 2018[6]). In total, FIES and ProUni covered approximately 22 percent of all private HEI students in 2014, of whom 50% identified as black (Zalaf Caseiro, 2016[27]).

However, a report by the Federal Court of Accounts (Tribunal de Contas da União, TCU) showed that FIES’ expansion was conducted without adequate planning and that did not lead to a significant expansion of net enrolment. Given high default rates and the 2015 economic recession, the Brazilian government has since established a cap of 250 000 loans/year and stricter conditions (TCU, 2016[28]).

Quotas in federal universities aim to mitigate social inequities

With the aim of tackling social inequities in the student body in public institutions, the Brazilian government introduced the Quota Law (Lei das Cotas), approved in 2012 – an affirmative action law which imposes quotas on federal universities for the recruitment of students from disadvantaged backgrounds. By 2016, 50% of enrolments were reserved for students from public secondary schools (25% for students with a per capita family income below one and a half minimum wages). Institutions are also be expected to respect minimum proportions of ethnic minority students, based on census statistics for the region where they are located (OECD, 2014[7]).

There have been moves to standardise admissions procedures

In 2010, the federal authorities introduced the Unified Selection System (Sistema Unificado de Seleção, SISU), with the objective of streamlining access processes for higher education. In the past, HEIs offered their own entrance examination (known as the vestibular) and the National Examination of Upper Secondary Education (Exame Nacional do Ensino Médio, ENEM) acted as a secondary school-leaving examination. Improvements to the ENEM content and administration, as well as a decision by the Ministry to use the ENEM as the only admission criteria for entry into federal universities (2009) have led to an increasing proportion of public and private HEIs using ENEM results to select students at entry. Institutions retain the autonomy to choose how to use students’ ENEM results. They might be the exclusive criteria or used as a complement to individual entry examinations. Programmes may also determine their own minimum grade requirements for overall ENEM results or for specific subjects (MEC, 2017[29]).

3.4. Higher education provision in Brazil

A diversified institutional landscape

A legal distinction between types of higher education institution

Public and private higher education institutions (HEIs) in Brazil are formally classified into three categories:

Colleges (faculdades): smaller, teaching institutions often dedicated to a specific field. Colleges currently account for 83% of HEIs in the country.

University centres (centros universitários): comprehensive institutions, mainly dedicated to teaching. University centres offer some post-graduate programmes and may conduct research, but there is no requirement to do so. University centres have greater autonomy in creating new programmes than colleges.

Universities (universidades): comprehensive institutions that are expected to conduct research and to offer post-graduate education. Universities also have autonomy to create new programmes.

Additionally, there are 38 Federal Institutes for Education, Science and Technology (Instituto Federal de Educação, Ciência e Tecnologia) and two Federal Technological Education Centres (Centros Federais de Educação Tecnológica, CEFETs), which are federal public institutions.

A higher education landscape dominated by private providers

The vast majority of new higher education institutions created in the last two decades have been private. In 2016, 87% of Brazil’s 2 407 HEIs were private institutions (MEC, 2018 (unpublished)[2]) the majority of which (88%) were categorised as colleges. These private institutions enrolled over 75% of undergraduate students. For-profit providers – legally authorised in 1988 – play an increasingly important role and are often controlled by large business groups, such as Kroton and Anhanguera, which are publicly traded (de Magalhães Castro, 2015[1]). In 2016, for-profit institutions represented 44% of the total number of institutions and 42% of students enrolled at ungraduated level.

Among the 296 public institutions, there are 107 federal, 123 state and 66 municipal institutions that enrol 15.5%, 7.7% and 1.4% of its students respectively (MEC, 2018 (unpublished)[2]). State and municipal institutions are mostly concentrated in the Southeast and Northeast regions of the country, whereas federal institutions are more evenly spread out across the country (MEC, 2018 (unpublished)[2]).

Table 3.1. Number of tertiary education institutions, by type and sector (2016)
|  | Type
| TOTAL | College | University centre | University | IF & CEFET
Total | 2 407 | 2 004 | 166 | 197 | 40
Public | 296 | 138 | 10 | 108 | 40
Federal | 107 | 4 | - | 63 | 40
State | 123 | 83 | 1 | 39 | n.a.
Municipal | 66 | 51 | 9 | 6 | n.a.
Private | 2 111 | 1 866 | 156 | 89 | n.a.
For profit | 1 052 | 978 | 54 | 20 | n.a.
Not for profit | 1059 | 888 | 102 | 69 | n.a.
Note: IF = Federal Institutes for Education, Science and Technology; CEFET = Federal Technological Education Centres
Source: (MEC, 2018 (unpublished)[2]), Censo da Educação Superior 2018 (Higher Education Census 2018), Data supplied by MEC on 5 October 2018

Universities and university centres have higher levels of autonomy

The 1988 Constitution grants universities and university centres, whether public or private, higher levels of autonomy than colleges (faculdades), in particular regarding academic and administrative matters. They have the right to establish new undergraduate and lato sensu postgraduate programmes and alter the number of study places in existing programmes without prior authorisation from MEC² and issue and register diplomas for their own programmes. Colleges must rely on accredited universities to register their diplomas. In addition, universities and university centres have the freedom to develop curricula - while following national curriculum guidelines - establish research programmes, allocate resources and expand geographic coverage.

However, public higher education institutions are subject to civil service regulations regarding their teaching workforce. They must follow strict remuneration and hiring procedures as permanent staff have the status of public employees. In addition, public institutions are subject to public sector regulations on purchasing and contracts. This has led many HEIs to establish separate foundations (fundações de apoio), subject to private law, that allow for greater administrative autonomy (Schwartzman, 2003[30]).

Undergraduate and postgraduate programmes

Different types of degrees in undergraduate and postgraduate education

Authorised HEIs may provide all kinds of undergraduate (ISCED 6) and postgraduate degrees (ISCED 7 and 8):

Bachelor’s degree (bacharelado): four-to-five-year degrees that usually lead to legally recognised and regulated professions (e.g. law and medicine).

Teacher’s license degree (licenciatura): four-year degrees that allow graduates to teach in pre-primary, primary and secondary education.

Advanced Technology Programmes (cursos superiores de tecnologia): three-year vocational and professionally-oriented programmes.

Specialisation (especialização): two-year programmes that are more professionally oriented, such as Master’s in Business Administration (MBAs).

Master’s degrees (mestrados): two-year programmes that may be professionally oriented or have a stronger academic focus.

Doctorate degrees (doutorados): four-year programmes with a strong academic and scientific focus.

Seven out of ten undergraduate students are enrolled in a bachelor’s degree (bacharelado). Nearly 20% of students at the undergraduate level are enrolled in a teacher training degree (licenciatura) and around 12% in vocational programmes (cursos superiores de tecnologia).

Table 3.2. Undergraduate enrolment, total and share
Number of students enrolled, by type of .degree and sector. In parenthesis the share of total enrolment (%).
|  | Bachelor's degree (bacharelado) | Teacher training degree (licenciatura) | Advanced technology programmes | Other programmes | Total
Public | Federal | 823,295 (10.2%) | 328,032 (4.1%) | 73,951 (0.9%) | 24,175 (0.3%) | 1,249,453 (15.5%)
| State | 305,990 (3.8%) | 229,781 (2.9%) | 79,726 (1%) | 8,213 (0.1%) | 623,710 (7.7%)
| Municipal | 92,684 (~0%) | 21,301 (0.0%) | 3,323 (~0%) | 27 (~0%) | 117,335 (0.3%)
Private | For profit | 2,249,357 (27.9%) | 593,227 (7.4%) | 543,730 (6.8%) | 1,015 (~0%) | 3,387,329 (42.1%)
| Not for profit | 2,078,410 (25.8%) | 348,153 (4.3%) | 245,499 (3%) | 2,365 (~0%) | 2,674,427 (33.2%)
Total |  | 5,549,736 (68.9%) | 1,520,494 (18.9%) | 946,229 (11.8%) | 35,795 (0.4%) | 8,052,254
Source: (MEC, 2018 (unpublished)[2]), Censo da Educação Superior 2018 (Higher Education Census 2018), Data supplied by MEC on 5 October 2018

Growing postgraduate provision, with idiosyncratic programme classification

In contrast to the situation in most OECD countries, in Brazil, Master’s-level programmes (ISCED 7) are divided into two categories: “stricto sensu” and “lato sensu” programmes. Master’s degrees (mestrado acadêmico) and Professional Master’s degrees (mestrado profissional) are classified as stricto sensu given their strong academic and scientific focus. More professionally oriented, postgraduate “specialisation” programmes, including MBAs, are classified as lato sensu provision (CAPES, 2018[31]). Lato sensu programmes are not regulated individually. HEIs are allowed to offer them if they meet certain criteria related to staff qualifications and programmes, but institutions must inform MEC of their existence. In 2017, 22.5% of Master’s students were enrolled in lato sensu programmes (MEC, 2018[32]).

Brazil has witnessed a significant expansion of postgraduate education in the last two decades. The number of Master’s and doctoral programmes increased more than three-fold between 1996 and 2014 (Figure 3.6). The relative importance of Professional Master’s programmes has also been increasing since the late 1990s. By 2014, Professional Master’s represented 14% of all Master’s programmes offered and 11.4% of Master’s degrees awarded.

Figure 3.6. Number of stricto sensu master’s and doctoral programmes, 1996-2014

Source: (CGEE, 2016[33]), Mestres e doutores 2015 - Estudos da demografia da base técnico-científica brasileira (Master's and doctoral graduates 2015 - a demographic study of the Brasilian technical and science base), Centro de Gestão e Estudos Estratégicos, Brasília, http://www.cgee.org.br.

The publication of Brazilian science and engineering articles increased on average by 11.8% a year between 2003 and 2013 (OECD, 2016[34]). Brazil’s citation impact increased from 0.73 in 2011 to 0.86 in 2016. However, only 6.4% of Brazilian papers were in the world’s top 10% most cited, below China (11%) and South Africa (10.2%) (Clarivate Analytics, 2018[35])

Distance education has been expanding, particularly in the private sector

In the past decade, participation in distance education has expanded significantly from 4.2% of total enrolment in 2006 to 18.6% in 2016. The share of undergraduate students in the private sector that are enrolled in distance programmes, at 22%, is larger than in public federal (5.9%) or state-level institutions (7.3%). Currently 92% of students enrolled in distance undergraduate degrees are in a private institution (MEC, 2018 (unpublished)[2]).

Teaching workforce

According to the 2016 Census of higher education, there were 398 000 higher education teachers in the country, of whom 55% were in private institutions, 30% in federal institutions and the remaining 15% in state and municipal HEIs. On average, the ratio of students per teacher is 17.1 in Brazil, although this varies considerably across institutions and sectors, ranging from 10.7 in Federal faculdades to 21.8 in private universities (INEP, 2016[36])

The qualifications of teaching staff vary between institutional types

As shown in Figure 3.7, over 50% of teaching staff in private and public universities have doctoral degrees, compared to 30% in federal technical institutions, 23% in university centres and 18% in colleges. The prevalence of teachers with doctoral degree is higher in public HEIs (59%) than in private institutions (22%).

Figure 3.7. Share of professors by educational attainment, by sector

Source: (INEP, 2016[36]), Sinopses Estatísticas da Educação Superior – Graduação - INEP (Sinopsis Higher Education Statistics - Undegraduate Education - INEP), http://portal.inep.gov.br/web/guest/sinopses-estatisticas-da-educacao-superior

On average, across all institutional types, half of teaching staff (52%) are hired on a full-time basis, another 27% are hired as part-time staff and 21% are hired by the hour. In public institutions, the share of full-time professors is much larger (85%) than the average in private HEIs (26%), with the highest proportion for full-time permanent staff in federal institutions (where staff are civil servants). Data from Brazilian household surveys suggest the earnings of a median professor in higher education correspond to the 96th earnings percentile, which means that only four percent of workers earn more (Ferreyra et al., 2017[15]).

Table 3.3. Teaching staff by contract, sector (2016)
| Full-time | Partial | Per hour
Public (total) | 85% | 11% | 3%
Federal | 92% | 7% | 0%
State | 76% | 18% | 6%
Municipal | 37% | 29% | 34%
Private (total) | 26% | 40% | 35%
Total | 52% | 27% | 21%
Source: (INEP, 2016[36]), Sinopses Estatísticas da Educação Superior – Graduação - INEP (Sinopsis Higher Education Statistics - Undegraduate Education - INEP), http://portal.inep.gov.br/web/guest/sinopses-estatisticas-da-educacao-superior

3.5. Main trends in participation, equity and outcomes

Participation and attainment

Increasing rates of enrolment and attainment

In 2017, 17% of 25-34 year-olds in Brazil had a tertiary education qualification, compared to 10% in 2007. Nevertheless, as noted earlier, tertiary attainment among young adults (25-34) in Brazil still lags behind the average of OECD countries (43%), and is below all other Latin American countries with available data: Argentina (18%), Chile (30%), Colombia (28%), Costa Rica (28%) and Mexico (23%).

Figure 3.8. Educational attainment of 25-34 year-olds (2017)

Percentage of 25-34 year-olds with a given level of education as the highest level attained

Source: (OECD, 2018[3]), Education at a Glance 2018: OECD Indicators, OECD Publishing, Paris, http://dx.doi.org/10.1787/eag-2018-en

The higher levels of educational attainment are a reflection of increased participation in higher education. The total number of students enrolled in undergraduate programmes in Brazil has increased nearly four-fold in the last two decades, from 1.7 million students in 1995 to roughly six million in 2009 and over eight million students in 2017 (Figure 3.9).

Figure 3.9. Enrolment in undergraduate programmes

Note: Data include enrolment in distance and non-distance education.

Source: (INEP, 2018[37]), Número de Matrículas da Educação Superior Por Categoria Administrativa e Abrangência Geográfica (Enrolment in Higher Education, by region and administrative category), https://inepdata.inep.gov.br/analytics/saw.dll?PortalGo

Strong growth in enrolment in the private sector

Participation in the private sector has not only experienced a significant increase in absolute terms, but also relative to the public sector. In 2017, the private sector represented over 75% of enrolment in undergraduate programmes (INEP, 2018[37])compared to only 58% in 1994. However, the private sector’s weight is not as important in postgraduate education although the number of programmes in private HEIs has expanded significantly in the past two decades (CAPES, 2018[31]). Less than one in three postgraduate students is enrolled in a private institution and this share is even lower for those attending more academically-oriented (stricto sensu) programmes, in particular Master’s (15.4%) and doctoral degrees (11.5%).

Distance education has expanded, driven by the private sector

In absolute terms, enrolment in distance education programmes doubled between 2009 and 2017 and now accounts for over 1.7 million undergraduate students (INEP, 2018[38]). However, a more fine-grained analysis shows that expansion has occurred exclusively in the private sector. Since 2012, there has been a decline in the number of undergraduate students enrolled in public distance education programmes (Figure 3.10). Nearly seven in ten students enrolled in distance programmes were attending for-profit institutions in 2014 (Salto, 2018[6]). Tuition fees in distance education programmes are on average considerably lower than in classroom-based (presencial) programmes – BRL 279 compared to BRL 779 – and, in contrast to classroom-based programmes, fees have declined - by 19.8% between 2012 and 2017 (HOPER, 2017[39]).

Figure 3.10. Enrolment in distance education programmes (2009-15), by sector

Source: (INEP, 2018[38]), Número de Matrículas - Educação Superior - Por Modalidade de Ensino, Grau Acadêmico e Categoria Administrativa (Enrolment - Higher Education - by sector and category), https://inepdata.inep.gov.br/analytics/saw.dll?PortalGo

Equity 4and the socio-economic profile of the student population

Greater levels of equity but important gaps remain

Access to higher education has become more equitable in recent years. The share of students from families in the bottom income quartile who attend higher education increased from 9.7% in 2003 to 18.8% in 2013 (Nascimento and Verhine, 2017[19]). Conversely, the share of students from the top quartile has decreased by 17.5 percentage points in the same time period.

Despite improvements, as Figure 3.11 shows, individuals from disadvantaged backgrounds are still much less likely to participate in higher education. Moreover, as previously discussed, despite coming from poorer backgrounds, they are less likely to attend public institutions, which do not charge tuition fees. Less than 60% of students enrolled in public institutions graduated from a private upper secondary school, compared to70% of those attending private institutions (INEP, 2018[40]).

Figure 3.11. Share of enrolment in higher education institutions, by income quartile and sector

Note: Q1 refers to the 25% poorest individuals from the total population, whereas Q4 refers to the 25% richest.

Source: (Nascimento and Verhine, 2017[19]), Considerações sobre o investimento público em educação superior no Brasil (Reflections about public investment in higher education in Brazil), Radar: tecnologia, produção e comércio exterior, Instituto de Pesquisa Econômica Aplicada, http://www.ipea.gov.br/portal/images/stories/PDFs/radar/170324_radar_49.pdf

Ethnic gaps in access to higher education have declined

Efforts to reduce inequities in access across ethnic groups, through affirmative action, have led to a significant increase in participation by black and mixed race students. The number of mixed race (pardo) graduates from federal universities increased from 28.3% to 37.7% between 2004 and 2014. The share of black graduates also increased in that time period from 5.9% to 9.8% (INEP, 2016[36]).

Distance education programmes attract more mature students

There is a large share of mature students (over the age of 30) enrolled in higher education. This share is more than twice as large in distance programmes (Figure 3.12). Whereas over 50% of students in on-campus programmes are aged 19-24.

Figure 3.12. Share of students in higher education by mode of study and age (2015)

Source: (SEMESP, 2017[17]), Mapa do Ensino Superior no Brasil 2017 (Map of Higher Education in Brazil 2017), Sindicato das Mantenedoras de Ensino Superior

Learning and employment outcomes

Tertiary graduates benefit from better employment prospects

In part due to the small share of tertiary-educated people in the population, those who do obtain a tertiary degree in Brazil can expect a notably higher earnings advantage than on average across OECD countries. Someone with a bachelor’s degree in Brazil earns over 2.4 times what someone who only attained upper secondary education earns (OECD average: 1.5) and someone with a Master’s, doctorate or equivalent earns almost 4.5 times more (OECD average, 2.0). These very large earnings premiums are common in Latin American countries with available data (Chile, Colombia, Costa Rica and Mexico) (OECD, 2017[20]).

Figure 3.13. Relative earnings of adults, by educational attainment (2016)

25-64 year-olds with income from employment; upper secondary education = 100

Note: 1. Year of reference 2015. 2. Index 100 refers to the combined ISCED levels 3 and 4 of the educational attainment levels in the ISCED 2011 classification. 3. Year of reference 2014. 4. Earnings net of income tax. 5. Year of reference 2012

Source: (OECD, 2018[3]), Education at a Glance 2018: OECD Indicators, OECD Publishing, Paris, http://dx.doi.org/10.1787/eag-2018-en

As in most OECD and partner countries, those with a tertiary degree in Brazil have better employment rates overall. At 6.5% in 2016, the unemployment rate for tertiary-educated adults in Brazil was about over four percentage points lower than for those who attained only upper secondary education (10.9%), and the inactivity rate was 50% lower (8% compared to 16% respectively). Again, these differences are much larger in Brazil than on average in OECD countries (OECD, 2017[20]). However, there is some evidence of a mismatch between the supply of graduates and the skills required in the labour market. Nearly 70% of managers in Brazil reported difficulty in filling positions, a larger share than in Argentina (41%), Costa Rica (40%) and Mexico (38%). According to the survey, the main reasons reported are lack of technical skills, lack of professional experience and insufficient number of applicants (McKinsey Global Institute, 2018[41])

References

[31] CAPES (2018), GEOCAPES - Sistema de Informações Georreferenciadas (GEOCAPES - Georeferenced Information System), CAPES, https://geocapes.capes.gov.br/geocapes/ (accessed on 01 August 2018).

[33] CGEE (2016), Mestres e doutores 2015 - Estudos da demografia da base técnico-científica brasileira (Master's and doctoral graduates 2015 - a demographic study of the Brasilian technical and science base), Centro de Gestão e Estudos Estratégicos, Brasília, http://www.cgee.org.br. (accessed on 25 September 2018).

[35] Clarivate Analytics (2018), Research in Brazil A report for CAPES by Clarivate Analytics, https://www.capes.gov.br/images/stories/download/diversos/17012018-CAPES-InCitesReport-Final.pdf (accessed on 01 August 2018).

[16] Corbucci, P. and P. Fonseca Marques (2003), Fontes de Financiamento das Instituições Federais de Ensino Superior: um Estudo sobre a Universidade de Brasília (Funding sources in federal higher education institutions: a study on the University of Brasília), IPEA, Brasília.

[18] Davies, N. (2017), “O Financiamento Públic às Instituições Privadas de Ensino (Public funding to private educational institutions)”, Revista Agenda Social, Vol. 10/1.

[1] de Magalhães Castro, M. (2015), “Higher Education Policies in Brazil: A Case of Failure in Market Regulation”, in Schwartzman, Simon Pinheiro, Rómulo Pillay, P. (ed.), Higher Education in the BRICS Countries - Investigating the Pact between Higher Education and Society, Springer, Dordrecht, http://dx.doi.org/10.1007/978-94-017-9570-8_14.

[8] Dwyer, T. et al. (2016), Jovens universitários em um mundo em transformação : uma pesquisa sino-brasileira (Young university graduates in a transforming world: a Chinese-Brazilian study), Ipea, Brasília, http://www.ipea.gov.br/agencia/images/stories/PDFs/livros/livros/160715_livro_jovens_universitarios.pdf (accessed on 03 September 2018).

[15] Ferreyra, M. et al. (2017), At a Crossroads: Higher Education in Latin America and the Caribbean, World Bank, Washington D.C., http://dx.doi.org/10.1596/978-1-4648-1014-5.

[39] HOPER (2017), Webinar HOPER-Presente e Futuro das Pequenas e Médias IES no Brasil (Present and future of small and medium-sized HEIs in Brazil), http://sys.hoper.com.br/webinar/WEBINAR-Mercado%20Educacional-ASBR2017.pdf (accessed on 17 September 2018).

[10] IBGE (2014), Pesquisa Nacional por Amostra de Domicílios - PNAD (National Household Survey), Instituto Brasileiro de Geografia e Estatística, https://www.ibge.gov.br/estatisticas-novoportal/sociais/populacao/9127-pesquisa-nacional-por-amostra-de-domicilios.html?=&t=o-que-e (accessed on 27 August 2018).

[23] IMF (2017), Technical Assistance Report-Supporting Implementation of the Expenditure Rule Through Public Financial Management Reforms, International Monetary Fund. Fiscal Affairs Dept..

[38] INEP (2018), Número de Matrículas - Educação Superior - Por Modalidade de Ensino, Grau Acadêmico e Categoria Administrativa (Enrolment - Higher Education - by sector and category), https://inepdata.inep.gov.br/analytics/saw.dll?PortalGo (accessed on 12 November 2018).

[37] INEP (2018), Número de Matrículas da Educação Superior Por Categoria Administrativa e Abrangência Geográfica (Enrolment in Higher Education, by region and administrative category), https://inepdata.inep.gov.br/analytics/saw.dll?PortalGo.

[25] INEP (2018), Relatório do 2º ciclo de monitoramento das metas do plano nacional de educação (Report on the second cycle of monitoring of the goals of the National Education Plan), INEP, Brasília.

[40] INEP (2018), Sinopses Estatísticas da Educação Superior 2017 – Graduação - INEP (Statistical Sinopsis for Higher Education 2017), http://portal.inep.gov.br/web/guest/sinopses-estatisticas-da-educacao-superior (accessed on 13 November 2018).

[36] INEP (2016), Sinopses Estatísticas da Educação Superior – Graduação - INEP (Sinopsis Higher Education Statistics - Undegraduate Education - INEP), http://portal.inep.gov.br/web/guest/sinopses-estatisticas-da-educacao-superior (accessed on 17 September 2018).

[21] McCowan, T. (2007), “Expansion without equity: An analysis of current policy on access to higher education in Brazil”, Higher Education, Vol. 53/5, pp. 579-598, http://dx.doi.org/10.1007/s10734-005-0097-4.

[41] McKinsey Global Institute (2018), Preparing Brazil for the Future of Work: Jobs, Technology, and Skills - Briefing Note, https://www.mckinsey.com/~/media/McKinsey/Featured%20Insights/Future%20of%20Organizations/Preparing%20Brazil%20for%20the%20future%20of%20work%20Jobs%20technology%20and%20skills/MGI-Future-of-Work-Brazil-Briefing-note.ashx (accessed on 17 September 2018).

[14] MEC (2018), Estrutura Organizacional - Ministério da Educação (Organisational Structure - Ministry of Education), http://portal.mec.gov.br/institucional/estrutura-organizacional (accessed on 31 July 2018).

[32] MEC (2018), Qual a diferença entre pós-graduação lato sensu e stricto sensu? (What is the difference between lato sensu and stricto sensu postgraduate programmes?), Minisério da Educação, http://portal.mec.gov.br/component/content/article?id=13072:qual-a-diferenca-entre-pos-graduacao-lato-sensu-e-stricto-sensu (accessed on 31 July 2018).

[29] MEC (2017), Portaria Nº 468, de 3 de Abril 2017 - dispõe sobre a realização do Exame Nacional do Ensino Médio - Enem, e dá outras providências (Decree Nº 468, 3 April 2017, on the undertaking of the Upper Secondary National Exam - ENEM and other matters), http://www.lex.com.br/legis_27370339_PORTARIA_N_468_DE_3_DE_ABRIL_DE_2017.aspx (accessed on 10 November 2018).

[4] MEC (2014), Planejando a Próxima Década Conhecendo as 20 Metas do Plano Nacional de Educação (Planning the next decade: get to know the 20 goals of the National Education Plan), Ministério da Educação, Brasília, http://pne.mec.gov.br/images/pdf/pne_conhecendo_20_metas.pdf (accessed on 08 November 2018).

[2] MEC (2018 (unpublished)), Censo da Educação Superior 2018 (Higher Education Census 2018), Data supplied by MEC on 5 October 2018.

[19] Nascimento, P. and R. Verhine (2017), “Considerações sobre o investimento público em educação superior no Brasil (Reflections about public investment in higher education in Brazil)”, Radar: tecnologia, produção e comércio exterior, Instituto de Pesquisa Econômica Aplicada, http://www.ipea.gov.br/portal/images/stories/PDFs/radar/170324_radar_49.pdf (accessed on 05 September 2018).

[3] OECD (2018), Education at a Glance 2018: OECD Indicators, OECD Publishing, Paris, http://dx.doi.org/10.1787/eag-2018-en.

[5] OECD (2018), OECD Economic Surveys: Brazil 2018, OECD Publishing, Paris, http://dx.doi.org/10.1787/eco_surveys-bra-2018-en.

[20] OECD (2017), Education at a Glance 2017: OECD Indicators, OECD Publishing, Paris, http://dx.doi.org/10.1787/eag-2017-en.

[12] OECD (2017), How's Life? 2017: Measuring Well-being, OECD Publishing, Paris, http://dx.doi.org/10.1787/how_life-2017-en.

[34] OECD (2016), OECD Science, Technology and Innovation Outlook 2016, OECD Publishing, Paris, http://dx.doi.org/10.1787/sti_in_outlook-2016-en.

[7] OECD (2014), Investing in Youth: Brazil, Investing in Youth, OECD Publishing, Paris, http://dx.doi.org/10.1787/9789264208988-en.

[9] OECD Data (2018), Income inequality, https://data.oecd.org/inequality/income-inequality.htm.

[22] Portal da Câmara dos Deputados (2017), “Câmara rejeita PEC que autorizava universidade pública a cobrar por curso lato sensu (The House rejects proposal for constitutional amendment that would authorise public universities to charge for latu sensu degrees)”, http://www2.camara.leg.br/camaranoticias/noticias/EDUCACAO-E-CULTURA/528130-CAMARA-REJEITA-PEC-QUE-AUTORIZAVA-UNIVERSIDADE-PUBLICA-A-COBRAR-POR-CURSO-LATO-SENSU.html (accessed on 01 August 2018).

[24] Presidência da República (2014), Lei No.13005 - Aprova o Plano Nacional de Educação (PNE) (Law No. 13005 - Approval of the National Education Plan), http://www.planalto.gov.br/CCIVIL_03/_Ato2011-2014/2014/Lei/L13005.htm (accessed on 04 September 2018).

[26] Presidência da República (2007), Decreto nº 6096 - Institui o Programa de Apoio a Planos de Reestruturação e Expansão das Universidades Federais - REUNI. (Decree No 6096 creating the Programme for Support to Restructuring and Expansion Plans for Federal Universities), http://www.planalto.gov.br/ccivil_03/_Ato2007-2010/2007/Decreto/D6096.htm (accessed on 08 November 2018).

[6] Salto, D. (2018), “To profit or not to profit: the private higher education sector in Brazil”, Higher Education, Vol. 75/5, pp. 809-825, http://dx.doi.org/10.1007/s10734-017-0171-8.

[30] Schwartzman, J. (2003), O Financiamento das Instituições de Ensino Superior no Brasil (Funding Higher Education Institutions in Brazil), IESALC - UNESCO, http://www.iesalc.unesco.org.veIES/2003/ED/PI/15 (accessed on 17 September 2018).

[17] SEMESP (2017), Mapa do Ensino Superior no Brasil 2017 (Map of Higher Education in Brazil 2017), Sindicato das Mantenedoras de Ensino Superior.

[28] TCU (2016), Auditoria Operacional no Fundo de Financiamento Estudantil (FIES) (Operational Audit of the Student Financing Fund (FIES)), https://portal.tcu.gov.br/biblioteca-digital/auditoria-operacional-no-fundo-de-financiamento-estudantil-fies.htm.

[11] Transparency International (2018), Corruption Perceptions Index 2017 - Transparency International, https://www.transparency.org/news/feature/corruption_perceptions_index_2017 (accessed on 01 August 2018).

[13] World Bank (2006), Brazil Crime, Violence and Economic Development in Brazil - Elements for Effective Public Policy, Report No. 36525-BR, http://documents.worldbank.org/curated/en/104001468231894133/pdf/365250BR.pdf (accessed on 27 August 2018).

[27] Zalaf Caseiro, L. (2016), Desigualdade de acesso à educação superior no Brasil e o Plano Nacional de Educação (Inequality in access to higher education in Brazil and the National Education Plan), INEP, Brasília.

Notes

 1.  Gross enrolment rate is calculated reflecting all students, regardless of age. Net enrolment is calculated by dividing the number of students of a particular age group enrolled in all levels of education by the size of the population of that age group.

 2.  New programmes and increases in the number of study places in existing programmes in the fields of law, medicine, dentistry, psychology and nursing are subject to prior authorisation by MEC for all institution types. For law programmes, MEC consults the Federal Council of the Brazilian Bar Association (Ordem dos Advogados do Brasil) and for programmes in medicine, dentistry, psychology and nursing, the National Health Council (Conselho Nacional de Saúde) (Presidência da República, 2017, p. art 41[2]).

4.

Market entry: New institutions and undergraduate programmes

This chapter focuses on the processes in place in Brazil to regulate the establishment of new private higher education institutions and of new undergraduate programmes in new and existing institutions in the federal higher education system. The establishment of new private institutions and the creation of new undergraduate programmes in all types of institution require higher education providers to seek regulatory approval from the Ministry of Education (MEC). This approval depends on compliance with administrative procedures and a positive outcome from external peer reviews designed to assess the quality of proposed new institutions and new and recently established programmes. The chapter analyses these regulatory and evaluation processes, highlighting strengths and weaknesses, and provides recommendations for their improvement

4.1. Focus of this chapter

This chapter focuses on the processes in place in Brazil to regulate the establishment of new higher education institutions (HEIs), and of new undergraduate programmes in both new and existing institutions in the federal higher education system.

Specific processes exist for institutional accreditation for private institutions…

Public HEIs in Brazil are created by federal, state or municipal governments, through legislation that automatically conveys authorisation to operate for the institution in question. As explained earlier in this report, once established, public institutions created and funded by the federal authorities form part of the federal higher education system and are legally subject to regulation and supervision by the Ministry of Education (MEC) and institutional and programme evaluation by INEP as part of the National System of Higher Education Evaluation (SINAES). Public institutions created and funded by state or municipal governments (which include some large institutions such as the University of São Paulo (USP) or the University of Campinas (UNICAMP) in the State of São Paulo) are regulated by state governments and are not obligated to follow the rules of SINAES.

All private higher education providers are considered part of the federal system and are required to obtain formal external accreditation (credenciamento) from MEC to allow them to begin operation.

HEIs in Brazil are legally classified as colleges (faculdades), university centres or universities. In broad terms, colleges are undergraduate teaching institutions; university centres are required to have more extensive undergraduate provision, more permanent staff and developed outreach activities; and universities, in addition to meeting these criteria, must have research activity and provide postgraduate education. Public HEIs created by federal, state, or municipal governments may be established in any of these three legal forms from the outset. In contrast, all new private institutions must first be established as colleges and may subsequently transition to the status of university centre or university if they meet the relevant criteria and successfully undergo a new external re-accreditation process. The specific criteria that private HEIs must meet to qualify for each institutional status (organização acadêmica) are set out in articles 16 and 17 of a 2017 decree governing quality assurance processes in the federal higher education system (Presidência da República, 2017[1]).

…and approval of new programmes

As a general rule, colleges in the federal higher education system, which are almost exclusively private¹, are required to obtain formal authorisation from MEC (autorização) to start new programmes. All applications for the establishment of a new higher education institution must be accompanied by at least one (and up to five) application(s) to create a new undergraduate programme. Existing colleges must submit applications for authorisation for each new programme they wish to create. As discussed below, the complexity of the procedures followed to obtain authorisation varies depending on the institutional quality score held by the college submitting the request for authorisation and field of study of the proposed programme. University centres (all of which are private) and universities² have a greater degree of autonomy and are not generally required to obtain authorisation in advance to start new programmes, but must notify MEC of the creation of all new programmes. Universities and university centres do require prior authorisation to start new programmes in medical fields and law (see below).

All HEIs in the federal system, whatever their legal form, are required to submit new programmes to an external quality assurance process called “recognition” (reconhecimento), once half of total teaching hours have been completed (in the second or third year, for example). All programmes offered by HEIs need to complete the recognition process successfully for the degrees they award to be valid in Brazil. As a result, reconhecimento is effectively part of the initial approval process for programmes. Failure to obtain recognition would mean that the diplomas obtained by graduates would not be valid nationally and may thus lead to the closure of the programme.

The approval of new institutions and programmes forms a key part of the wider regulatory work of Secretariat for Regulation and Supervision of Higher Education (SERES) in the Ministry of Education (MEC), the full scope of which is illustrated in Table 4.1 below.

Table 4.1. Regulatory acts by SERES in 2017
Regulatory Acts | Number
Regulatory acts relating to HEIs |
Credenciamento (Accreditation) | 208
Credenciamento EaD (Accreditation, Distance Provider) | 70
Recredenciamento EaD (Re-accreditation, Distance Provider) | 36
Transferências de Mantença (Transfer of Ownership) | 273
Regulatory acts relating to programmes |
Autorização não-vinculada (Authorisation, not linked to accreditation) | 1361
Reconhecimento (Programme Recognition) | 1686
Renovação de Reconhecimento (Renewal of Programme Recognition) | 6781
Autorização EaD (Authorisation of New Programme, Distance Education) | 225
Reconhecimento EaD (Renewal of programme recognition, distance education) | 266
Renovação de Reconhecimento EaD (Renewal of Programme Recognition, Distance Education) | 188
Aditamento de polos (Addition of Distance Education Sites) | 88
Aumento de vagas (Authorisation for increase in vagas) | 267
Other regulatory acts |
Aditamento (Modification of existing regulatory acts) | 645
Chamamento Público (Public call) | 2
Total Acts | 12 096
Source: SERES, “Regulação e Supervisão da Educação Superior” p.17, December 2017, presentation to OECD Review Team.

4.2. Strengths and weaknesses of the current system

This section analyses the strengths and weaknesses of the current systems of regulation and evaluation for the accreditation of new private providers and the authorisation and recognition of new undergraduate programmes.

Establishment of new institutions: accreditation

Relevance: the rationale and objectives of the current system

In all systems of external quality assurance, processes of “market entry” by new higher education institutions must take care to balance quality and quantity. Entry requirements must be sufficiently restrictive to prevent bad programmes, such as those in which large numbers of students are unable to complete their studies, or complete their studies but acquire few skills and have poor prospects for employment. At the same time, these requirements must not create barriers to entry that unnecessarily limit supply and leave students without access to beneficial study opportunities. In addition, badly designed arrangements for the establishment of new institutions may require new institutions to adopt governance, management, or staffing arrangements that hamper innovation and efficiency.

Brazil’s system of accreditation (credenciamento) for new private HEIs exists to ensure that new institutions meet minimum standards of quality in providing higher education programmes to undergraduate students. The arrangements for the establishment of new private HEIs, which must always be coupled with the creation of programmes, strike a balance between ex ante and near-term ex post review to assure quality. Like other higher education systems, Brazil applies a lower level of scrutiny to institutions established by public authorities – in practice, institutions established by federal, state and municipal governments. Public institutions are not subject to the same accreditation requirements as private institutions, may be created as universities or university centres, and are exempt from any other forms of ex ante review. Public law authorises heightened control over the creation of private provision, permitting newly established private institutions to take only the form of “college” (faculdade), rather than fully developed university institutions.

Effectiveness: quality indicators used and division of responsibilities

MEC necessarily carries out accreditation of new private institutions based on planned provision rather than functioning programmes and institutions, and thus must focus on the planned inputs and processes, rather than observed processes, outputs or outcomes. New private higher education institutions begin the process of accreditation when a legal representative of the maintaining institution submits to SERES documents outlining various dimensions of the planned institution and a self-evaluation of its expected performance. The information submitted (Presidência da República, 2017, pp. art. 20-21[1]) includes:

1. Information about the legal, financial and tax status of the operating organisation (mantenedora).

2. An Institutional Development Plan (PDI), containing:

i. A description of the institutional mission, goals and profile,

ii. An institutional pedagogical programme describing educational, research and outreach policies,

iii. Details of the number and nature of planned study programmes, including the number of study places (vagas),

iv. Details of (planned) teaching staff,

v. Planned digital document and records management systems,

vi. Details of physical infrastructure, including library, laboratories etc., where relevant,

vii. Details of financial capacity and a financial sustainability plan,

viii. Where relevant, specific information relating to distance education.

3. Internal statutes.

4. Information about the proposed management staff, including their academic experience.

5. Proof that suitable premises and facilities (imóvel) are available, including technical appraisal of their accessibility for students with special needs and compliance with fire and safety regulations.

Institutional accreditation is linked to a review of proposed programmes. Private institutions seeking initial accreditation have up to three years following approval of their initial institutional plan to submit and obtain authorisation of between one and five programmes (autorização vinculada a credenciamento). Additionally, campuses that are located outside of the municipality in which a private institution has its seat of operations must undergo a separate process of accreditation.

Following review and approval by SERES of documents submitted to it through the e-MEC online platform, a process of peer review is undertaken. Three external evaluators drawn from INEP’s BASis database undertake an on-site inspection and review visit, evaluating the conditions for the proposed institution. This institutional review may be combined with review of up to two programmes (see below) by the same review commission (Presidência da República, 2017, p. art.19[1]).

For the institutional review, the evaluation commission uses specific evaluation criteria and scoring detailed in an evaluation template (“instrument”) for accreditation developed by INEP (INEP, 2017[2]). The evaluation process implemented by peer reviewers is organised around five axes, and assesses the proposed institution against 45 qualitative indicators, each of which is evaluated on a five-point scale. Some of the indicators apply only to distance education and others only to physical campuses, meaning a given institution is only ever assessed against a maximum of 41 indicators.

Of the 45 indicators in total, 30 could best be classified as planned inputs to institutional management and the educational process (general institutional policies, staffing, infrastructure and equipment) and another 15 as planned processes (policies for more specific institutional processes and proposed activities). For obvious reasons, real outputs cannot be considered in this form of ex-ante evaluation. The principal foci of institutional accreditation are the Institutional Development Plan (PDI) (30%), planned academic policies (20%), planned management policies (20%) and infrastructure (20%). The final score generated by this evaluation, on a scale of one to five, is referred to as the “institutional score” or Conceito Institucional (CI). Institutions need a score of at least three to receive accreditation from SERES.

Table 4.2. Indicators used for on-site inspections for institutional accreditation
Axis | Number of indicators | Weight
Planning and institutional evaluation | 3 | 10
Institutional development | 7* | 30
Academic policies | 10 | 20
Management policies | 7 | 20
Infrastructure | 18** | 20
Total | 45 | 100
Note: * 2 indicators apply only to distance education institutions; ** 4 indicators apply only to distance education institutions and 4 only to campus-based institutions.
Source: OECD calculations based INEP (2017) External institutional evaluation instrument - classroom-based and distance – accreditation, adopted by INEP in October 2017 (INEP, 2017[2]).

According to SERES, 829 institutional accreditation reviews were performed in the period 2015-2017, and on average two to three courses associated with each institutional proposal were reviewed (2 013 courses in total). SERES promulgated 208 acts of accreditation in 2017. It has not reported the number of accreditation proposals that it modified or rejected.

Institutional accreditation for private colleges is valid for a period of three to five years; depending on the CI score (three to five) they receive. After this period, colleges must undergo a process of re-accreditation (recredenciamento), which formally applies to all types of private and public institution, and which we discuss in Chapter 7.

Effectiveness and efficiency: use and effects of the current system

The cost of institutional accreditation falls exclusively on private higher education institutions, since public institutions are exempt from its requirements. Private institutions bear the direct financial costs of complying with accreditation requirements, including costs for staff and renting premises in the period between submission of an accreditation request and approval to begin operations.

Overall, however, the institutional accreditation requirements in place since the adoption of the 2004 legislation on SINAES do not appear to have created excessive barriers to the market entry of private higher education providers in Brazil. As highlighted in Chapter 3, data on enrolment and the number of HEIs show that Brazil’s higher education system has grown swiftly over the last decade, and private sector institutions have provided the majority of new study places. Moreover, higher education leaders with whom the review team met, including representatives of private institutions, did not report that accreditation requirements resulted in unmet demand among prospective students.

Well-designed accreditation of new higher education providers brings benefits to society, protecting students, as consumers, from poor or fraudulent provision. In contrast to some other countries in the Latin America and Caribbean region (IESALC, 2017[3]), compliance with Brazil’s system of institutional accreditation appears to be nearly universal. Private institutions do not frequently operate without institutional accreditation. Moreover, the requirements of institutional accreditation appear to be sufficiently rigorous to limit fraudulent or grossly unqualified private institutions from entering the higher education marketplace.

Nonetheless, there are examples of accredited higher education institutions offering programmes that are not authorised, and organisations that are not accredited higher education institutions offering fraudulent diplomas (Governo do Brasil, 2017[4]). While the Ministry’s e-MEC platform provides a single national registry of accredited institutions and authorised programmes, it is primarily an administrative database. Incidents of allegedly fraudulent provision suggest that not all students have ready access to information that allows them to confirm the validity of the institutions and programmes in which they plan to study. While the layout and functionality of the e-MEC site are not designed to be used by students and their families, the information contained in the system could easily be exploited as part of a more user-friendly information service for students.

There are specific concerns in Brazil – as in other countries – regarding the expansion of distance education, including HEIs that only provide distance programmes. Distance education now accounts for almost 20% of total enrolment in Brazil, with over 90% provided by the private sector. Private distance education institutions and the programmes they provide are subject to the same procedures for institutional accreditation (and programme-level authorisation and recognition – see below) as providers of traditional classroom-based higher education. A limited number of qualitative indicators relating specifically to distance education have been incorporated into the evaluation templates used for accreditation, covering pedagogical approaches, digital technologies and infrastructure.

Brazilian legislation requires distance education programmes to respect the requirements of national curriculum guidelines (DCNs), for fields where these exist, and distance programmes have hitherto mostly been blended programmes, with some face-to-face instruction and assessments, often conducted in decentralised distance education learning centres (referred to as “poles”). This pattern might be considered to be positive, as internationally, blended programmes have been shown to be more effective than fully online programmes (Escueta et al., 2017[5]).

However, recent legislative changes have made it easier for private higher education providers to establish large numbers of distance education “poles” (up to 250 a year), in multiple locations, without the need for the facilities in each location to be inspected by INEP evaluators (Presidência da República, 2017[6]; MEC, 2017[7]). Some stakeholders in Brazil are concerned that this will promote the uncontrolled expansion of distance education, without adequate quality guarantees (Estadão, 2018[7]). Furthermore, the specific evaluation criteria for distance education institutions (and programmes) used currently are few in number and underdeveloped in light of the risks associated with this kind of provision (limited staff-student interaction, the risk students are isolated, the challenges of organising fair and rigorous assessments and examinations, etc.).

Creation of new programmes: authorisation and recognition

Relevance: the rationale and objectives of the current system

The principal focus of quality assurance with respect to new provision or “market entry” in Brazil is the offer of new undergraduate study programmes by private HEIs. Recent data show that the majority of new programmes are created in the private sector. Indeed, between 2015 and 2016, the number of registered classroom-based and distance undergraduate programmes in private HEIs increased by 1 092 to 23 824 (a 5% increase), while the number of programmes in federal universities fell by 161 to 4 876 (a 3% fall) (INEP, 2018[6]).

MEC closely supervises the establishment of programmes through processes of ex-ante authorisation (autorização) for a proportion of new programmes in colleges and recognition (reconhecimento) of all new programmes in all types of HEI (in practice, mostly in private HEIs), once they have completed at least half the teaching hours of their first cycle of operation.

In principle, new courses enter the higher education system after careful and integrated scrutiny: on-site reviews at the stage of authorisation (in some cases) and recognition (in all cases) examine the conditions in place that affect the supply of the programme. These initial checks of the supply conditions for programmes remain the primary guarantor of the quality of undergraduate programmes in Brazil. They are later complemented by ongoing monitoring of a small set of indicators, based primarily on the results of assessment of student learning outcomes using the National Examination of Student Performance (ENADE) and periodic “renewal of programme recognition” that follows a three-year cycle. In cases where quality problems are detected through the monitoring indicators, or through other sources, such as complaints, INEP conducts further on-site visits for renewal of recognition. We discuss these later quality checks in Chapter 5.

Authorisation

Colleges must seek authorisation of all new programmes before instruction begins. The standard authorisation process starts with analysis by SERES of documents relating to the new programme submitted by the HEI. These documents include a “Programme Pedagogical Project” (projeto pedagógico do curso, PPC), setting out the programme structure, number of study places (vagas), proposed curriculum, teaching methods and use of technology. In addition, HEIs provide details of proposed teaching staff and proof of available teaching spaces, including, where relevant, decentralised learning centres (“poles”) for distance education (Presidência da República, 2017, p. art.43[1]). Subsequently, in the standard authorisation procedure, an on-site review is organised by INEP. This review is undertaken by external reviewers from the BASis database, using the dedicated evaluation template for authorisation (INEP, 2017[7]), discussed below. The evaluation report and a score, referred to as a Conceito de Curso (CC) and awarded on a scale of one to five, are then transmitted to SERES. Finally, SERES reviews the evaluation report, approves the CC and makes a decision to authorise the programme, if the CC is three or above, or to refuse authorisation.

The authorisation of new programmes proposed by colleges is a risk-adjusted process. Recent changes to the regulatory regime allow colleges to obtain authorisation for new courses under certain circumstances based exclusively on a desk-based analysis by SERES of the programme documents submitted by the HEI, without undergoing an on-site inspection. Colleges with the minimum institutional quality score (CI) of three can start up to three new programmes a year without on-site reviews, provided they already have officially recognised (i.e. quality assured) programmes in the same disciplinary field. Colleges with institutional quality scores of four and five are permitted to create, respectively, up to five and ten new programmes a year under the same lighter regulatory conditions, in fields where they have existing quality assured programmes (MEC, 2017[8]).

Programmes in law, medicine, dentistry, psychology and nursing form a major exception to the general principles guiding authorisation. Programmes in these fields now always require prior authorisation from MEC and an on-site authorisation review, even in university centres and universities (Presidência da República, 2017, p. art.41[1]). To inform its decisions in these fields, MEC takes advice from, respectively, the federal council of the Brazilian Bar Association (Ordem dos Advogados do Brasil, OAB) and the National Health Council (Conselho Nacional de Saúde). Compared to previous legislation, the 2017 decree on quality assurance (Presidência da República, 2017[1]) extended the requirement for systematic authorisation with on-site inspections to nursing programmes and also made increases in the number of study places in all undergraduate law and medicine programmes dependent on MEC authorisation (SEMESP, 2017[9]).

Recognition

All HEIs, including university centres and universities, must seek regulatory “recognition” (reconhecimento) from SERES for every undergraduate programme when a programme’s first cohort of students has completed between 50 and 75 percent of the workload of the course. Recognition is needed for the diplomas issued to graduates of the programme to be valid nationally in Brazil. A separate recognition process is required for each programme offered in campuses outside the municipality where the HEI has its headquarters. Formally, the process of recognition also applies to federal public institutions. In practice, the low levels of programme creation in the federal public sector mean federal public institutions are comparatively rarely involved in processes of recognition.

HEIs are required to submit requests for the recognition of their programmes to SERES, providing the same set as documents as is required for authorisation of programmes (programme pedagogical project etc.). SERES undertakes a desk-based analysis of these documents and INEP organises an on-site review process by external evaluators, who use a separate evaluation instrument to rate programmes and generate a programme quality score (Conceito de Curso, CC) on a scale of one to five (INEP, 2017[10]). For programmes in law, SERES seeks an opinion from the Brazilian Bar Association, and for programmes in Medicine, dentistry, psychology and nursing, from the National Health Council, on the decision to recognise the programme.

If the CC score is at least three and, where relevant, the Bar Association and Health Council issue positive opinions, SERES confirms the CC and issues an official recognition of the programme. For programmes previously subject to authorisation, the CC resulting from the recognition process becomes the new quality score for the programme.

If the result of the on-site evaluation is negative (a score of two or less), SERES requires the HEI to draw up a “Commitment Protocol” (protocolo de compromisso) which sets out how the quality problems detected will be addressed within a 12-month timeframe (Presidência da República, 2017, p. art.54[1]). This stage of the supervisory process is referred to as a “remediation procedure” (procedimento saneador). If it considers there is an immediate risk for students, SERES may also impose one or more sanctions (medidas cautelares), including suspension of new student intakes (see Box 4.1). Internal data transmitted to the OECD team by SERES suggests these kinds of sanction are virtually never applied in remediation procedures.

At the end of the period established by the Commitment Protocol, the programme is subject to another on-site inspection by INEP evaluators. If it still fails to meet minimum quality requirements, SERES launches a “sanctioning procedure” (procedimento sancionador), which may entail the same sanctions mentioned in Box 4.1 (Presidência da República, 2017, p. art.73[1]). For serious cases in private institutions, the relevant legislation allows for the withdrawal of institutional accreditation, which would effectively lead to the closure of the institution. Again, in practice, such cases are rare. Legally, some of the sanctions can be applied to public institutions, but the legal status of these institutions as public bodies means they may not have their institutional accreditation withdrawn.

Box 4.1. Sanctions used by SERES in supervision of (private) higher education

I - suspension of admission of new students;

II - suspension of the offer of undergraduate or lato sensu postgraduate programmes;

III - suspension of the autonomy of the HEI;

IV - suspension of the prerogative to create new distance education poles by the HEI;

V - suspension of regulatory processes that the HEI or other HEIs owned by the same operating organisation have submitted to SERES;

VI – prohibition of filing new regulatory processes to SERES by the HEI or other HEIs owned by the same operating organisation;

VII - suspension of the HEI’s right to enter into new Student Financing agreements as part of the FIES system;

VIII - suspension of the HEI’s right to participate in a selective process for the offer of scholarships from the University for All Programme (PROUNI);

IX - suspension or restriction of the HEI’s right to participate in other federal access programmes.

Source: Article 63 of Decree 9 235 of 15 December 2017 (Presidência da República, 2017[1])

Effectiveness: indicators used for authorisation and recognition of new courses

The on-site evaluation templates (“instruments”) used by external reviewers for the processes of authorisation and recognition were revised by INEP in October 2017. They establish nearly identical review templates. The judgement criteria in the template for authorisation (INEP, 2017[7]) focus on planned inputs and activities, while those in the template for recognition (INEP, 2017[10]) refer to real inputs and activities, verified in practice by the external review commission sent by INEP.

Both templates direct reviewers to evaluate programmes on three dimensions: the proposed pedagogical approach and organisation of the programme (Organização Didático-Pedagógica); the instructional workforce (Corpo Docente e Tutorial); and infrastructure (Infraestrutura). The assessment of the pedagogical approach and organisation of the programme considers the extent to which the planned curriculum meets the requirements of subject-specific National Curriculum Guidelines (Diretrizes Curriculares Nacionais, DCN) approved by the National Education Council, which exist for many, but not all, disciplines in higher education. Together, these dimensions contain more than 50 indicators, some of which apply to all programmes, others specifically to distance education or classroom-based programmes, and still others to programmes offering clinical or field-based learning. Crucially, peer reviewers are responsible for scoring the indicators on a five-point qualitative scale.

The 50+ indicators used in each on-site evaluation template focus either on programme inputs (teaching staff, infrastructure) and processes (pedagogical processes, support to students etc.). Although both instruments assess the expected “profile” specified for the graduates that the programmes educate, they do not consider programme outputs. This choice is necessary because the reviews take place prior to, or midway through, the study programme for the first student cohort. Other than the distinction between planned and real inputs and activities, the principal difference between the two evaluation instruments rests with the weight they assign to the indicators, with the instructional workforce taking on greater weight in the process of recognition (40%) than in authorisation (20%).

The indicators assessed under the “teaching staff” dimension of the evaluation instruments focus primarily on the qualifications of staff, their employment status and the extent to which their profiles match the needs of the programme. The choice to assign greater weight to assessment of these factors during the recognition process than in authorisation reflects the fact that the staff will actually be in place and working at the time of recognition, so the composition of the teaching workforce can be judged more accurately. However, the judgement criteria reward the presence of full-time staff with doctoral degrees and attach little value to professional experience, thus disadvantaging professionally oriented programmes. At the same time, relatively little weight is attached to assessment of the pedagogical and didactic approaches implemented by the programme, despite their crucial role in supporting students to acquire relevant learning outcomes.

The weighted sum of scores provided in the on-site review are used to calculate the programme score, the Conceito de Curso (CC), the value of which ranges from one to five, and provides the basis for SERES to authorise or recognise new programmes, or not.

Table 4.3. Indicators used for authorisation (autorização)
Dimension | Number of indicators | Weight
Pedagogical and didactic organisation of the programme | 24 | 40
Instructional workforce | 14 | 20
Infrastructure | 16 | 40
| 54 |
Source: INEP (2017) Evaluation instrument for undergraduate programmes - classroom-based and distance – authorisation (INEP, 2017[7])

Table 4.4. Indicators used for recognition (reconhecimento)
Dimension | Number of indicators | Weight
Pedagogical and didactic organisation of the programme | 24 | 30
Instructional workforce | 16 | 40
Infrastructure | 18 | 30
| 58 |
Source: INEP (2017) Evaluation instrument for undergraduate programmes - classroom-based and distance – recognition and renewal of recognition (INEP, 2017[10])

The on-site evaluation templates now make special provision for the authorisation and recognition of distance education courses. For example, in evaluating the pedagogical and didactic organisation of the programme, three indicators consider respectively the tutoring approach, tutoring staff and virtual learning environment for distance education programmes (INEP, 2017[7]; INEP, 2017[10]). Under the section on instructional workforce, peer reviewers are also to assess whether programmes have staff with “experience in teaching in distance education” and “experience of tutoring in distance education” (in this context, “tutor” is used to refer to teaching assistants, who support core academic teaching staff in delivering a programme). Infrastructure indicators have, likewise, been augmented to take account of distance education programmes.

However, 45 out of 55 indicators in the templates are applicable to both classroom-based and distance programmes. The specific indicators of programme quality related to curriculum, instruction, learning support, and assessment in distance programmes are less developed than those used in accreditation systems in other OECD and partner countries, including the United States (DEAC, 2018[11]). Developing appropriate measures of quality that reflect the specific characteristics of distance education is, however, a challenge shared by many higher education systems.

Effectiveness and efficiency: use, effects and efficiency of programme authorisation and recognition

What are the principal effects of programme authorisation and recognition for Brazilian higher education? First, authorisation and recognition play a critical role in regulating the enrolment capacity of the nation’s higher education system. Through these processes, private higher education institutions propose the number of study places they believe programmes can properly accommodate in their Programme Pedagogical Project (PPC). The on-site reviews evaluate supporting evidence for this claim³, and their assessment is confirmed by SERES, which officially determines how many study places the programme may have.

The impact of the authorisation and recognition processes on the quality of programmes provided is much more difficult to ascertain. There are no programmes in private institutions in Brazil that have not been subject to recognition that could form a comparison group with which to compare recognised programmes in an effort to analyse the quality effects of the regulatory and evaluation processes. There are no other readily available and comparable indicators of programme quality in Brazil that would provide an alternative means to assess programme quality - and make it possible to judge whether recognition provides an effective guarantee of quality.

The formal requirement for all courses to obtain official recognition in the early stages of their operation provides a basic guarantee of the quality of programmes. The procedures in place force higher education providers to reflect seriously about the design of the programmes they are providing and put in place a range of documents and processes – described in the programme pedagogical project – that should contribute positively to the delivery of relevant and high quality programmes. Nevertheless, the factors verified through the on-site evaluation at the stage of recognition are all conditions for the delivery of quality programmes, but do not provide a guarantee that programmes deliver high quality education in practice.

Moreover, the processes used to evaluate the quality of new programmes are subject to four principal lines of criticism with respect to their reliability, usefulness, and cost effectiveness.

First, representatives of private institutions consulted by the OECD review team complained that external reviewers appointed by INEP to implement on-site reviews very frequently come from public universities, while the vast majority of authorisation and recognition processes occur in private institutions. More generally, institutional representatives argued that those who are called upon to carry out reviews sometimes lack expertise with respect to the programme under review. In the first case, reviewers may bring to private institutions unfamiliarity with their circumstances, or even an active hostility to their institution. In the second, the risk is that reviewers lack sufficient expert knowledge to make sound judgments about the didactic and pedagogical profiles of the programmes they review.

INEP claims that recent improvements to the BASis database of reviewers and the rules governing the allocation of external experts should address both concerns. The ordinance governing on-site evaluation requires that review commissions for different types of programme have direct experience with programmes in the same field and mode of provision (MEC, 2017[12]). Moreover, a recent “administrative instruction” (MEC, 2017[13]) requires that at least one reviewer in institutional reviews in private HEIs has experience in a private HEI. It has not been possible to determine, however, to what extent these rules are applied in practice and, in the latter case, whether experience in the private sector is required for programme-level reviews in private institutions.

Second, there is concern about the subjectivity or unreliability of qualitative assessments. The process of on-site review for programme authorisation and recognition (as revised in late 2017) asks reviewers to make qualitative judgments on a five-point Likert scale, using pre-formulated judgement criteria. Despite the attempts by INEP to formulate the judgement criteria clearly, these scales still leave considerable room for interpretation. They call upon reviewers to make distinctions that that are likely to be inconsistent between individuals. The OECD review team was told by campus officials that the same programme offered in different campuses with otherwise near-identical supply conditions received different marks from on-site reviewers.

Third, the OECD review team heard frequent criticisms from institutional representatives of the delay and burden associated with the on-site review process for authorisation and recognition. Describing the whole system of on-site reviews as it functioned in 2012 (before recent reforms), a document written by the Association of Private Higher Education, ABMES, notes:

The evaluation system is nearing collapse. INEP holds approximately 5 000 assessment visits per year, or about 100 per week. The logistics to support an operation of this size, nationwide, and every day is overwhelming. For example, there are more than 400 flights per week to be scheduled, budgeted, accounted for and issued by INEP. Yet, for a system with nearly 30 000 undergraduate programs and 3 000 institutions, not counting new authorization and accreditation procedures for courses and institutions, 5 000 visits are insufficient. This causes crowding of the evaluation system and a growing backlog. There are higher education institutions with applications for recognition awaiting for years the visits of committees. (de Magalhães Castro, 2015[14])

INEP and SERES argue that the situation has improved since 2012. In particular, they point to the fact that HEIs that have received adequate quality scores (a CI of three or above) are exempted from on-site reviews at the stage of authorisation for programmes in fields where they already have courses (within certain limits). They argue that the most recent regulatory changes in Decree 9 235/2017 (Presidência da República, 2017[1]) reduce burden for institutions with an established quality record, allowing them to create additional study places more easily, for example.

While there has indeed been a shift in the regulatory approach, the market entry process for new undergraduate programmes in the federal higher education system remains administratively burdensome for private HEIs and the evaluation agency (INEP) when compared to equivalent processes in many OECD countries. In Brazil, despite the recent changes, all new programmes are required to go through the recognition process, with on-site reviews that depend on peer review and are logistically complex to organise.

In Anglophone OECD countries and a number of non-Anglophone European higher education systems, HEIs can create programmes and issue valid diplomas without prior authorisation (European University Association, 2018[15])⁴. In these and other systems, authorities often link quality review procedures more closely to risk of poor quality than is the case in the Brazilian system, with less complex procedures in place for institutions that can demonstrate they present a lower risk. Although the large private higher education sector in Brazil creates specific risks, which are not found in all higher education systems, there is certainly scope for Brazil to draw on risk management practice in other quality assurance systems⁵.

Finally, on-site visits carried out in support of programme recognition permit higher education institutions initially to award degrees without providing evidence about the initial performance of the programme, such as rates of attrition among its students. Additionally, the process of recognition does not systematically elicit information from the students who the programmes serve (as the ENADE process does at a later stage), or external stakeholders who have experience of working with the programme and its students, such as public sector employers and private firms which provide internships.

4.3. Key recommendations

1. Improve the reliability and visibility of information about institutions’ accreditation status to ensure students and families are well informed

Although MEC, with the support of evaluations coordinated by INEP, regulates the entry of new institutions into the Brazilian higher education marketplace more comprehensively than in other systems undergoing rapid expansion, the quality assurance system is not fully effective in preventing fraudulent and unauthorised provision. The first line of defence against unaccredited higher education providers is students themselves. Informed students understand which institutions are accredited and not, and why this matters to them, and are able to identify and avoid unaccredited institutions. In principle, comprehensive information about accredited institutions and recognised programmes is available through the online e-MEC. However, e-MEC is not a user-friendly source of accreditation information. More accessible public Internet resources found in other higher education systems could serve as references for the Brazilian authorities in this regard (UK Government, 2018[16]). In the medium-term, the aim should be to develop a comprehensive online portal providing students and prospective students not only with programme-level information on quality assurance results, but also on issues such as graduation rates and graduate employment outcomes (see discussion on programme indicators below).

2. Over time, increase the focus on institutions as units of evaluation in the external quality assurance system to reduce burden, while maintaining effectiveness

We have noted four main concerns about the processes used to authorise and recognise new study programmes. Despite attempts to address concerns about the composition of review commissions and reduce requirements for authorisation in some cases, the Brazilian system of programme review at market entry remains complex and burdensome and may not represent the best use of the country’s resources. There is a need for a system in which the burden and benefit of new programme approval are re-balanced.

Permitting institutions themselves to play a wider role in assuring the quality of programmes, while maintaining an enhanced system of programme level monitoring indicators, could significantly reduce the burden of programme approval through authorisation and recognition. Programme-focused regulatory decisions – for new and existing programmes - account for more than 10 000 of the 12 000 acts that SERES handles annually.

The Brazilian system of quality assurance currently focuses proportionally more efforts on the programme-level than on the institutional level as a unit of evaluation and monitoring. To some extent, the current system regards HEIs as “holding units” for programmes, which are then the main focus of detailed analysis in the quality assurance system. In Chapter 7 we take up the question of institutional quality, and make suggestions for a more rigorous and comprehensive process of institutional reaccreditation, with a view to permitting higher education institutions with demonstrated capacity to assume responsibility for quality of the programmes that they offer and to become “self-accrediting institutions”.

3. In the near term, take steps to improve the evaluation process for programmes that remain subject to programme-level authorisation and recognition

The OECD review team sees a clear case for maintaining programme-level authorisation and strict market entry requirements at programme level for HEIs that lack a strong track record of good quality provision and are not able to demonstrate adequate capacity to self-accredit their own programmes. It is thus important to increase the effectiveness of these processes in promoting quality practices for institutions that remain subject to programme-level authorisation and/or recognition. Priorities for improving current practice in the short-term include:

Further improving the criteria used to select and assign peer reviewers for on-site reviews to increase the fit between reviewer expertise and programme review responsibilities. It is particularly important that reviewers for professionally oriented programmes have adequate understanding of the objectives and operation of such programmes and are able to make robust assessments about the quality of provision in teaching institutions that lack a traditional academic focus and research activities.

Continuing and increasing efforts to improve the training of peer reviewers, with a view to improving the reliability and impartiality of scoring.

Increasing the weight attached to the organisation and implementation of teaching and learning in the evaluation instrument for recognition, reflecting the importance of these factors for students.

In cooperation with international peers, refining and expanding the specific indicators used for the evaluation of distance education programmes, so that these address the particular risks associated with this type of provision. This should consider how best to evaluate decentralised distance education centres (“poles”).

Using the recently introduced process of feedback about the performance of peer reviewers to monitor and revise selection and training.

4. In the longer term, take steps to reduce further the burden and improve the effectiveness of quality assurance processes for programmes outside self-accrediting institutions

In the longer term, two issues should be considered in particular. First, the procedures for on-site visits could be fundamentally reformed. Responsibility for reviewing institutional infrastructure and basic institutional policies could be assigned to a well-trained and professionalised inspectorate. The expert judgment of academic peers (who currently review all aspects of institutions and programmes) could then be applied to a more limited set of indicators than at present, focused on core teaching and learning activities. A sequenced process of accreditation and authorisation could be implemented in which a professional inspectorate initially carried out its work, and academic peers would be engaged only for institutions and programmes that have passed a first stage of review. Second, it will be important to identify ways in which the more extensive, quantitative, and comparable information about intermediate programme performance can be incorporated into the process of programme recognition. Examples include student attrition from programmes, and student feedback concerning the teaching and learning environment.

References

[17] de Magalhães Castro, M. (2015), “Higher Education Policies in Brazil: A Case of Failure in Market Regulation”, in Schwartzman, Simon Pinheiro, Rómulo Pillay, P. (ed.), Higher Education in the BRICS Countries - Investigating the Pact between Higher Education and Society, Springer, Dordrecht, http://dx.doi.org/10.1007/978-94-017-9570-8_14.

[14] DEAC (2018), DEAC Accreditation Handbook, Distance Education Accrediting Commission (DEAC), Washington, D.C., http://www.deac.org (accessed on 15 November 2018).

[5] Escueta, M. et al. (2017), “Education Technology: An Evidence-Based Review”, NBER Working Paper Series, No. 23744, National Bureau of Economic Research, Cambridge, http://www.nber.org/papers/w23744 (accessed on 23 November 2018).

[8] Estadão (2018), Governo vai fiscalizar oferta de curso a distância (Government will investigate supply of distance education courses), Estadão, https://educacao.estadao.com.br/noticias/geral,governo-vai-fiscalizar-oferta-de-curso-a-distancia,70002327119 (accessed on 02 December 2018).

[18] European University Association (2018), University Autonomy in Europe, https://www.university-autonomy.eu/ (accessed on 15 October 2018).

[4] Governo do Brasil (2017), MEC identifica irregularidades em cursos de 27 instituições de ensino superior (MEC identifies irregularities in programmes in 27 HEIs), News site of the Brazilian Government, http://www.brasil.gov.br/noticias/educacao-e-ciencia/2017/06/mec-identifica-irregularidades-em-cursos-de-27-instituicoes-de-ensino-superior (accessed on 14 November 2018).

[3] IESALC (2017), Educación superior y sociedad : Aseguramiento de la Calidad en America Latina, Instituto Internacional de Unesco para la Educación Superior en América Latina y el Caribe (IESALC), Caracas, http://www.revencyt.ula.ve (accessed on 14 November 2018).

[9] INEP (2018), Sinopses Estatísticas da Educação Superior 2017 – Graduação - INEP (Statistical Sinopsis for Higher Education 2017), http://portal.inep.gov.br/web/guest/sinopses-estatisticas-da-educacao-superior (accessed on 13 November 2018).

[13] INEP (2017), Instrumento de Avaliação de cursos de graduação Presencial e a distância - Reconhecimento e Renovação de Reconhecimento (Evaluation instrument for undergraduate programmes - classroom-based and distance - recognition and renewal of recognition), Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira, Brasília, http://www.publicacoes.inep.gov.br (accessed on 11 November 2018).

[10] INEP (2017), Instrumento de Avaliação de cursos de gradução - Presencial e a distância - Autorização (Evaluation instrument for undergraduate programmes - classroom-based and distance - authorisation), http://www.publicacoes.inep.gov.br (accessed on 11 November 2018).

[2] INEP (2017), Instrumento de Avaliação Institucional Externa - Presencial e a distância - Credenciamento (External institutional evaluation instrument - classroom-based and distance - accreditation), Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira , Brasília, http://download.inep.gov.br/educacao_superior/avaliacao_institucional/instrumentos/2017/IES_credenciamento.pdf (accessed on 11 November 2018).

[16] MEC (2017), Instrução normativa No 2, de 18 de dezembro de 2017 (Normative instruction 2 of 18 December 2017), Ministério da Educação / Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira, Brasília, http://portal.imprensanacional.gov.br/web/guest/consulta?p_p_id=101&p_p_lifecycle=0&p_p_state=maximized&p_p_mode=view&_101_struts_act…1/5 (accessed on 15 November 2018).

[11] MEC (2017), Portaria Nº 20, de 21 de dezembro de 2017. (Ordinance 20 of 21 December 2017, relating to procedures and decision-making standards for the processes of accreditation, re-accreditation, authorisation, recognition and renewal of recognition of classroom-based and distance higher education programmes, as well as modifications to these, in HEIs in the federal education system), http://www.in.g (accessed on 14 November 2018).

[15] MEC (2017), Portaria Normativa No 19, de 13 dezembro de 2017 Dispõe sobre os procedimentos de competência do Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira - INEP referentes à avaliação de instituições de educação superior, de cursos de graduação e de desempenho acadêmico de estudantes., Ministério da Educação, Brasília, http://www.abmes.org.br (accessed on 15 November 2018).

[7] MEC (2017), Portaria Normativa no.11, de 20 de junho de 2017 - Estabelece normas para o credenciamento de instituições e a oferta de cursos superiores a distância, em conformidade com o Decreto nº 9.057, de 25 de maio de 2017., http://www.in.gov.br/autenticidade.html (accessed on 04 December 2018).

[6] Presidência da República (2017), Decreto Nº 9.057, de 25 de maio de 2017 Regulamenta o art. 80 da Lei nº 9.394, de 20 de dezembro de 1996, que estabelece as diretrizes e bases da educação nacional. (Decree 9057 concerning distance education), http://www.planalto.gov.br/ccivil_03/_Ato2015-2018/2017/Decreto/D9057.htm (accessed on 02 December 2018).

[1] Presidência da República (2017), Decreto Nº 9.235, de 15 de dezembro de 2017 - Dispõe sobre o exercício das funções de regulação, supervisão e avaliação das instituições de educação superior e dos cursos superiores de graduação e de pós-graduação no sistema federal de ensino. (Decree 9235 of 15 December 2017 - concerning exercise of the functions of regulation, supervision and evaluation of higher education institutions and undergraduate and postgraduate courses in the federal education system), http://www.planalto.gov.br/ccivil_03/_Ato2015-2018/2017/Decreto/D9235.htm (accessed on 10 November 2018).

[12] SEMESP (2017), Decreto no 9235/17: Análise comparativa, SEMESP, São Paulo, http://www.semesp.org.br/wp-content/uploads/2017/12/Comparativo-Decretos-21.12.pdf (accessed on 15 November 2018).

[20] TEQSA (2018), Risk Assessment Framework - Tertiary Education Quality and Standards Agency, Tertiary Education Quality and Standards Agency, https://www.teqsa.gov.au/risk-assessment-framework (accessed on 15 November 2018).

[19] UK Government (2018), Check if a university or college is officially recognised - GOV.UK, https://www.gov.uk/check-a-university-is-officially-recognised (accessed on 15 November 2018).

Notes

 1.  In 2016, there were 1 866 private colleges (“faculdades”) in Brazil and only four federal public colleges. In addition, there were 134 state and municipal public colleges, but these are not subject to the federal regulatory and quality assurance system for institutions and undergraduate programmes (Source: INEP).

 2.  In 2016, there were 63 federal public universities, 89 private universities and 156 private university centres in Brazil. No federal public institutions had the status of “university centre”.

 3.  Peer reviewers use indicator 1.20, “Número de vagas” (number of study places), to evaluate programme capacity. An on-site evaluation with a score of three or higher authorises the programme to have the number of study places proposed in the PPC. The programme may subsequently seek additional study places by submitted a request to SERES, a new on-site evaluation and another regulatory act called “Aumento de vagas” (increase in study places).

 4.  This is the case in Austria, Ireland, Luxembourg, Norway, Poland, Sweden, Switzerland, United Kingdom (European University Association, 2018[15])

 5.  Australia, for example, has a highly developed approach to risk in quality assurance, outlined by the Tertiary Education Quality and Standards Agency in their Risk Assessment Framework (TEQSA, 2018[20])

5.

Assuring and promoting quality for existing undergraduate programmes

This chapter examines the processes in place in the federal higher education system in Brazil to monitor the quality of established undergraduate programmes and take action in the event of poor performance. As currently designed, the cycle of ongoing quality monitoring involves the use of large-scale student testing, administered through the National Examination of Student Performance (ENADE), the results of which feed, with other indicators, into a composite indicator of programme performance – the Preliminary Course Score (CPC). When programmes score poorly on the CPC measure, they are subjected to an on-site peer review visit and may ultimately face sanctions imposed by the Ministry of Education. The chapter provides a critical assessment of these processes and provides recommendations for their improvement.

5.1. Focus of this chapter

Undergraduate programmes are subject to an ongoing cycle of evaluation and regulation

Once undergraduate programmes have been recognised under the procedures discussed in the previous chapter, they are subject to an ongoing cycle of evaluation, coordinated by INEP on behalf of the Ministry of Education. As currently designed, this cycle involves the collection and collation, by INEP, of programme-level data, including the results of the programme’s students in a national assessment of learning outcomes, to create a composite indicator of programme performance every three years. As a general rule, in cases where programmes score low ratings in relation to this composite indicator, a new on-site programme review is undertaken by external evaluators. The results of a programme in relation to the composite indicator and, where used, the on-site review determine whether or not SERES renews its official recognition, thus guaranteeing that the diplomas awarded by the programme retain national validity.

Large-scale, external student testing forms part of this cycle

One of the most distinctive features of this ongoing quality assurance process for undergraduate programmes in Brazil is the use of large-scale, discipline-specific student testing. The assessment of the performance of students from every undergraduate programme is an explicit requirement of the 2004 legislation that established SINAES, the National System of Higher Education Evaluation (Presidência da República, 2004, p. art.2[1]). Each year, students graduating from programmes registered in a particular set of disciplinary fields are required to take a mandatory competence assessment – the National Examination of Student Performance (Exame Nacional de Desempenho de Estudantes, ENADE). Disciplines are assigned to three broad groups, with disciplines in group I evaluated one year, group II the year after and group III the year after that, meaning each discipline is subject to ENADE every three years¹. The ENADE tests contain a general competence assessment common to exams in all fields in a single year and a discipline-specific component. In addition, all students participating in ENADE are required to complete a student feedback questionnaire providing biographical information and a personal assessment of their programme.

The results of ENADE tests feed into a composite indicator of programme quality

The results achieved in ENADE by students in a given programme are converted into an average, which is then assigned a score on a scale of one to five, based on its position in the distribution of the average scores for all programmes in the federal higher education system in the same disciplinary field. The resulting score out of five is referred to as the “ENADE score” or Conceito ENADE. The Conceito ENADE is subsequently used, alongside a standardised indicator based on a comparison of ENADE scores for each student with their previous performance in the Exame Nacional do Ensino Médio (ENEM) school-leaving exam; data on staff involved in the programme; and scores from the student feedback questionnaire, to generate an overall score for each programme on a scale of one to five. This programme score – misleadingly named the Preliminary Course Score or Conceito Preliminar de Curso (CPC)² - is used to provide an indication – albeit partial - of the overall performance of the programme in question (MEC, 2017[2]).

If the CPC is three or above, the programme usually has its official recognition renewed by SERES (a regulatory process called Renovação de reconhecimento), without further direct evaluation until the subsequent round of ENADE, three years later (MEC, 2017[2]). If a programme obtains a CPC score of one or two, INEP organises another on-site evaluation visit, using the same criteria and evaluation template as used for the initial process of programme recognition, discussed in the previous chapter (INEP, 2017[3]). If the site visit leads to a positive assessment, with a Conceito de Curso (CC) of three or above, the programme in question has its recognition renewed. If it obtains a score lower than three, it will be subject to supervisory measures by SERES and ultimately risks losing formal recognition – and thus national validity for the diplomas it awards. SERES may request that programmes that achieve a CPC of three or above are also subject to site visits if there are other reasons to do this.

The steps in this cycle of evaluation and regulation are illustrated in Figure 5.1 below.

Figure 5.1. The cycle of evaluation and regulation for undergraduate programmes

Note: The ENADE cycle for each subject area is repeated every three years.

Source: OECD based on information from INEP and SERES (MEC, 2017[2])

5.2. Strengths and weaknesses of the current system

The sections which follow provide an assessment of the strengths and weaknesses of the current model of ongoing quality assurance for undergraduate programmes in Brazil, involving the systematic use of ENADE and the Preliminary Course Score (CPC) and selective use of site visits.

Measuring student performance: ENADE

Relevance: rationale and objectives of the current system

Brazil is one of the few countries worldwide that implements a national system of testing for undergraduate students and is probably the only higher education system that draws on external assessment of learning outcomes to such a large extent in its regulatory, supervisory and quality assurance model. The most notable other examples of system-wide external student testing in higher education in the world are also in Latin America.

Colombia has compulsory competency testing for graduating undergraduate students in the form of the Saber Pro exams, although the compulsory modules applied to all students in that system focus on measuring generic competencies³. In the Colombian system, additional discipline-specific modules are only taken by students when their higher education institutions opt in to this component (ICFES, 2018[4]). In Mexico, CENEVAL, an independent foundation, implements a system of discipline-specific tests for students graduating from academic bachelor programmes, the Exámenes Generales para el Egreso de Licenciatura (EGEL). However, in the Mexican context, whether or not students take this exam depends on the specific policies of the HEI they attend, not on specific requirement from public authorities. Unlike Brazil, Mexico does not have a comprehensive and compulsory system of quality assurance in higher education and EGEL is therefore used by HEIs to monitor their own performance and demonstrate the effectiveness of their programmes to the outside world, rather than as an input to universal evaluation and regulatory processes (OECD, 2018 (forthcoming)[5]).

The formal objective of ENADE is set out in the 2004 legislation establishing SINAES:

ENADE shall assess students' performance in relation to the curriculum guidelines for their respective undergraduate course, their ability to adjust to the requirements arising from the evolution of knowledge and their competence in understanding themes outside the specific scope of their profession, linked to Brazilian and worldwide realities and other areas of knowledge. (Presidência da República, 2004, p. art.5[1])

The most recent Ministry of Education ordinance describing INEP’s evaluation activities in higher education further specifies that ENADE should measure the “skills and competencies” that undergraduate students have “acquired in their training”. It should do this based on the content in the relevant National Curriculum Guidelines (Diretrizes Curriculares Nacionais); the National Catalogue of Advanced Technology Programmes (Catálogo Nacional de Cursos Superiores de Tecnologia); “associated norms”; and “current legislation regulating professional practice” (MEC, 2017, p. art.41[6]). The National Curriculum Guidelines (DCN) and the National Catalogue of Advanced Technology Programmes specify expected learning outcomes for programmes in different fields at a general level. The formal expectation is thus that ENADE should assess the extent to which students have acquired, through their undergraduate programme, the learning outcomes specified in these documents. In addition, as shown in the citation above, the 2004 legislation calls for ENADE to measure the ability of students to adjust to the “evolution of knowledge” and the ability of students to understand wider themes outside their immediate field of study.

Defining and measuring learning outcomes in higher education in a systematic way has been an increasing focus in higher education practice and policy in recent decades (CEDEFOP, 2016[7]). This trend reflects a widespread recognition that gaining a better understanding of the extent to which students really acquire new knowledge and skills through higher education would provide useful feedback for teachers and educational institutions and useful information for governments and society at large on the educational performance of higher education (OECD, 2013[8]). On a conceptual level, it is desirable to have reliable information on learning outcomes – and not simply inputs and processes involved in the learning process – in order to judge how effective and efficient the education provided is in practice. However, while the explicit definition of expected learning outcomes has become the norm across many higher education systems, final assessment of these learning outcomes is nearly always left to individual teachers and institutions (or to employers or professional organisations). In contrast to practice at school level, where many OECD and partner countries use external high school leaving examinations, attempts at the comparable measurement of student learning outcomes in higher education, between institutions and disciplines, have been rare (OECD, 2013[8]).

In Europe, for example, the systematic establishment of expected learning outcomes for higher education programmes has been a key feature of the Bologna Process. The capacity of programmes to deliver these learning outcomes is a strong focus of many European quality assurance models. Nevertheless, no European quality assurance system has so far attempted to incorporate direct assessment of learning outcomes in its processes, although some – such as that in the Netherlands (NVAO, 2016[9]) and Sweden (UKÄ, 2018[10]) – have included qualitative assessment of selected student outputs in programme-level reviews.

The OECD initiated the Assessment of Higher Education Learning Outcomes (AHELO) (OECD, 2013[8]) project and the European Commission has supported a project to develop frameworks to assess learning outcomes in different higher education disciplines (Comparing Achievements of Learning Outcomes in Higher Education in Europe - CALOHEE) (CALOHEE, 2018[11]). However, neither of these initiatives has yet been fully implemented in large-scale testing.

Despite the widely accepted potential value of assessing learning outcomes in theory, in practice, external assessments at higher education level face substantial challenges:

A first challenge is defining what to measure. Whereas in school education, all students are expected to follow a common core of subjects, at least until the end of compulsory schooling, students in higher education study subjects with a narrower, but deeper focus. Within their chosen discipline, higher education students also tend to specialise in certain sub-domains, especially in the later years of their course. Accurately measuring the skills and competences acquired by students during their programme requires testing instruments that are adequately tailored to the content of the programmes in question. If tests are aligned too closely to specific programme content, they can only be applied to comparatively small numbers of students, who have been exposed to that specific content. This makes creating a system-wide testing system complex and costly and can restrict comparability of results to comparatively small cohorts. If the tests are made more general, to allow them to be applied to a larger number of students, there is a risk that the test content is no longer sufficiently aligned to the programme content to allow causal inferences to be made about the effects of the programme on student learning.

The challenge for test developers is, therefore, where to set the boundaries of discipline-specific knowledge, in terms of breadth and depth, to maintain a clear link to programme content, while creating a test that is widely applicable. In some cases, such as the Colombian Saber Pro exam, test developers have avoided this problem by focusing compulsory test components exclusively on generic competences and using the same test for students from all disciplines. The first problem with this approach is that generic competences are often an implicit, rather than explicit, intended learning outcome of higher education programmes. Although higher education programmes may help students to improve their reading skills and basic problem-solving, this might be a by-product of the main teaching and learning activities, rather than an explicit focus. Generic competence tests therefore set out to test competences that higher education programmes – rightly or wrongly – might not have set out to develop in their students, while ignoring discipline-specific content and skills that are the prime focus of the course.

A second key challenge is how to identify and formulate test items that can measure, in a reasonably brief test, the kinds of knowledge and discipline-specific competencies that students might be expected to acquire through a programme lasting three, four or more years. Alongside the breadth of theoretical knowledge that could be covered in a written test, lies the challenge of creating assessment models for the practical and professional skills that are central to many higher education programmes such as nursing, medicine, engineering or architecture. Although it is conceivably possible to cover a wide range of topics and tasks in a standardised test lasting several hours, a balance needs to be struck between coverage of items and creating optimal conditions for a student to perform to the best of their ability. Long tests are tiring, while short tests can create excessive stress that impacts negatively on student performance.

A final issue relates to the risk that standardised testing, even when adequately differentiated between disciplines and types of programme, can unduly influence the programme content and pedagogical approaches used by institutions and teaching staff. When broad curriculum guidelines are translated into a specific test, this necessarily entails making specific judgements about what students should learn. When the test in question matters, this encourages teachers to “teach to the test”, with a clear risk that legitimate learning objectives and subjects not covered by the test are excluded and that programme coordinators feel constrained in make changes to their curricula. Standardised testing might thus constrain curriculum breadth, innovation and responsiveness to changing circumstances.

In light of these considerations, the OECD review team considers that the objectives of ENADE, as currently formulated, are unrealistic. The relevant legislation, noted above, requires ENADE to measure students’ performance in relation to the content of relevant national curriculum guidelines, their ability to analyse new information and their wider understanding of themes outside the scope of their programme. This creates problems, as follows:

The requirement to measure understanding of unspecified “themes outside the scope of the programme” is inherently problematic because it is so general and the knowledge and skills assessed, by definition, are not part of the programme’s core intended learning outcomes. It is thus unclear how those running programmes could be expected to equip students with such a wider range of unspecified knowledge and skills or why they should be held accountable for students’ not having these competencies at the end of their studies.

As ENADE is a written examination with a restricted duration, it is impossible to measure the full range of learning outcomes that any adequately formulated curriculum guidelines should contain. At best, written examinations (computer or paper-based) can measure a scientifically robust selection of discipline-specific knowledge and skills and, potentially, generic competences such as logical reasoning or problem-solving (if such competencies are intended learning outcomes). An exam like ENADE can only ever assess a small proportion of what students will have been expected to learn over the duration of their programme. It would be beneficial if this were acknowledged in the stated objectives of the exam.

By implying that ENADE sets out to measure students’ learning outcomes in relation to the National Curriculum Guidelines for undergraduate programmes, there is a risk that the content of ENADE (in a given year or previous years) comes to be seen to define what is important in the National Curriculum Guidelines. In practice, the National Curriculum Guidelines in Brazil (MEC, 2018[12]) specify very general content for programmes, leaving considerable scope for HEIs and teachers to innovate and adapt content to current needs. If ENADE has an excessively narrow approach, it risks undermining this freedom. In interviews, representatives of HEIs in Brazil, indicated that the content of ENADE did indeed influence the content of their programmes.

If ENADE is to be maintained, Brazil’s legislators and quality assurance authorities need to provide a more credible account of what it can realistically achieve and how risks for innovation and responsiveness can be mitigated.

Effectiveness: quality indicators used and generated

ENADE aims to measure student performance through an assessment composed of ten questions focused on general competencies (formação geral), of which two call for short discursive answers and the rest multiple choice answers, and 30 discipline-specific questions, of which three are discursive and 27 are multiple choice. The general competency questions are common to all ENADE exams in a single year. Exams are taken with paper and pen in test centres hosted by students’ higher education institutions and students have up to four hours to complete the whole exercise. Participation in the examination is compulsory for all students graduating in courses in the fields being assessed in a given year, with attendance being a pre-requisite for receiving the diploma for their degree⁴. The tests are administered on the three-year cycle described above, with fields following the order set out in Table 5.1.

Table 5.1. ENADE cycle
Year | Fields examined
1 | a) Bachelor's degrees in the areas of Health, Agrarian Sciences and related areas;
b) Bachelor's degree in Engineering;
c) Bachelor's degrees in the area of Architecture and Urbanism;
d) Advanced Technology Programmes (Tecnologias) in the areas of Environment and Health, Food Production, Natural Resources, Military and Security;
2 | a) Bachelor's degrees in the areas of Maths, Natural Sciences, Computing and related areas;
b) Evaluation areas leading to a double qualification with Bachelor's degree and Licenciatura;
c Evaluation areas leading to a Licenciatura;
d) Advanced Technology Programmes (Tecnologias) in the areas of Control and Industrial Processes, Information and Communication, Infrastructure, Industrial Production;
3 | a) Bachelor's degrees in the areas of Applied Social Sciences, Human Sciences and related areas;
b) Advanced Technology Programmes (Tecnologias) in the areas of Management and Business, School Support, Hospitality and Leisure, Cultural Production and Design
Source: Adapted from Regulatory Ordinance no 19, of 13 December 2017 (MEC, 2017[6])

Both the general competencies and discipline-specific sections of ENADE are marked out of 100, generating ‘raw’ marks out of 100 for each section, for each participating student. These are combined, with general competencies accorded a weighting of 25% and the discipline-specific section 75%, to generate an overall average ‘raw’ mark per student and per programme. Data from the 2016 round of ENADE (INEP, 2017[13]) show that average marks on the general competencies section, which is the same for all students, ranged from 38.2% for students from Advanced Technology Programmes in cosmetology to 60.3% for (bachelor’s degrees in) medicine. Average marks for the discipline-specific components, which are not comparable between disciplines, ranged from 38.3% in physiotherapy bachelor’s programmes to 67% in medicine.

The raw average values for each programme in a single field are then standardised by calculating their standard deviation (distance from the mean score for programmes in a given field) and attributing the standard deviations to value bands from one to five, as shown in Figure 5.2 (INEP, 2017[14]). Outliers are excluded in setting the minimum and maximum scores. This process is carried out for all programmes with at least two graduating students participating in the exam. The resulting score from one to five for each programme is the ENADE score or Conceito ENADE. 40% of programmes evaluated in 2016 received an ENADE score of three and 6% a score of five (INEP, 2017[13]).

Figure 5.2. Standardisation of ENADE scores

Source: INEP (2017) Presentation ‘Enade 2016 Resultados e Indicadores – DAES, setembro 2017’. (INEP, 2017[13])

The administration of ENADE is a complex exercise. Every year, INEP coordinates the development of tests that include 30 discipline-specific items for numerous fields (27 in 2018⁵). The agency oversees the implementation of these tests on a single day for around 200 000 students from over 4 000 programmes in 1 000 HEIs across a vast country; and coordinates the marking of the exam papers, attribution of scores and calculation of resulting indicators.

However, since its inception in 2004, ENADE has been subject to criticism, especially from within the Brazilian academic community. Based on the evidence gathered during the review, the OECD review team considers that there are at least five principal weaknesses in the way ENADE is currently designed and implemented.

1. The first problem relates to the participation of students and their motivation to make an effort in the test. It is evident that a proportion of the students who should be taking the test each year are not doing so. Across years, between 10-15% of students registered to take the test each year do not turn up on the day. Moreover, there are concerns among stakeholders in Brazil that some HEIs seek to avoid registering a proportion of students for ENADE. This may either be through maintaining deliberate ambiguity about the attribution of programmes to an ENADE test field – a problem recently highlighted by Brazil’s Federal Court of Accounts (TCU, 2018[15]). Alternatively, it may occur as a result of deliberate policies by HEIs to ensure the academic progress of “weak” students is slowed, so that they are not scheduled to graduate in a year when their discipline is subject to ENADE and thus avoid taking the exam.

At the same time, ENADE is a high stakes exam for HEIs (as it is used in the quality rating of their programmes), but a low stakes exam for students. Although attendance is compulsory, ENADE scores have no effect on students’ academic record and there is evidence that a significant proportion of students do not complete large parts of the test (Melguizo and Wainer, 2016[16]). Although some institutions organise prizes for ENADE participation, there is no systematic use of incentives for success in the exam. Evidence from other OECD and partner countries suggests that if the results of tests have no real consequences for students, this impacts negatively on student motivation and performance (Wolf and Smith, 1995[17]; Finney et al., 2016[18]). Moreover, interviews with representatives of HEIs conducted during the review visit to Brazil suggest that ENADE is taken more seriously by both students and staff in less prestigious for-profit institutions, than in public institutions. This appears to be primarily because the results of ENADE have a greater potential impact on the reputation of private institutions, which have to compete for students (public institutions are free and thus usually oversubscribed). This, and the participation issues, have negative implications for the validity of the results as an accurate reflection of the learning outcomes of all students and for the comparability of results between programmes and institutions.

2. A second concern relates to the development, selection and use of test items for each ENADE test. At present, there is no robust methodology to ensure that the difficulty of each test item is taken into account in the composition of the test and thus that a) tests in the same field are of equivalent difficulty between ENADE cycles and b) tests in different fields are of a broadly similar level of complexity. This means that it is not possible to compare the raw results or the Conceito ENADE between years or between disciplines. Unlike in external school leaving exams across the world, no reliable attempt is made to ensure exams are of consistent difficulty between years or that an exam in, say, philosophy is of broadly equivalent complexity in relation to expected learning outcomes to ones in maths or chemistry. Some differentiation in expected standards is important. For example, a bachelor’s degree would be expected to take students to a higher level than an Advanced Technology Programme in a related subject. However, it is reasonable to expect all ENADE exams for equivalent levels (bachelor’s degrees, short-cycle degrees) to test students at an equivalent level in relation to the expected learning outcomes for their programme.

A related question is whether the number of discipline-specific items included in ENADE (30) is adequate to generate a reliable indication of students’ learning outcomes from an undergraduate programme. The answer almost certainly depends on what the exam is for. A robustly designed examination with only 30 items, may be able to provide a general indication of a students’ level of knowledge and competencies in a specific disciplinary field. However, such a test is unlikely to provide reliable evidence of students’ performance in specific sub-fields or aspects of the curriculum, which limits its usefulness as a tool to help teaching staff and institutions improve the design of their programmes.

3. A third problem is that there are no explicit quality thresholds or expected minimum levels of performance for ENADE tests. “Raw” scores (out of 100) are widely published and discussed, with a frequently expressed assumption that students are not performing well (EXAME, 2017[19]). However, the absence of a clearly calibrated level of difficulty in tests means it is impossible to say whether a score of 50% represents good or bad performance. If the test is set at a high level of difficulty, it might be a very good mark, if the test is at a low level, it would not be a good mark. Without tests of a comparable standard of difficulty and without defined quality thresholds (pass, good, excellent, etc.), ENADE scores are simply numbers, reported according to their position to all other scores in the same test. It is thus impossible to know if students in programmes that achieve 50% or 60% in ENADE are performing well or poorly.

4. A fourth problem relates specifically to the design of the general competencies (formação geral) component of ENADE. This is currently composed of general knowledge questions regarding current affairs and social issues, including two questions that call for short discursive answers. This reflects the requirement of the 2004 legislation that ENADE test the wider knowledge and understanding of students of issues outside the scope of their studies. However, as noted, this is a flawed objective. Unless all undergraduate programmes have knowledge of current affairs and social issues as explicit intended learning outcomes – which is not the case – it is unreasonable to judge individual programmes on students’ performance in these areas. It noteworthy that ENADE does not include questions designed to test students’ logical reasoning or similar generic competencies that one might reasonably expect all higher education graduates to possess.

5. A final issue is that the standardisation of ENADE scores compounds the lack of transparency about what ENADE results really mean. As noted, raw marks are simply attributed to a five-point scale based on the standard distribution of scores in a single subject in a given year. As tests may vary in difficulty and students obtain very different distributions of scores, where a programme falls on a standard distribution of the scores for all programmes says little about the actual quality of the programme in question. In simple terms, scoring a three in one discipline in a particular year may require a far higher level of performance than scoring a three in another discipline in the same year or the same discipline in another round of ENADE. If all students score low marks – and thus all programmes are performing poorly - the distribution will be skewed towards the lower end of the performance scale, but as the bar (average) is set low, programmes will still emerge as having scores of three and more.

Effectiveness: use and effects of ENADE results

The ENADE score is published separately for each programme in the e-MEC online platform (MEC, 2018[20]) and subsequently used as an input to the Preliminary Course Score (CPC), discussed in more detail below. The CPC is used by SERES to make regulatory decisions on whether programmes should have their official recognition renewed directly every three years, or should undergo a further on-site inspection (MEC, 2017[2]). ENADE results are thus used by Brazilian authorities to judge whether programmes are of adequate quality to allow their continued operation. The results of ENADE are also widely reported in Brazilian media and, in some cases, used by institutions as part of their marketing efforts, alongside the programme and institutional quality scores.

Students have access to their own ENADE results, but it is unclear to what extent they actually have an interest in these, given the low importance of the test for their own careers. Importantly, the OECD team understands that institutions, and staff in programmes, do not have access to the individual scores for their programmes. Institutions consulted by the OECD review team report that they did not use of ENADE results in efforts to improve the design and content of programmes. During the review visit, representatives of institutions consistently indicated that they did not see ENADE as providing useful feedback to help them improve their programmes. This is a pattern confirmed by recent studies on the use of ENADE results in HEIs in Brazil (De Sousa and De Sousa, 2012[21]; Oliveira et al., 2013[22]; Santos et al., 2016[23]). In contrast, some HEI representatives consulted by the OECD team, in line with the findings of the same studies, highlighted the use of ENADE results in marketing programmes to prospective students and promoting their institutions.

Efficiency: the cost-effectiveness of ENADE

Although the OECD review team does not have access to a detailed breakdown of the costs of implementing ENADE, these account for a substantial part of INEP’s budget for evaluation of higher education, which amounted to over 118 million reals (USD 30.7 million) in 2017 (INEP, 2018[24]). It is questionable whether the quality and usefulness of the results achieved with the exam as currently configured justify the investment of public resources committed. As noted, through their use in the CPC, ENADE results are used by MEC as an important indicator of the quality of undergraduate programmes in Brazil.

The first question is therefore whether ENADE provides an accurate indicator of programme quality. For the reasons outlined above, it is the view of the OECD review team that, in its current form, it does not. The second question is whether an improved version of ENADE, addressing the current design and implementation weaknesses noted in the preceding section, would be generate information about the quality of undergraduate programmes that could not be provided by other, potentially more readily available, indicators. The third question is whether, even if an improved ENADE brought additional information about quality that could not be provided by other indicators, this additional information justified the considerable costs of designing and implementing such a system of standardised testing. These are questions that the Brazilian authorities and academic community should consider in planning the future of the federal quality assurance and regulatory regime for higher education.

Monitoring programme performance: use of programme indicators

Relevance: rationale and objectives of the current system

INEP and MEC use a set of programme-level indicators, drawing heavily on ENADE results, to monitor the ongoing performance of undergraduate programmes in the federal system of higher education. The legal basis for this is, in principle, provided for in the 2004 legislation establishing SINAES, which states that evaluation of programmes “shall use diversified procedures and instruments, among which must be visits by expert committees from the respective areas of knowledge” (Presidência da República, 2004, p. art.4[1])”. On-site reviews are always used as a basis for initial recognition of undergraduate programmes, but are not used systematically in the ongoing monitoring of programme performance and the periodic renewal of programme recognition. As discussed below, in many cases, SERES makes judgements about the quality of programmes and approves the renewal of their official recognition based on programme indicators generated by INEP.

A recent MEC implementation ordinance for SINAES (MEC, 2017[6]) states that INEP has the responsibility to calculate and publish indicators of the quality of higher education, in line with methods established in technical notes approved by CONAES (the supervisory board for the higher education evaluation system). It does not, however, specify in detail what these indicators should be. The same ordinance states simply that indicators produced by INEP are to “support” (subsidiar) public policies in higher education. In practice, however, centrally collated programme indicators play a central role in the federal quality assurance system. The current indicators and methods used, as well as their application in regulation and supervision by SERES, are specified in technical notes prepared by MEC and INEP (MEC, 2017[2]; INEP, 2017[22]).

In the current system, INEP and MEC use indicators as a means to identify potentially poor quality programmes that warrant more in-depth evaluation and supervision. Programmes that score below three out of five on the composite Preliminary Course Score (CPC) after each three-year ENADE cycle are systematically subject to on-site inspections by external review commissions, with a positive evaluation score (a CC of three or above) a pre-requisite for renewal of their official recognition. Courses that score three or above on the CPC generally have their programme recognition renewed automatically by SERES (MEC, 2017[2]).

In a system as large as Brazil’s, with a wide diversity in the quality of provision, there is a clear rationale for using centrally collated indicators to monitor programme quality. Notwithstanding the minimum quality guarantees provided by the recognition process, wider quality concerns in the higher education system in Brazil, particularly in parts of the for-profit private sector, mean that ongoing monitoring of programmes by public authorities is justified. However, in a system of such size, it is challenging to conduct regular programme-level reviews for all active undergraduate programmes, as is the practice in some smaller OECD higher education systems (such as the Netherlands or Portugal, for example).

Despite the practicality of an indicator-based approach, some commentators in Brazil consulted by the OECD review team question whether dispensing with on-site reviews for some programmes in favour of indicator measures is consistent with the spirit of the 2004 legislation (which says programme evaluation must involve on-site visits). More fundamentally, the question is which indicators to use to monitor programme quality and how to combine these in meaningful ways.

Effectiveness: quality indicators used or generated

To monitor programme performance, INEP currently uses a set of indicators comprising a) measures of student performance and assumed learning gain (based on ENADE test results); b) the profile of the teaching staff associated with the programme and; c) feedback from students about teaching and learning, infrastructure and other factors, from the questionnaires they complete in advance of taking the ENADE test. When new ENADE results are available for each programme, after each three-year cycle of testing, INEP calculates a programme score – the Preliminary Course Score or Conceito Preliminar de Curso (CPC) using the weightings set out in Table 5.2.

Table 5.2. Indicators used to calculate the CPC
Dimension | Components | Weights
Student performance | Score from students taking ENADE | 20.0% | 55%
Indicator of Difference between Observed and Expected Performance (IDD) | 35.0%
Staff (Corpo Docente) | Proportion of Masters | 7.5% | 30%
Proportion of PhDs | 15.0%
Score for employment status of staff (regime de trabalho) | 7.5%
Student perception about the educational process | Score relating to the organisation of teaching and learning | 7.5% | 15%
Score relating to physical infrastructure | 5.0%
Score relating to additional academic and professional training opportunities | 2.5%
Source: INEP (2017) Nota Técnica Nº 3/2017/CGCQES/DAES (INEP, 2017[22])

Under “student performance”, the ENADE score (Conceito ENADE) discussed in the previous section is complemented by an indicator of the assumed “added value” of the programme for each student’s performance in ENADE. The Indicator of difference between observed and expected performance (Indicador da Diferença entre os Desempenhos Observado e Esperado) or IDD is calculated by comparing each student’s actual results in ENADE with the performance that would be expected on the basis of their previous performance in the national high school leaving exam, ENEM (Exame Nacional do Ensino Médio).

This process uses results data associated with each individual’s unique identification number (número do Cadastro de Pessoas Físicas or CPF). The “expected” performance for each student in ENADE in relation to the total population of ENADE participants is calculated on the basis of ENEM results using a regression model. This is then compared with the student’s actual performance in ENADE. The difference between the expected and observed performance is considered as the added value of the programme (positive or negative). The average differences for all students⁶ in a single programme are combined to create an IDD score for the programme (INEP, 2017[23]).

The data on the composition of staff are drawn from INEP databases and awards higher ratings to programmes with higher proportions of teaching staff with master’s and doctoral degrees and with full-time, permanent status. This tends to reward public institution that have higher proportions of staff with doctorates (see Figure 3.7) and full-time contracts (see Figure 3.3).

The data on student feedback are drawn directly from the online student questionnaire that students have to complete as part of their registration for ENADE. The 2017 questionnaire contained 26 questions asking for biographical details and a further 41 questions focused on the programme (INEP, 2018[24]). Raw data for the student questionnaire results and staff composition are converted into standardised scores for each programme.

The CPC has been criticised in Brazil for several reasons. Criticisms focus on the choice of indicators, the weighting attributed to them and the way they are combined into a single composite indicator. The reliability of the ENADE score as an indicator of programme quality was discussed above. The IDD, which accounts for 35% of the CPC score, is both contested and poorly understood in the wider academic and policy community in Brazil.

The objective of measuring the added value of a programme for the knowledge and skills of students (learning gain) is, in principle, commendable. Not only does it go to the heart of educational effectiveness (how much a student actually learns during their programme), but measuring “added value” also makes it possible to take into account and reward the efforts made by programmes that take in weaker students, often from less advantaged social groups. A measurement of learning gain can, in principle, make it possible to identify programmes that successfully help students with lower levels of knowledge and skills on entry to increase their knowledge and skills over the duration of the programme, even if the final performance of these students remains lower than in programmes that took in high performing students. Added value could thus be a means to recognise the work of less prestigious programmes and institutions that perform a valuable societal role.

However, the IDD as currently calculated in Brazil is not a true measure of learning gain, but a proxy indicator, based on the following assumptions:

1. That students’ performance at the age of 18 in general, high-school-level tests in natural sciences, humanities, languages and maths (INEP, 2017[23]) is a reliable basis on which to predict their future performance in tests designed to measure their learning outcomes after a specialised undergraduate degree.

2. That ENADE accurately measures students’ performance at the end of their undergraduate programme.

3. That when students perform better or worse than their predicted relative performance, this results primarily or significantly from the design and delivery of their undergraduate programmes.

The first assumption is not unreasonable. However, the limitations of ENEM results as a predictive indicator must be fully acknowledged and the results of the regression analysis (the predicted relative performance of students at the end of their undergraduate studies) must be treated with due caution. The subject composition and different stakes involved in ENEM and ENADE are likely to reduce the predictive power of the IDD calculations. Not only will ENEM subjects differ considerably in scope and depth from the subjects included in the disciplinary component of ENADE, but, as highlighted earlier, ENADE is a low stakes exam for students. In contrast, ENEM is comparatively high stakes exam, the result of which have a major impact on students’ chances of accessing higher education. Depending on their profile, the relatively high degree of specialisation of higher education programmes might increase some students’ chances of performing well, compared to generalist high-school exams (assuming they chose to study fields in higher education that correspond to their strengths and interests). Conversely, the low stakes of ENADE tests may reduce the chance of their performing to their maximum potential.

The second assumption is problematic. As discussed above, as well as their low-stakes status, the composition and design of ENADE tests limit their effectiveness as measures of learning outcomes.

These factors also reduce the reliability of the third assumption above. All quality assurance procedures make assumptions about the impact of programme design and delivery on students’ performance. It is certainly possible that programmes will have some influence on students’ performance in the current ENADE tests and that significant differences between real and expected performance, as calculated by the IDD, are due to programme-level factors. However, the combination of the boldness of the underlying assumptions, the poor design of ENADE and the potential influence of factors outside the control of the programme on student performance mean that the IDD provides only limited information on programme quality. It is thus highly questionable whether it should account for 35% of the weight in a composite indicator used as a basis for renewing programme recognition.

The legislation establishing ENADE actually calls for the testing of both first and final year students, in order to allow a direct comparison between their results on entry to, and exit from, undergraduate programmes. This system was implemented between 2004 and 2010. However, the test for incoming students was abandoned, largely on cost grounds, and the current system using ENEM introduced as a substitute. Although testing on entry and exit from a course with coordinated and well-calibrated testing tools also involves many challenges, the OECD team considers it is the only realistic way to measure learning gain within programmes, if this is the real objective of quality assurance authorities.

The indicators used for the composition of teaching staff and student feedback are less ambitious in their aims and less fundamentally problematic. However, by rewarding staff with PhDs and full-time status, the indicators for teaching staff are designed for academic, research-oriented institutions and fail to take into account the value of part-time teachers with professional experience, who are vital for more professionally oriented programmes.

The use of student feedback as an indicator is a positive element in the system. Students are the main stakeholders in the higher education process and their views need to be heard within a well-designed quality assurance system. However, the questionnaire used is long and formulated using excessively bureaucratic language (INEP, 2018[24]). Moreover, it is not clear whether students in Brazil, who sit the ENADE tests in their own institutions, are accustomed to providing honest and objective feedback about their teachers and programmes throughout their studies and whether they are positively encouraged to do so in ENADE. In particular, a concern to maintain the reputation of the institution that has awarded their diploma may make students hesitant to provide negative feedback.

It is widely accepted, including in the recent report from the Federal Court of Accounts (TCU, 2018[15]), that the weightings attributed to the different indicators in the CPC are arbitrary, with no discernible scientific basis. This further compounds the lack of transparency for students, families and society at large about what the scores attributed to courses really mean in practice. It is possible – although the OECD review team is not aware of specific studies – that the current weightings in the CPC formula have a significant impact on where programmes fall in standard distributions and thus their CPC score.

It is positive that the CPC sets out to include indicators of the teaching process (through the imperfect proxy of teaching staff status); qualitative feedback from students (the main beneficiaries of the system) and measures of student learning outcomes. It does not, however, contain a measure of the attrition rate of students (what proportion of students entering a programme complete it) or the subsequent employment outcomes of graduates. Both these factors – although especially the first - are taken into account in quality-related policies in other higher education systems.

Effectiveness: use and effects, efficiency and cost-effectiveness

In light of the shortcomings of the set of indicators employed and the arbitrary weighting of the different factors, a key concern is the way the CPC is currently used in the broader quality assurance and regulatory process for undergraduate provision. As discussed, programmes that score three or above, in most cases, have their recognition renewed automatically until the next ENADE cycle, at which point a new CPC is calculated⁷. Those that score two or less are systematically subject to a review visit (discussed below), which provides an updated programme score (CC) as a basis for the regulatory decision on whether or not to renew recognition.

As argued earlier, the principle of using indicators to identify “at risk” programmes and target scarce resources for on-site inspections makes sense, especially in a system as big as Brazil’s. However, the CPC does not provide a reliable mechanism to identify poorly performing courses. The absence of quality thresholds in ENADE and the standardisation processes used to create the ENADE score, combined with the weaknesses of the IDD, mean it is far from clear whether a CPC score of three represents an adequate standard of quality or not. A reform of the monitoring indicators used and the way they are combined is necessary.

Monitoring programme performance: use of on-site inspections

Relevance: rationale and objectives of the current system

When programmes are identified through the CPC as performing poorly – often meaning they have poor relative performance in ENADE – they are subject to an on-site inspection by external evaluators, coordinated by INEP (MEC, 2017, p. 5[2]). The evaluators assess the supply conditions for the programme using the same evaluation template that was already used for programme recognition (reconhecimento) and was discussed in Chapter 4 of this report (INEP, 2017[3]). The results of the new on-site inspections triggered by the CPC process or special requests from SERES are used as a basis for decisions for programmes’ renewal of recognition.

As noted, the objective of targeting on-site inspections on weakly performing programmes has advantages, as the systematic use of periodic on-site inspections for all programmes in Brazil would almost certainly be unfeasible for logistical and financial reasons. However, it also means programme-level site visits at this stage in the evaluative process always have a punitive character and that peer reviewers are not exposed to good practice in well-established programmes, which could inform their judgements about, and recommendations to, poorly performing programmes.

The objective of the on-site visits for renewal of recognition, as currently conceived, is to (re)check compliance with basic standards, rather than explicitly to promote and support improvement of the programmes concerned, following a serious self-evaluation. The formal objective for the renewal of recognition visits expressed in the relevant regulation (Presidência da República, 2017[25]) is for them to inform the process of renewal of recognition. The extent to which quality assurance should seek to support institutions in quality improvement is open to debate in a highly commercialised system like that in Brazil, although most quality assurance systems worldwide do seek to support quality enhancement, as well as basic compliance.

Effectiveness: quality indicators used and generated

The indicators used in the evaluation template for recognition and renewal of recognition of programmes were discussed in Chapter 4. A clear problem with these indicators in relation to renewal of recognition is that they focus on exactly the same inputs and processes as examined in the initial recognition of the programme, when the first cohort of students had only completed between 50-75% of the course workload, and before any students had graduated. The courses that undergo renewal of recognition on-site inspections are identified primarily because of their poor performance on an – albeit imperfect – set of indicators of output and student feedback. Rather than directly examining the problems identified by poor performance in ENADE or poor student feedback, the on-site evaluators complete a questionnaire measuring variables that have already been verified in an earlier inspection. Moreover, these variables in many cases might be expected to remain constant over time.

It should be recalled that the evaluation template for recognition and renewal of recognition places a 40% weighting on the category “teaching staff” and 30% on “infrastructure”, with the remaining 30% on teaching and learning policies and practices. The indicators and judgement criteria relating to teaching staff mostly focus on the qualifications and experience of the individuals in question, with only three indicators dealing with the activities (atuação) of staff or their interaction with each other.

It is conceivable that some, very poor quality, providers do not maintain the basic infrastructure and teaching workforce to allow their programme to operate correctly and that were initially verified in the recognition process. However, for programmes that have maintained the basic conditions for providing the programme, the balance of indicators in the current evaluation template does not generate overall scores that will reveal the most obvious signs of poor quality. A greater focus on teaching activities and outputs and outcomes (attrition rates, learning outcomes, graduation rates and employment outcomes), would make the evaluation template more effective in identifying the real causes of poor performance.

Effectiveness: use and effects, efficiency and cost-effectiveness

Programmes are inspected at this stage in the evaluation cycle on the grounds that they have scored poorly on the CPC measures of process and outcomes. The resulting inspection then attributes a new quality score – an updated Conceito de Curso (CC) – that effectively replaces the CC attributed at the time of initial recognition and exists alongside the CPC score in the e-MEC system. The CC is, however, based on entirely different indicators from the CPC, despite the very similar names.

Frequently, it appears that programmes which score poorly on the CPC measure subsequently achieve a higher score on the CC (TCU, 2018[15]). As such, these programmes nominally recover the higher quality score. It is entirely understandable that the CPC and the inspections leading to the CC can generate different values. They measure almost entirely different things.

From a conceptual perspective, this is all the more problematic because the more output-focused measures contained in the CPC would – if calculated on a more reliable basis – provide a better indication of the real performance of the programme. Curriculum plans, teachers and infrastructure are all enabling factors for good education and are rightly considered in initial programme approval. However, notwithstanding the potential for staff to leave the programme and infrastructure to be changed, these factors are likely to remain constant over a number of years. Once there is evidence of the broader performance and outcomes of programmes, this evidence should be prioritised in assessments of quality. In the end, if a well-designed programme with good infrastructure and well-qualified teachers fails to train students effectively without very good reasons, the programme is not of high quality.

5.3. Key recommendations

1. Undertake a thorough assessment of the objectives, costs and benefits of large-scale student testing as part of the quality assurance system

Officially ENADE currently seeks to assess students’ acquisition of knowledge and skills specified in the relevant National Curriculum Guidelines (DCN) or the equivalent documents for Advanced Technology Programmes, as well as their understanding of unspecified “themes outside the specific scope” of their programme. This is an unrealistic objective and no standardised test could achieve this. Moreover, as discussed in the preceding analysis, the current design and implementation of the ENADE tests are characterised by significant weaknesses. At present, ENADE results are used extensively as a basis for regulatory decisions (renewal of programme recognition), but are not used by institutions and teachers to identify areas where they programmes need to be strengthened.

Given the long-standing commitment of Brazil’s public authorities to standardised testing of tertiary students, there are sunk costs and considerable expertise in implementing large-scale testing has been developed. Politically, ENADE is widely accepted and viewed by many as an important part of Brazil’s system for quality assurance in higher education.

The OECD team believes, however, that, in its current form, ENADE does not represent an effective use of public resources. As such, as a basis for decisions on the future of the system, a thorough reflection is needed about the objectives of large-scale standardised testing in Brazilian higher education and the costs and benefits of different approaches to implementing it. The main questions to answer are:

1. Can an improved version of ENADE, addressing the current design and implementation weaknesses noted in this report, be implemented and generate reliable information about the quality of undergraduate programmes?

2. Could the information about the quality of programmes generated by a revised ENADE be provided by other, potentially more readily available, indicators? What is the specific and unique added value of ENADE results?

3. If a revised version of ENADE does indeed have the potential to generate valuable information that cannot be obtained from other sources, does the value of this information justify the costs of implementing ENADE? How can the costs of implementation be minimised, while still allowing ENADE to generate reliable and useful results?

The OECD team believes two factors should be considered in particular. First, for ENADE to have the greatest possible added value, it needs to be able to provide reliable information that can help teachers and institutions to identify areas of weakness in their programmes (in terms of knowledge coverage or skills development). ENADE results cannot simply be a blunt indicator used to inform the regulatory process, as other indicators such as attrition rates or employment outcomes could be used for this purpose. Second, the current requirement to apply the ENADE test to all programmes every three years increases the fixed cost of implementing the system. It is important to consider whether sampling techniques could be deployed to reduce costs, while maintaining reliability.

2. If a reformed version of ENADE is retained, ensure the objectives set for the exam are more realistic

If the decision is taken to maintain a revised version of ENADE, it is crucial to ensure the objectives set for it in the relevant legislation and implementing decisions are realistic and clearly formulated. The objective of a reformed ENADE could be to provide:

An indication – rather than a comprehensive picture - of the performance level of students in relation to intended learning outcomes, as one indicator, alongside others, in a comprehensive system of external quality evaluation and;

Data on student performance that can be used directly by teachers and institutions in identifying weaknesses in their programmes as a basis for improvement (quality enhancement).

To achieve these objectives, the test should focus on measuring knowledge and skills that programmes explicitly set out to develop in their students. This means abandoning claims to measure abstract general knowledge with no direct link to the programme and focusing on a) selected discipline-specific knowledge and skills and b) generic competencies that can realistically be developed in an undergraduate programme. The latter category might include critical thinking and problem-solving. These can theoretically be tested for using discipline-specific test items. Indeed, the authors of the recent outputs of the European COLOHEE project argue that generic competencies are best assessed using discipline-specific test items (CALOHEE, 2018[11]).

3. Improve the design of ENADE tests to ensure they generate more reliable information on learning outcomes that can also be used by teachers and HEIs

If maintained, ENADE tests should be designed in a more rigorous way to ensure that they are of comparable levels of difficulty within subjects from one year to the next and that tests for different disciplines are of equivalent difficulty for equivalent qualifications (bachelor’s, advanced technology programme, etc.). This may require a shift from classic test theory to item-response theory.

As part of this process, performance thresholds and grades should be established clearly in advance. The objective to be to provide students with easily understood grades and programmes with easily understood and usable grade-point averages and grade distributions. The approaches to both test design and performance thresholds used by CENEVAL in Mexico or testing organisations in the United States could provide valuable inspiration on how a revised form of the ENADE tests could be developed. It is important for INEP to draw on the expertise of other organisations involved in standardised testing internationally in the development of new approaches and test formats, to ensure it benefits from a wide range of expertise.

4. Explore ways to make the results of ENADE matter for students

If maintained, ENADE needs to be made into a higher stakes exam for students, so that they make an effort to actually demonstrate the level of knowledge and skills they possess. Currently, it is difficult to make ENADE results count towards individual’s degree scores, not only because of institutional autonomy, but because only every third cohort has to take ENADE and including ENADE in degree results may be perceived as unfair to students in a year where the test exists.

At the least, the ENADE score could be included in the students’ diploma supplement. Alternatively, ENADE could be made into a curriculum component for the years in which, or – in the case of sampling - for the students to whom, it is administered with the requirement that an equivalent test for students in other years be administered by institutions. It is not yet clear if this would be possible legally.

5. Introduce a new indicator dashboard, with a broader range of measures, to monitor programme performance and identify “at risk” programmes

The use of the Preliminary Course Score (CPC) cannot be justified in its current form for the reasons discussed above. However, systematic programme level data is a crucial tool for monitoring a system as diverse and variable in quality as Brazil’s. The most promising option would be to include a broader set of more transparent indicators in an ongoing monitoring system, with thresholds established to indicate “at risk” performances on different indicators. This information could then be used to inform regulatory decisions and feed into subsequent evaluation steps (such as on-site reviews). The system should apply to all programmes, with data obtained from institutions and other sources, as appropriate, and consolidated in a renewed version of e-MEC.

Such a system could use a more diverse set of indicators of teaching staff, real (not standardised) ENADE results (based on established performance thresholds), an indicator of drop-out rates and, when possible through linking data sources through the CPF number, information on employment rates and earnings. Indicators of the socio-economic profile of students could be included in the system, with higher tolerances for issues like drop-out or ENADE performance for programmes with intakes from lower socio-economic groups. Such variation in tolerances should be limited, as all students should be expected to reach minimum standards and all programmes maintain a certain proportion of their students.

A revised form of the IDD could potentially be maintained alongside the other indicators in the indicator dashboard, provided its status as a proxy for expected performance and its limitations are made clear, and its weight in the overall monitoring system is reduced. However, even a revised IDD is likely to remain a complex indicator that may not always be well understood by the public and stakeholders in the higher education system. The costs and benefits of maintaining such a comparatively un-transparent measure should be assessed. If, in the longer term, resources permit, a return to examination on entry and exit from programmes could be considered, although the costs and benefits of any such system should be considered carefully.

The OECD review team understands that INEP is already planning (October 2018) to “disaggregate” the components of the CPC and complement these with additional indicators to inform the regulatory process. Hopefully, this recommendation will support this process.

6. As part of a new system of institutional accreditation, exclude institutions with demonstrated internal quality assurance capacity from on-site programme reviews for the duration of their accreditation period

As discussed in Chapter 7 there is scope to exempt institutions from systematic programme level review that have a track record of good performance and that can demonstrate a high level of internal quality assurance capacity. This would require existing systems for institutional accreditation and re-accreditation to be strengthened. If problems are identified in relation to programme indicators in the indicator dashboard, such institutions would be responsible for addressing these issues internally. Addressing poor quality would become a key focus of institutional review and poor performance or failure adequately to address problems could lead institutions losing self-accrediting status in the subsequent round of institutional review. This move would reduce some of the burden of external programme-level reviews for renewal of recognition (as well as the initial recognition process).

7. Maintain programme-level supervision for other institutions, with targeted on-site reviews for poorly performing programmes and randomly selected highly performing programmes.

For the remaining institutions, programme-level review would be maintained. The new programme-level indicator dashboard (which would cover all programmes, including in self-accrediting institutions) would allow poor programmes to be identified and replace the current CPC system. If annually collected data on completion rates and employment outcomes were included in the dashboard, alongside input indicators and periodic results from a reformed ENADE, this would allow more effective continuous monitoring of programmes.

Problematic programmes could first be called upon to submit an improvement plan that could be assessed remotely, largely in line with current supervision procedures. SERES, or a future quality assurance agency (see below), could decide on timeframes for improvement and whether and when an on-site visit would be required. It is crucial that SERES, or a successor agency, have the capacity to close poor programmes rapidly if programme indicators fail to improve without clear justification and evaluators give a negative assessment following an on-site inspection.

However, while targeting of resources is important, the risk of evaluators only being exposed to poor quality programmes – and thus lacking good reference points – needs to be addressed. As such, it is recommended that reviewers also take part in reviews of randomly selected programmes that obtain good scores in relation to monitoring indicators – potentially including programmes in “self-accrediting” institutions - to allow them to gain more insights into the range of practices and performance that exists in their field in the country.

8. Develop a separate evaluation instrument for on-site reviews of established programmes

The current process for on-site reviews of established undergraduate programmes uses the same evaluation and judgement criteria as the instrument for programme recognition (which occurs when the first student cohort has completed between half and three-quarters of the programme). This instrument pays insufficient attention to programme outputs and outcomes (notably the results of (a revised) ENADE, attrition and graduation rates and employment outcomes) and to the teaching and student support practices that would be expected to have the greatest influence on these outputs and outcomes. A new instrument should thus be developed for on-site reviews of established programmes, which places most emphasis on these factors. The earlier suggestion for an inspectorate to examine infrastructure and basic institutional policies would mean that site visits by peer reviewers should focus exclusively on the learning environment and possible causes of poor outputs and outcomes.

References

[11] CALOHEE (2018), CALOHEE Outcomes Presented - CALOHEE, https://www.calohee.eu/ (accessed on 16 November 2018).

[7] CEDEFOP (2016), Application of learning outcomes approaches across Europe: A comparative study, Publications Office, Luxembourg, http://dx.doi.org/10.2801/735711.

[21] De Sousa, B. and J. De Sousa (2012), “Resultados do Enade na Gestão Acadêmica de Cursos de Licenciaturas: Um Caso em Estudo”, Estudos em Avaliação Educacional, Vol. 23/52, pp. 232-253, http://publicacoes.fcc.org.br/ojs/index.php/eae/article/view/1938/1921.

[19] EXAME (2017), Estudantes acertam menos de 50% das provas específicas do Enade (Students score less than 50% on discipline-specific ENADE tests), EXAME, https://exame.abril.com.br/brasil/estudantes-acertam-menos-de-50-das-provas-especificas-do-enade/ (accessed on 16 November 2018).

[18] Finney, S. et al. (2016), “The Validity of Value-added Estimates from Low-Stakes Testing Contexts: The Impact of Change in Test-Taking Motivation and Test Consequences”, Educational Assessment, Vol. 21/1, pp. 60-87, http://dx.doi.org/10.1080/10627197.2015.1127753.

[4] ICFES (2018), Información general del examen Saber Pro, Instituto Colombiano para la Evaluación de la Educación, http://www2.icfes.gov.co/estudiantes-y-padres/saber-pro-estudiantes/informacion-general-del-examen (accessed on 13 November 2018).

[27] INEP (2018), ENADE - Questionário do Estudante, Sistema ENADE, http://portal.inep.gov.br/questionario-do-estudante (accessed on 17 November 2018).

[24] INEP (2018), Relatório de Gestão do Exercício de 2017 do Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira - Inep.

[13] INEP (2017), Enade 2016 Resultados e Indicadores, Diretoria de Avaliação da Educação Superior - Daes, Brasília, http://download.inep.gov.br/educacao_superior/enade/documentos/2017/apresentacao_resultados_enade2016.pdf (accessed on 16 November 2018).

[3] INEP (2017), Instrumento de Avaliação de cursos de graduação Presencial e a distância - Reconhecimento e Renovação de Reconhecimento (Evaluation instrument for undergraduate programmes - classroom-based and distance - recognition and renewal of recognition), Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira, Brasília, http://www.publicacoes.inep.gov.br (accessed on 11 November 2018).

[25] INEP (2017), Nota Técnica No 3/2017/CGCQES/DAES (On the calculation of the CPC), Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira, Brasília, http://download.inep.gov.br/educacao_superior/enade/notas_tecnicas/2015/nota_tecnica_daes_n32017_calculo_do_cpc2015.pdf (accessed on 16 November 2018).

[14] INEP (2017), Nota Técnica Nº 32/2017/CGCQES/DAES (Methodology for calculating the Conceito ENADE), Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira, Brasília, http://download.inep.gov.br/educacao_superior/enade/notas_tecnicas/2016/nota_tecnica_n32_2017_cgcqes_daes_calculo_conceito_enade.pdf (accessed on 16 November 2018).

[26] INEP (2017), Nota Técnica Nº 33/2017/CGCQES/DAES (On calculation of the IDD), Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira, Brasília, http://download.inep.gov.br/educacao_superior/enade/notas_tecnicas/2016/nota_tecnica_n33_2017_cgcqes_daes_calculo_idd.pdf (accessed on 17 November 2018).

[12] MEC (2018), Diretrizes Curriculares - Cursos de Graduação (Curriculum Guidelines - Undergraduate Programmes), Ministério da Educação, http://portal.mec.gov.br/component/content/article?id=12991 (accessed on 06 November 2018).

[20] MEC (2018), e-MEC Instituições de Educação Superior e Cursos Cadastrados (Registered HEIs and programmes), http://emec.mec.gov.br/ (accessed on 14 November 2018).

[2] MEC (2017), Nota Técnica Nº 13/2017/CGARCES/DIREG/SERES/SERES (setting out parameters and procedures for the renewal of recognition of programmes), Ministério da Educação, Brasília, http://portal.mec.gov.br/index.php?option=com_docman&view=download&alias=64921-nt-13-2017-seres-pdf&category_slug=maio-2017-pdf&Itemid=30192 (accessed on 15 November 2018).

[6] MEC (2017), Portaria Normativa No 19, de 13 dezembro de 2017 Dispõe sobre os procedimentos de competência do Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira - INEP referentes à avaliação de instituições de educação superior, de cursos de graduação e de desempenho acadêmico de estudantes., Ministério da Educação, Brasília, http://www.abmes.org.br (accessed on 15 November 2018).

[16] Melguizo, T. and J. Wainer (2016), “Toward a set of measures of student learning outcomes in higher education: evidence from Brazil”, Higher Education, Vol. 72/3, pp. 381-401, http://dx.doi.org/10.1007/s10734-015-9963-x.

[9] NVAO (2016), Assessment framework for the higher education accreditation system of the Netherlands, Nederlands-Vlaamse Accreditatieorganisatie (NVAO), The Hague, https://www.nvao.com/system/files/procedures/Assessment%20Framework%20for%20the%20Higher%20Education%20Accreditation%20System%20of%20the%20Netherlands%202016_0.pdf (accessed on 17 November 2018).

[8] OECD (2013), Assessment of Higher Education Learning Outcomes Feasibility Study Report Volume 1 - Design and implementation Executive Summary, OECD, Paris, http://www.oecd.org/edu/ahelo (accessed on 16 November 2018).

[5] OECD (2018 (forthcoming)), The Future of Mexican Higher Education: Promoting Quality and Equity, http://Forthcoming.

[22] Oliveira, A. et al. (2013), “Políticas de avaliação e regulação da educação superior brasileira: percepções de coordenadores de licenciaturas no Distrito Federal”, Avaliação: Revista da Avaliação da Educação Superior (Campinas), Vol. 18/3, pp. 629-655, http://dx.doi.org/10.1590/S1414-40772013000300007.

[28] Presidência da República (2017), Decreto Nº 9.235, de 15 de dezembro de 2017 - Dispõe sobre o exercício das funções de regulação, supervisão e avaliação das instituições de educação superior e dos cursos superiores de graduação e de pós-graduação no sistema federal de ensino. (Decree 9235 of 15 December 2017 - concerning exercise of the functions of regulation, supervision and evaluation of higher education institutions and undergraduate and postgraduate courses in the federal education system), http://www.planalto.gov.br/ccivil_03/_Ato2015-2018/2017/Decreto/D9235.htm (accessed on 10 November 2018).

[1] Presidência da República (2004), Lei no 10 861 de 14 de Abril 2004, Institui o Sistema Nacional de Avaliação da Educação Superior – SINAES e dá outras providências (Law 10 861 establishing the National System for the Evaluation of Higher Education - SINAES and other measures), http://www.planalto.gov.br/ccivil_03/_ato2004-2006/2004/lei/l10.861.htm (accessed on 10 November 2018).

[23] Santos, M. et al. (2016), “O impacto das avaliações disciplinares no ensino superior”, Avaliação: Revista da Avaliação da Educação Superior (Campinas), Vol. 21/1, pp. 247-261, http://dx.doi.org/10.1590/S1414-40772016000100012.

[29] SIAFI (2018), Siafi - Sistema Integrado de Administração Financeira do Governo Federal - Main Expenditures INEP, Siafi - Sistema Integrado de Administração Financeira do Governo Federal.

[15] TCU (2018), RA 01047120170 Relatório de auditoria operacional. Atuação da Secretaria de Regulação e Supervisão da Educação Superior do Ministério da Educação - Seres/Mec e do Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira - Inep nos processos de regulação, supervisão e avaliação dos cursos superiores de graduação no país. (Operational audit report: Activities of SERES/MEC and INEP related to regulation, supervision and evaluation of higher education programmes in the country), Tribunal de Contas da União, Brasília, https://tcu.jusbrasil.com.br/jurisprudencia/582915007/relatorio-de-auditoria-ra-ra-1047120170/relatorio-582915239?ref=juris-tabs (accessed on 13 November 2018).

[10] UKÄ (2018), Guidelines for the evaluation of first-and second-cycle programmes, Universitetskanslersämbetet (Swedish Higher Education Authority), Stockholm, http://www.uka.se (accessed on 17 November 2018).

[17] Wolf, L. and J. Smith (1995), “The Consequence of Consequence: Motivation, Anxiety, and Test Performance”, Applied Measurement in Education, Vol. 8/3, pp. 227-242, http://dx.doi.org/10.1207/s15324818ame0803_3.

Notes

 1.  Every three years is the legal minimum frequency, established by the 2004 legislation, for applying ENADE to every programme

 2.  The CPC is not “preliminary”, as each programme already has another score – the Conceito de Curso (CC) – resulting from the initial recognition process that all programmes must undergo. Moreover, when programmes are not subject to further on-site reviews, the CPC is not replaced by a “final” score, but rather exists alongside the initial CC awarded to the programme.

 3.  Critical reading; quantitative reasoning; citizen competencies; written communication; and English

 4.  The relevant regulations provide for a limited number of acceptable reasons for being excused from ENADE

 5.  14 at Bachelor level and 13 Advanced Technology Programmes.

 6.  In order for a programme to have the IDD calculated, it must have a minimum of two graduating students in ENADE with information in the ENEM database dating from the previous three years. It also needs 20% of the total number of participating ENADE students to have data in the ENEM database.

 7.  Point III.5 of the latest INEP Technical Note on the use of evaluation in the regulatory process states: “Other already recognised courses that have achieved a satisfactory result (CPC ≥ 3) in the CPC of the reference year 2015, which are not included in the situations described in the previous paragraphs: The recognition renewal process shall be opened by the Ministry of Education and the act shall be issued in sequence without the need for a manifestation by the HEI and without any [additional] formality.” (MEC, 2017[70])

6.

Assuring the quality of postgraduate education

This chapter focuses on the mechanisms used for the external quality assurance of academic postgraduate education in Brazil. Brazilian postgraduate education comprises “stricto sensu” programmes, with a strong academic and scientific focus, and vocationally oriented “lato sensu” programmes, such as Masters of Business Administration (MBA). “Stricto sensu” programmes are subject to a specific system of quality evaluation and regulation, implemented by the Foundation for the Coordination of Improvement of Higher Education Personnel (CAPES). The chapter analyses these processes, examining the systems in place to evaluate new courses, to allow them to enter the National Postgraduate System, as well as the periodic programme reviews that are undertaken every four years. Based on the strengths and weaknesses identified, the chapter provides recommendations for fine-tuning the system and planning for the future.

6.1. Focus of this chapter

A focus on academic postgraduate provision

This chapter focuses on the mechanisms in place at national level in Brazil to assure the quality of postgraduate education in the country. As noted in Section 3.4, Brazilian postgraduate education falls into two distinct categories. Courses with a strong academic and scientific focus, which include Master’s degrees (mestrado acadêmico), Professional Master’s degrees (mestrado profissional) and doctoral education (doutorado), are classified as “stricto sensu” postgraduate provision and form part of the National System of Postgraduate Education (SNPG). They are subject to a specific system of quality evaluation and regulation, implemented by the Foundation for the Coordination of Improvement of Higher Education Personnel (Coordenação de Aperfeiçoamento de Pessoal de Nível Superior, CAPES), a public foundation under the responsibility of MEC. In parallel, many higher education institutions offer professionally oriented, postgraduate “specialisation” programmes, including Master’s in Business Administration (MBA), which are classified as “lato sensu” provision and are not subject to external programme-level quality assurance as part of SINAES or organised by CAPES¹. The focus in this chapter is on the quality assurance processes for stricto sensu postgraduate provision.

Academic Master’s degrees are still viewed as research degrees

The existence of a separate, long-established and highly developed system of external quality assurance for postgraduate education programmes is a distinctive feature of Brazilian higher education, reflecting the historical development of the science base in the country. In many OECD countries, external quality assurance of Master’s courses is undertaken by the agencies responsible for supervision of undergraduate education, often with the implicit understanding – in Europe notably - that Master’s degrees are a vehicle to deepen knowledge and skills gained at undergraduate level and a requirement for a wide variety of jobs in the economy. In Brazil, in contrast, stricto sensu Master’s courses – including so-called “Professional Master’s” – are widely understood as the first stage in an academic or research career – a situation that is largely a reflection of the relatively recent expansion of doctoral education in the country.

A system with strong external regulation of doctoral programmes

Responsibility for the quality of doctoral training in many higher education systems internationally has been left to individual universities, with limited intervention from public authorities (European University Association, 2018[1])². In such cases, incentives and signals relating to the way programmes are organised are (increasingly) provided in an indirect way by public research funding agencies, through the criteria used to award doctoral training or research grants. The more direct approach adopted by public authorities in Brazil reflects a long-standing concern to expand the population of highly qualified researchers in the country, as a means to boost domestic scientific and innovation capacity. In practice, as discussed below, aspects of the evaluation system implemented by CAPES to ensure the quality of postgraduate training share characteristics with assessments undertaken in other higher education systems to monitor the research performance of higher education institutions.

6.2. Strengths and weaknesses of the current system

CAPES evaluates and regulates market entry of new courses and oversees the periodic evaluation of established programmes

The system of external quality assurance for academic postgraduate education in Brazil began in its current form in 1998. It comprises two distinct processes:

1. Evaluation of proposals for new courses as a basis for regulating the entry of postgraduate training to the system and;

2. Periodic evaluations of established postgraduate programmes, currently undertaken on a four-year cycle, allowing their continued operation (permanência) or, in case of poor performance, leading to their closure.

CAPES is responsible for coordinating the evaluation process. The evaluation of courses and programmes is undertaken by selected academic peers from the same scientific field working in field committees (Comissões de Área), with the scores attributed by the field committees to proposals and existing programmes subsequently approved (or adjusted) by the Technical and Scientific Council for Higher Education (Conselho Técnico e Científico da Educação Superior - CTC-ES) composed of academics from all knowledge areas.

A distinction between programme (programa) and course (curso)

CAPES uses the Portuguese terms programa (programme) and curso (academic course or programme) in a specific way. A programa comprises the staff, infrastructure and activities associated with the provision of postgraduate education in a specific field, whether at Master’s level, doctoral level or both. It is the principle unit of analysis for the periodic evaluations of postgraduate provision. Of the 3 472 stricto sensu academic postgraduate programas evaluated in the most recent CAPES four-year review in 2017, just over 60% combined both Master’s and doctoral provision, 37% involved only Master’s provision and 2% involved only doctoral provision (CAPES, 2017[2]). The term curso is used to refer to a single course of study at a particular level: a Master’s, Professional Master’s or doctorate. A programa may thus contain two cursos (a Master’s and a doctorate) or effectively be synonymous with a curso, when only one type of curso is provided. For the sake of clarity, this chapter uses the English terms programme and course, to allow a distinction to be made where necessary.

The sections that follow review, in turn, the strengths and weaknesses of the CAPES evaluation processes for approving new courses (Avaliação de Propostas de Cursos Novos, APCN) and periodic review of established programmes (Avaliação Quadrienal).

CAPES: approval of new postgraduate courses

Relevance: rationale and objectives of the current system

In Brazil, academic postgraduate education in all types of higher education institution³ is collectively considered as part of a National System of Postgraduate Education (SNPG), which – as a system - has the aim of training highly quality teaching and academic staff; training (highly) qualified staff for non-academic sectors and strengthening the country’s scientific, technological and innovation capacity more generally. CAPES states that the evaluation system it operates is designed to certify the quality of postgraduate education in the country, as a basis for allocation of publicly funded scholarships and research funding, and to identify regional disparities strategic knowledge areas as a basis for strategic actions to address such gaps (CAPES, 2018[3]). As such, the CAPES evaluation system as a whole serves at least three purposes:

1. It is a mechanism for ensuring the quality of postgraduate training (and thus – in theory - the quality of the human resources trained) as a form of guarantee for students and their future employers;

2. The results of the evaluation process (in particular the periodic reviews) provide objective criteria for the allocation of public funding for researcher training (notably grants to Master’s and doctoral students) and research projects (with the implicit expectation that the high-quality programmes identified will make good use of public resources);

3. The results of the evaluation also identify how well the country is developing research capacity in different scientific fields and across the territory of the Union, allowing corrective policy measures to be developed as necessary.

The specific approval process for new courses (APCN) is designed to ensure only academic teams with demonstrated expertise, a proven track record of quality research and adequate facilities are authorised to provide academic postgraduate education. Course proposals are assessed by a field committee composed of academic peers from the field in which the course seeks to operate. Following a standard assessment and validation process, new courses are formally approved if they score at least three on a nominal scale of one to five⁴, taking into account a range of variables discussed below. Only once approved can courses recruit students; obtain national recognition for their diplomas, and obtain funding from CAPES for student scholarships and institutional capacity building⁵.

The APCN process consciously sets a comparatively high bar for entry into the system of academic postgraduate training and for the creation of doctoral training provision in programmes that already operate a Master’s level. In so doing, it seeks to maintain high minimum standards for postgraduate education, protect students against poor quality provision and ensure efficient targeting of public funding. During the review visits, the OECD team noted a high degree of support for the principle of maintaining a high threshold for entry into the academic postgraduate education system.

Effectiveness: quality indicators used or generated

CAPES evaluations of postgraduate courses and programmes rely to a large extent on qualitative assessments undertaken through peer review. These qualitative assessments, which may take into account quantitative data, are ultimately translated into a single score (conceito) attributed to courses and programmes. Standing evaluation committees composed of Brazilian academics in specific disciplines are established for 49 scientific fields. The work of each committee is coordinated by a “field coordinator”, elected by the academic programmes in the field in question for a renewable term of four years. The field coordinator and the members of the field committee undertake their work in the CAPES evaluation processes on a voluntary basis, alongside their main academic jobs.

The field committees are responsible for assessment new courses and undertaking the periodic reviews (discussed below). Following initial eligibility checks by CAPES staff, the relevant field committee assesses proposals for new courses using a standard set of criteria for the field. The topics assessed follow a standard model developed by CAPES, but with specific assessment and judgement criteria, and the weighting of individual variables, tailored to each field by the field committee in question. The field committee collectively assesses proposals in relation to the assessment topics and assessment criteria to provide ratings for each topic on a five-point scale from “very good” to “deficient”. The ratings for the different topics are combined to generate an overall assessment score for the programme. Programmes that are finally approved are attributed an initial default CAPES rating (Conceito CAPES) of three out of five (three being the minimum quality threshold required). Programmes are only attributed higher scores following a periodic review.

As illustrated in Table 6.1, the evaluation criteria for academic courses (academic Master’s and doctorates) for new course proposals include the relevance of the new course to national and institutional development; the design and proposed scale of the course; the qualifications and scientific output of the staff involved and their planned involvement in the course; and available infrastructure. The same criteria are also used to evaluate proposals for new Professional Master’s courses although with specific evaluation criteria modified to take account of the more practical orientation of these courses and their closer links to the world of work. CAPES regulations (CAPES, 2017, p. art.6[4]) specifically highlight that professional programmes are likely to require a different staff profile. They recommend giving weight to the professional experience of teaching staff, even if they do not hold a PhD, and reducing the emphasis on staff being full-time, given that many will teach alongside other professional roles.

Table 6.1. Criteria for evaluation of new postgraduate course proposals
| Criteria
Fit with institutional development plan | Consistency of proposal to the Institutional Development Plan (PDI) of the proposer and commitment of the institution's leaders to the initiative
Consistency of proposal and programme design | Staff qualifications
Relevance of ongoing research activities by the team proposing the programme
Appropriateness of curriculum structure to subject of programme
Clarity of proposal in relation to students and planned graduate profile | Selection criteria for students
Number of study places (vagas)
Fit of graduate profile with national priorities and needs
Academic capacity | Evidence that the team proposing the programme has academic, didactic, technical and / or scientific competence and qualifications linked to the objective of the proposal
Permanent staff | Demonstration that an adequate number of permanent staff with exclusive dedication are allocated to the programme and will be able to deliver the type and volume of training proposed.
Scientific output of staff | Indication of a maximum of five research outputs for each permanent staff member for the last five years
Infrastructure | Adequacy of the educational and research infrastructure: physical facilities, laboratories, experimental facilities and library
Adequacy of computer equipment, network access and multimedia information sources for teachers and students
Adequacy of secretarial infrastructure and administrative support
Source: Adapted from Portaria CAPES nº 161/2017, article 4 (CAPES, 2017[4])

The field committee assesses these dimensions based on the electronic application from the proposing institution (submitted through the CAPES Sucupira platform), with the option to request a site visit if considered necessary. The OECD team understands that site visits as part of initial course approval, although formally provided for, are comparatively rare in practice⁶.

The criteria examined in the process for approval of new courses cover a wide range of the variables that might reasonably be expected in an ex-ante assessment of a proposed postgraduate programme. The criteria focus on most key factors that might be expected to contribute to the quality of the future training provided. Nevertheless, there is scope to review – and certainly to substantiate better - the prominence and weight attributed in the evaluation template to the different factors considered.

The current evaluation system attaches considerable weight to the status and intellectual outputs of the staff who will be involved in the proposed course. This is entirely consistent with the objective of checking that adequate conditions are in place to allow students to have access to knowledgeable teachers and mentors and receive their training in an environment where high quality research is undertaken and valued. This reflects a model of academic postgraduate education that views research culture and peer effects among individuals involved in research as key contributors to the learning and scientific development of students. In placing such emphasis on these factors, however, there is a risk that other variables affecting the quality of the training offered are attributed too little attention, particularly at Master’s level and in professionally oriented programmes.

The key concerns of the OECD team relate to the comparatively limited attention attributed to, first, the relevance of new courses to national or regional needs and developing knowledge areas and, second, the design of the training programme, and support and personal development opportunities offered to students.

Under the section dealing with students, the existing standard evaluation template for new courses includes an assessment of relevance of the “graduate profile” the proposed course is intended to generate – in other words, the types of knowledge and skills graduates are expected to possess (this is also a consideration in the evaluation instruments used in SINAES (INEP, 2017[5]). However, this fundamental issue is considered under the same broad heading as practical issues like student selection and the number of study places. Although the coherence of the proposed course with the Institutional Development Plan (PDI) of the host institution is assessed, there is no explicit assessment of the relevance of course to the needs of Brazil, in terms of knowledge development and highly qualified human resources. There is no obvious place in the current framework where the contribution of new courses to new or emerging fields of knowledge is assessed.

These problems are compounded by the fact that assessments are carried out exclusively by academic staff from a specific discipline, using largely traditional measures of academic performance. While academics in a given field may be expected to have a good understanding of the developments in that field in an international context, particularly in less applied areas, they may, understandably, have less understanding of how knowledge and skills in the field can contribute to national development goals or respond to societal challenges. There is scope to include more perspectives from non-academic bodies in this aspect of the assessment.

Similarly, although five of the 49 academic fields are nominally classified as inter-disciplinary (biotechnology, environmental science, education, material science and “inter-disciplinary”), the strong focus on traditional disciplines and scientific output in these disciplines may create barriers to new courses in innovative fields of study that may ultimately be important for the future of Brazil’s postgraduate training system. The risks of working in disciplinary silos are by no means unique to Brazil, but do warrant further attention in the way CAPES evaluations are structured and organised.

The second key issue that deserves greater attention in the assessment of new programmes is organisation of training and support for students. At present, the CAPES evaluation template includes an assessment of the “appropriateness of curriculum structure to subject of programme”, but little obvious room to assess how the training programme will help to develop students’ knowledge and skills and monitor their progress. Across the OECD, higher education institutions and research funding bodies have increasingly focused on developing postgraduate training with a greater explicit focus on helping students to acquire research skills and transversal competences (in collaborative working, communication, project management, entrepreneurship, for example) that they can exploit subsequently in a wide range of settings. Evaluation systems in other higher education systems do place more emphasis of these issues⁷.

In the discussion of quality indicators, it is important to acknowledge that the current CAPES assessment system has developed distinct criteria to be applied in the evaluation of Professional Master’s courses. In particular, the criteria for this type of course take account of the different staff profile required to successfully implement more applied forms of training. Between 2010 and 2017, the number of Professional Master’s courses in Brazil increased from 247 to 703, suggesting that the authorisation system is functioning for this type of provision. However, developing appropriate quality criteria for applied research and postgraduate programmes has proved challenging in all OECD higher education systems and there is certainly scope for ongoing mutual learning. Within Brazil, it is important to monitor the implementation of existing Professional Master’s programmes, to discuss strengths and challenges with programmes, students and industry and public sector partners and to ensure lessons learnt feed back into the evaluation indicators used.

A final consideration about the indicators used in assessment of proposals for new courses is the absence of an explicit requirement for a course or programme development plan with measurable, time-bound targets. Requiring programmes to develop such a plan and establish clear targets would create a useful reference for subsequent periodic reviews.

Effectiveness - division of responsibilities

A defining characteristic of the CAPES evaluation system is the strong role of academic peers in both defining the evaluation criteria and undertaking programme evaluations. Field coordinators and committees have leeway to adapt commonly agreed evaluation templates to the requirements of their specific fields, by defining field-specific assessment criteria and adapting weightings between broad evaluation criteria. In practice, field committees stick very closely to the standard evaluation template, but adjust specific evaluation criteria for individual topics. The strong involvement of the academic community in both policy-setting and implementation, as well as the flexibility afforded to field committees in the process, have contributed to the widespread acceptance of the CAPES evaluation system and a shared sense of “ownership”. This contrasts with the evaluation processes for undergraduate programmes implemented as part of SINAES, which are widely perceived as top-down.

Despite the strengths of the current division of responsibilities within the CAPES evaluation system, the operation of an evaluation system that relies heavily on the voluntary contribution of academic staff organised in discipline-specific field committees is not without problems.

First, there is the practical issue of the availability and commitment of academic peers. Although academics involved in the CAPES evaluation process consulted by the OECD review team felt the time and effort required of them for the current system for approval of new programmes remained reasonable, they highlighted that the CAPES system as a whole is becoming unmanageable for field committees, as the number of postgraduate programmes increases. We return to this issue in the discussion of the four-year reviews below.

Another potential risk with the current system is that authorisation to start a new academic postgraduate programme depends to large extent on the opinion of academics who work in “rival” postgraduate programmes in the same field and who may have an interest in restricting expansion of provision to limit competition for students and research funds. In practice, the OECD review team found no evidence that this potential conflict of interests has led to any undue restrictions on the creation of new programmes. The number of postgraduate programmes has increased considerably over the last decade. Moreover, evaluation criteria for new proposals are clear, field committees need to justify their evaluation scores in detail, the final evaluation score is validated by CAPES’ inter-disciplinary Technical and Scientific Council, and transparent procedures exist for proposing institutions to appeal against decisions.

More seriously, as highlighted above, the reliance on disciplinary committees composed exclusively of Brazilian academics risks creating an excessively narrow academic focus in evaluations. While scientific excellence and traditional measures of academic output remain the basis for postgraduate education, it is important to complement assessment of this basis with perspectives from outside academia, to ensure the development of postgraduate education responds to broader national and regional needs. Equally, as highlighted above, it is crucial that there is room for innovation in the definition of study fields and the way programmes are implemented.

On a practical level, the current process for the evaluation of new courses involves limited or no direct interaction between those proposing the new courses and those evaluating the proposals. This may be justified by the limited availability of time and resources and a desire to ensure the evaluation is independent and transparent. Nevertheless, the CAPES evaluation system is notable for being largely “paper-based”. Other quality assurance systems tend to employ site visits, or at least, as in the Programa Nacional de Posgrados de Calidad in Mexico, an interview with the course coordinator as part of the initial authorisation or accreditation process (CONACyT, 2015[6]).

Effectiveness: use and effects

As noted above, a successful CAPES evaluation is a pre-requisite for all new academic postgraduate courses to begin operation. On passing the initial evaluation process, new courses are attributed a provisional evaluation rating of three out of five. On this basis, they have access to CAPES funding for capacity building and student scholarships. Funding for grants is allocated by CAPES to the programme, which is then responsible for awarding scholarships to students.

The results of the entry evaluation for new courses are made public on the CAPES website and are used by courses in their marketing and student recruitment processes. For understandable reasons, the approval of new courses, which are evaluated in varying numbers every year, does not attract as much public attention as the results of the four-year periodic evaluations discussed below and which cover the entire stock of postgraduate programmes.

Efficiency and cost-effectiveness

Academics involved in the CAPES field committees consulted by the OECD review team tended to indicate that the time and financial resources invested in the evaluation of new courses were proportionate to the goals of the system and remained manageable in light of the average number of new proposals received annually. Although the costs associated with initial approval of postgraduate courses have not been made available to the OECD at the time of writing, the absence of systematic review visits and the use of academic field committees who work on a voluntary basis clearly limit costs for the Brazilian State.

The Review team understands that no assessment of the value of the time dedicated to evaluation of new courses by academic staff in the field committees – and thus also the cost to their home institutions - is currently available. Given the comparatively rapid rate of expansion of postgraduate provision in Brazil in recent years and the related increase in the number of proposals for new courses, it will be important to develop a better understanding of the number of person-hours used in the evaluation process and the associated costs.

Given the concern in Brazil to maintain a high quality-threshold for entry of new courses to the academic postgraduate system in the country, the existing system of systematic peer review for all new programmes appears to be appropriate in the current Brazilian context. In the longer term, as the scale of the postgraduate system continues to evolve, it may be desirable (or necessary) to move away from programme-level initial accreditation to allow institutions that meet specific conditions and have adequate institutional quality assurance processes to launch academic postgraduate programmes under their own authority. Such models of institutional self-accreditation exist in many mature higher education systems, although the OECD team recognise that such an approach may not yet be appropriate for an expanding system such as that in Brazil.

CAPES: four-year programme reviews

Relevance: rationale and objectives of the current system

Every four years⁸, CAPES implements a comprehensive evaluation of all academic postgraduate programmes that have already been accredited and been in operation sufficiently long for students to have produced academic results. Although the specific objectives of this process are not formulated very explicitly in the relevant secondary legislation, the four-year reviews appear to fulfil a double role:

They provide a means to ensure postgraduate programmes (continue to) meet at least minimum defined quality standards, as programmes scoring less than three out of five lose CAPES funding and the national validity of their diplomas, and;

The reviews provide an incentive for programmes to strive for improvement – as measured by the defined criteria – as programmes can obtain a higher score (than that awarded in the initial approval process or in the previous round of periodic reviews) and thus greater prestige and, potentially, greater funding.

From a quality assurance perspective, the reviews are in practice very much focused on external assessment and ensuring accountability, with limited or no focus on supporting programmes to improve (quality enhancement).

Effectiveness: quality indicators used and generated

As for the evaluation of proposals for new courses, the four-year reviews are coordinated by CAPES, but undertaken by the 49 field committees under the leadership of their field coordinator. The field committees draw on information on staff, students, graduates and details of scientific outputs reported by each postgraduate programme through the online Sucupira platform as a basis for their assessment of each programme. As in the case of the assessment of new courses, field committees use a standard evaluation grid which they adapt to the specificities of their field, in particular through formulating specific evaluation criteria for each topic and adjusting the weights between topics.

As seen in Table 6.2, the assessment includes a review of the programme proposal and its relevance, although this is not attributed any points in the final score. The criteria relating to staff are similar to those used in the evaluation of new courses, but verified using data from Sucupira. Similarly, most of the criteria relating to students are based on quantitative data reported by the programmes.

The quality of student publications (including published dissertations and theses) and the quality of the academic output of staff in academic journals are assessed using a standard classification of publication “vehicles” (from international peer-reviewed journals to university online publications), recorded in an online database called Qualis. Part of the work of each field committee each year is to review an established classification of publication vehicles relevant for their field and attribute a “quality rating” on a seven-point scale (A1, A2, B1, B2, B3, B4, B5), where A1 typically includes the most prestigious international journals in the field with high impact factors. Citation impact, assessed through mechanisms such as the Scopus database and citation scores such as the h-index⁹, is a significant criterion in the rating of journals in Qualis in many CAPES fields. However, while the use of impact factors is well-established, but not uncontested, in the hard sciences, there is an ongoing debate in Brazil, as in other countries, about the extent to which such measures capture the impact and relevance of work in the social sciences, humanities and arts.

The development of the Qualis classification database means that publications produced by each programme (and reported in Sucupira) are automatically attributed a quality rating on the basis of the assigned rating of the publication vehicle used. As the Qualis classification is undertaken for each field, the same journal may have a different rating in different fields. Individual fields have also developed Qualis-like rating systems for artistic and technical outputs, although these systems are less well established and more complex to implement.

Table 6.2. Periodic programme evaluation: criteria for academic programmes
| Criteria | Weighting (range)
Programme proposal | 1.1. Coherence, consistency, comprehensiveness and ‘currentness’ (atualização) of the priority research fields, lines of research, projects in progress and curricular proposal. | 0
1.2. Future planning for the programme taking into account challenges for the knowledge field in terms of knowledge production, training, social engagement and the destinations of graduates.
1.3. Infrastructure for teaching, research and outreach / engagement
Academic staff | 2.1. Profile of the academic staff, considering levels of qualification, diversification in the origin of training, ongoing training and experience and the compatibility of these with the programme proposal. | 10-20 %
2.2. Adequacy and time commitment of permanent teachers to research activities and the training programme.
2.3. Distribution of research and training activities among the staff involved in the programme.
2.4. Contribution of programme staff to undergraduate teaching and / or research activities, paying attention to the repercussion that this item may have on the training of future participants in the postgraduate programme (only when there is a direct link with an undergraduate programme).
Students, theses and dissertations | 3.1. Number of theses and dissertations defended in the evaluation period, in relation to the number of permanent teaching staff and the size of the student body. | 30-35%
3.2. Distribution of the focus of theses and dissertations defended in relation to the profile of teaching staff.
3.3. Quality of theses and dissertations and contribution of the academic output of undergraduate (if the HEI has undergraduate courses in the area) and postgraduate students to the overall output of the programme, as measured by publications and other indicators relevant to the field.
3.4. Efficiency of the programme in the training students: time taken for graduation of Master’s students and doctoral candidates.
Scientific outputs | 4.1. Quality rated publications by permanent staff member. | 35-40%
4.2. Distribution of quality rated publications in relation to the permanent teaching staff of the programme
4.3. Technical output, patents and other outputs considered relevant.
4.4. Artistic outputs, in areas where such output is relevant
Social engagement and impact | 5.1. Insertion and regional and (or) national impact of the programme. | 10-20%
5.2. Integration and cooperation with other research and development programmes and professional development related to the area of knowledge of the programme, with a view to the development of research and postgraduate studies.
5.3. Visibility or transparency given by the programme to its performance.
Source: CAPES (2017) Regulamento para a Avaliação Quadrienal 2017 (2013-2016) Programas acadêmicos e profissionais. (CAPES, 2017[7])

In contrast, no standardised classification system exists for books or book chapters published by academic staff (or students) in programmes. This means, for fields where books are a major vehicle for intellectual output, field committees have to assess books and book chapters individually. Typically, programmes are invited to identify books and book chapters that they believe meet particular quality criteria established by the field committee and these are then all reviewed quickly. Books and book chapters identified as having particular merit are then read in full by members of the field committee. The OECD team understands that the assessment of books and book chapters represents one of the largest calls of the time of members of some field committees (notably in the humanities, social sciences and some of the hard sciences).

The social engagement and impact (inserção social) of programmes is reviewed in a qualitative fashion on the basis of documentary evidence submitted by programmes. Some of the field committees examine the destination of graduates under the topic of ‘Insertion and regional and (or) national impact of the programme’. However, it is not clear how this assessment is made and whether it is based on systematic surveys of graduate destinations.

Considered in the round, the set of indicators used in the CAPES four-year evaluations covers many of the key variables that would widely be assumed to contribute to high quality postgraduate provision. It is positive that the evaluation grid, under different headings, takes into account factors such as staff-to-student ratios, time to graduation and cooperation networks with external research and non-academic organisations, for example.

However, the most striking feature of the four-year reviews is the strong focus on the scientific output of the academic staff involved in the programmes being evaluated. As noted earlier in this chapter, the presence of competent researchers is crucial to the capacity of programmes to share subject knowledge and research expertise with students and create an environment conducive to the students undertaking their own high-quality research. However, the CAPES evaluation is – nominally at least – an evaluation of postgraduate training programmes, not a research performance evaluation like the Research Excellence Framework (REF) used in the United Kingdom. As such, it is questionable why the system does not allocate less weight and fewer resources to assessing the performance of staff and more to assessing the performance of students and outcomes of graduates.

The current system does attempt to measure the quality of dissertations and theses and other papers and outputs of students. However, this assessment is in most cases based on proxies provided by a notional quality rating attached to the “vehicles” in which the dissertations or theses (or derivatives thereof) are published. It is questionable whether it is reasonable to expect Master’s students or even doctoral candidates to be publishing outputs in journals at a similar level to established academic staff. While postgraduate students do publish in high quality academic publications, they are typically in a minority in most established higher education systems. Other forms of publication – such as non-peer-reviewed online journals - do not necessarily provide a reliable guarantee of quality.

There is no easy solution to these problems in a system like the current CAPES four-year reviews. Quality assurance systems which rely on on-site review visits (principally at the Master’s level), do sometimes involve a qualitative review of a sample of student dissertations. Other systems rely on other mechanisms to ensure the quality of postgraduate student outputs – essentially placing trust in standard processes. Several English-speaking countries rely heavily on external marking (by academic peers) of papers and dissertations at Master’s level, as a means to assure quality across the system. Many systems – including Brazil – insist doctoral theses are peer reviewed and finally approved by defence panels composed of leading academics in the field.

As noted, there is also some attempt in the current CAPES system to assess the destinations of graduates from programmes. However, on the basis of available evidence, this aspect of performance is not currently addressed adequately. The ability of graduates from programmes to find relevant employment in, or outside, the academic sector and draw on their skills must be assumed – in part at least – to reflect the quality of the training they have received. It would be desirable to further develop systems in Brazil to allow graduates to be tracked and to include graduate outcomes more prominently in the postgraduate training evaluation system.

A final issue that deserves attention in this discussion of indicators is the way in which field committees identify and assess programmes deemed to be of international quality or excellence, with strong internationalisation and international engagement (inserção internacional). These are attributed CAPES scores of 6 or 7 and subsequently have access to additional resources. Each field committee is responsible for establishing transparent criteria for allocating these top scores. In all cases, programmes are initially scored on a scale of one to five, and then programmes with doctoral provision that score five (that score ‘good’ or ‘very good’ on all other criteria) are assessed against additional criteria understood to indicate international excellence. Common criteria include the amount of external research funding attracted by the programme, the number and intensity of international cooperation and the proportion of outputs published in international journals.

Given the assumed link between internationalisation and academic excellence, the principle of making achievement of the highest scores for academic postgraduate programmes dependent on objective measures of international activity appears sound. Although the rigour and appropriateness of the indicators used to measure internationalisation may vary between fields, the types of measures used appear generally to be appropriate, objectively measurable and comparable to indicators of internationalisation used in other higher education systems. In the 2017 four-year CAPES evaluation, 184 programmes (5.3% of academic programmes) achieved a score of seven and 298 (8.6%) programmes achieved a score of six. It is credible that a higher education system of the size and maturity of Brazil’s would have such numbers of programmes that could be considered of high quality in an international context. As discussed below, there is scope to bring more international perspectives into the assessment of quality and the determination of whether programmes genuinely deliver international standards of excellence.

Effectiveness: division of responsibilities

As discussed earlier in this chapter, the reliance of CAPES on peer review is both a strength, for the acceptance and credibility of the system in the academic community, and a potential risk factor, as the scale of the postgraduate training system in Brazil expands and increases the burden of undertaking peer evaluations. CAPES has hitherto been successful in attracting and obtaining the commitment of well-regarded Brazilian academics to work as part of its field committees, including for the four-year reviews. However, some of the academics involved in CAPES evaluations interviewed by the OECD review team expressed concern that it was becoming harder to engage academics and – from a purely logistical perspective – the system of peer review as currently configured is no longer sustainable.

A second key issue in the current staffing of CAPES evaluation processes is the risk of endogamy (inbreeding). Even in a country size of Brazil – particularly given the relatively small size of its postgraduate training system – the number of established academics in a given field of study is limited. The number working in very high-quality departments and programmes at an international level is even smaller. As such, there is the risk that the people making judgements on whether or not a given programme is of international standard have close connections with the programmes they are judging. It is likely that their appreciation of the relative merits or deficiencies of the programme is conditioned by the tight-knit academic community of which they are a part. Moreover, the comparatively small pool of evaluators and their background may lead the evaluation process to reward programmes that reproduce existing models of education, rather than innovate.

It would be beneficial to bring the perspective of international peers into the CAPES evaluation process, particularly for assessment of those programmes judged to be of international standing. A large-scale involvement of international peers in assessment would almost certainly be impractical because of the costs involved, the difficulty of securing participation and language issues. Nevertheless, some targeted involvement of academic peers, including through electronic communication, from outside the country may be feasible.

Effectiveness: use and effects

The results of the four-year CAPES evaluations have far-reaching effects. For programmes that fail to meet the minimum quality standard of three out of five, the evaluation essentially leads to the closure of the programme. Programmes failing to achieve a score of three lose their right to CAPES funding and see the national validity of their diplomas withdrawn. It is understood that students in programmes that achieve scores below three typically have to transfer to other programmes to complete their studies.

The national research funding council, the Conselho Nacional de Desenvolvimento Científico e Tecnológico (CNPq), also takes into account the association of researchers with particular evaluated programmes in in the assessment of application for individual research grants, including the “Research Productivity Grants” (Bolsas de Produtividade em Pesquisa).

Efficiency and cost-effectiveness

The costs of the current CAPES evaluation system for CAPES itself are comparatively modest owing to the reliance on the voluntary participation of academics in the field committees. However, as the system evolves, it will inevitably have to find ways to maintain the value of peer involvement in the evaluation, while reducing the sheer volume of work required to evaluate each programme.

The use of Qualis appears to be an efficient and relatively effective way of providing information on the broad quality of a proportion of the scientific output of programmes. In contrast, the time and effort dedicated to the review of books and book chapters by some of the field committees seems disproportionate to the information about the quality of programmes that is obtained from the exercise.

In parallel, however, the CAPES evaluation system is notable for the absence of site visits to programmes or interviews with programme coordinators and its almost exclusive reliance on domestic peers for its review work. These features of the system keep costs down. Any future decision to increase the use of visits and interviews, or the involvement of international peers are likely to add to the costs of the system.

6.3. Key recommendations concerning CAPES evaluations

1. Adjust the weighting of evaluation criteria in assessment of new courses to focus more on relevance, training and continuous improvement

The approval of new postgraduate courses through the systems of peer review currently in place creates an effective mechanism for assuring the quality of new academic postgraduate education in Brazil. Nevertheless, the OECD review team considers the current evaluation process for new courses could be improved by adopting the following modifications:

Revise the structure of the evaluation fiche for new courses to create a more transparent structure that follows the intervention logic for postgraduate training programmes, moving from inputs (including institutional context, supervisory staff, facilities) to processes (programme structure, approaches to incorporating practical experience, methods for supervision, mentoring and assessment) and expected outputs (graduation times and rates, graduate profiles) that provides a clearly formulated and valid rationale for each indicator used.

Include a separate section in the evaluation fiche on the relevance of the programme to national development needs, taking into consideration the development of new scientific areas and the knowledge and skills required for the further development of the country, including in natural sciences, social sciences and the arts.

Increase the weight attached in the evaluation of new courses to the training dimension of programmes and support provided to students, with an assessment of the likely capacity of the programme to equip students with relevant research and transversal skills (such as collaborative working, communication, project management or entrepreneurship).

Include a more explicit requirement for a programme development plan for all new programmes approved, setting out specific and measurable goals over time. This would act as a reference for subsequent periodic reviews and introduce a clearer focus on continuous improvement. The approach used by CONACyT in Mexico for assessment of programmes for the Programa Nacional de Posgrados de Calidad (PNPC) might provide some inspiration in this regard (CONACyT, 2015[6]).

2. Bring additional perspectives into the evaluation of new programmes

As argued in the preceding analysis, the current field committees undertaking the assessment of new programme proposals are composed exclusively of academic peers from the field in question. To bring a broader range of perspectives to the process and potentially promote innovation and inter-disciplinary cooperation, CAPES should involve one or more academics from other academic fields in the field committees undertaking the assessment of new courses.

In addition, to bring in expertise and perspectives from outside the academic community, CAPES should consider appointing specialists in economic development and the evolution of skills and knowledge requirements, as well as representatives of the private economy and the wider public sector to the Scientific and Technical Council (CTC-ES). If implemented effectively, this could ensure that final decisions on programme approval take into account broader national needs and developments.

3. Maintain programme-level accreditation in the medium-term, but consider the long-term desirability of transitioning to institutional self-accreditation for established institutions and programmes

Brazil’s postgraduate education system has grown rapidly in recent years and might still be considered to be in a phase of consolidation, when compared to postgraduate education systems in many other OECD and partner countries. In the medium term, it therefore makes sense to maintain course-level accreditation, to maintain oversight of the continued development of the system and ensure the promotion of quality. In the longer term, it could be possible to move to a system of institutional self-accreditation linked to strengthened model of institutional accreditation (see chapter 7). This would allow universities to start academic postgraduate programmes if they met certain criteria in terms of staff and profile and had been judged to have strong institutional quality systems in an institutional quality review. The provision of publicly funded scholarships and additional programme funding should certainly remain dependent on positive external evaluation of the programme, in line with practice in many OECD systems.

4. Clarify the objectives of periodic evaluations and rebalance the focus of evaluation criteria to include greater focus on student outputs and outcomes

The periodic (four-year) evaluations of postgraduate programmes currently devote disproportionate attention and resources to assessing the outputs of academic staff. Although the quality of staff is an important factor in the quality of postgraduate programmes, the CAPES evaluations should focus on assessing the conditions for, and performance of, postgraduate training, not the research output of academic departments. The OECD review team therefore recommends:

Rebalancing the weighting in the evaluation criteria for four-year assessments, by increasing the weight attributed to educational processes, student outputs and employment outcomes, and reducing the weight attributed to staff outputs.

Reducing the time and resources allocated to assessment of staff output and assessing only a limited sample of research output. The Qualis for journal rankings could be maintained, but should also be reviewed, to introduce more uniformity in the classification of journals between knowledge fields. Less time should be devoted to assessment of individual outputs (particularly books and book chapters). This would contribute to reducing the workload for field committees and making the entire peer review system more manageable in the medium-term.

Systematically including interviews with course and programme coordinators as part of the periodic assessment of courses and programmes, to gain addition insights into the operation and performance of the programme and answer questions arising from documentary evidence.

If a more detailed research assessment exercise is considered necessary to promote quality in the research function of higher education in Brazil, the relevant authorities should establish this as a separate, but related exercise, with clear and distinct objectives. All activities undertaken as part of the CAPES evaluation processes should focus on ensuring quality and promoting quality enhancement in the postgraduate training system.

5. Ensure those judging whether programmes are of international standing really have an international perspective.

Given Brazil’s aspiration to develop a world-class postgraduate training system, it would be valuable to gain an international perspective on the programmes judged nationally to be among the best in the country. The OECD review team therefore recommends that CAPES systematically involve non-Brazilian academics in the assessment of programmes pre-selected by field committees as candidates for being programmes of international quality or excellence. In light of the number of programmes involved, it is likely to be most feasible to concentrate this international involvement on programmes proposed for the top score of seven. It may be possible to organise international peer review committees who are able to review synthesised information about the programmes under review in English or Spanish and potentially conduct group interviews remotely or in person with programme coordinators.

6. Undertake evaluations of specific components of the CAPES system and aspects of academic postgraduate provision as inputs to future policy

The OECD review team identified two specific issues where further information and analysis appears to be required in order to plan future policy for academic postgraduate education in Brazil and its external quality assurance:

First, the full costs associated with the current system of external peer review are a “black box”. Peer review is inherently time-consuming and therefore expensive. The time academic staff spend involved in peer review is time they are not dedicating to their core activities of teaching, research and engagement with society. In order to help plan the future development of the system of peer review, CAPES should undertake an assessment of the cost of the time used by members of the field committees in the evaluation process, including the unit cost per programme evaluation.

Second, there is a wider question relating to the future of academic (stricto sensu) Master’s programmes. As noted, Master’s programmes in most OECD countries are now viewed as either purely professional qualifications (as in the United States), or an extension and deepening of undergraduate studies, which prepares students for work in knowledge-intensive sectors (as in most of Europe). A doctorate is regarded as a pre-requisite for an academic or research career in most of the world, including, increasingly, in Brazil. This leaves the question as to what academic Master’s programmes are for. Is the intention that Master’s graduates should go on to undertake a PhD and work in academia, or should they be prepared for work in the wider economy? If the latter is the case, it is questionable whether Master’s programmes should continue to be part of the highly academic and research-focused CAPES evaluation processes (notwithstanding the recommendations about rebalancing above).

It would be valuable to undertake a systematic evaluation of the role of Master’s education in Brazil, including a specific focus on the profile and effectiveness of the Professional Master’s programmes created in recent years. This evaluation should consider, in particular, the destinations of previous graduates from these programmes and the views of the academic community and private and public sector employers on the relevance and future role for Master’s level education in Brazil.

References

[3] CAPES (2018), Sobre a Avaliação - Objetivos da Avaliação, CAPES website, http://www.capes.gov.br/avaliacao/sobre-a-avaliacao (accessed on 17 November 2018).

[2] CAPES (2017), CAPES divulga resultado final da Avaliação Quadrienal 2017, CAPES Website, http://www.capes.gov.br/sala-de-imprensa/noticias/8691-capes-divulga-resultado-final-da-avaliacao-quadrienal-2017 (accessed on 17 November 2018).

[4] CAPES (2017), Portaria Nº 161, de 22 de agosto de 2017 - Avaliação de Propostas de Cursos Novos, APCN, de pós-graduação stricto sensu. (Ordinance 161 of 22 August 2017 - Evaluation of Proposals for New Programmes (APCN), postgraduate stricto sensu), https://capes.gov.br/images/stories/download/legislacao/30082017-Portaria-N-161-de-22-de-agosto-de-2017.pdf (accessed on 17 November 2018).

[7] CAPES (2017), Regulamento para a Avaliação Quadrienal 2017 (2013-2016) Programas acadêmicos e profissionais., Coordenação de Aperfeiçoamento de Pessoal de Nível Superior, Brasília, http://capes.gov.br/avaliacao/sobre-a-avaliacao/legislacao-especifica (accessed on 17 November 2018).

[6] CONACyT (2015), Programa Nacional de Posgrados de Calidad (PNPC) Consejo Nacional de Ciencia y Tecnología - Marco De Referencia, Consejo Nacional de Ciencia y Tecnología, Mexico City.

[1] European University Association (2018), University Autonomy in Europe, https://www.university-autonomy.eu/ (accessed on 15 October 2018).

[5] INEP (2017), Instrumento de Avaliação de cursos de graduação Presencial e a distância - Reconhecimento e Renovação de Reconhecimento (Evaluation instrument for undergraduate programmes - classroom-based and distance - recognition and renewal of recognition), Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira, Brasília, http://www.publicacoes.inep.gov.br (accessed on 11 November 2018).

Notes

 1.  The capacity of higher education institutions to provide lato sensu postgraduate programmes is verified through the institutional accreditation and re-accreditation procedures implemented by INEP as part of SINAES. The operation of lato sensu “specialisation” programmes more broadly is governed by a 2007 Resolution of the National Education Council. (CNE, 2007[94])

 2.  Of 32 European higher education systems examined by the European University Association’s ‘Autonomy Scorecard’, only ten require universities to seek prior accreditation to start a doctoral programme.

 3.  Including Federal, State and municipal public institutions and private institutions

 4.  The OECD understands that the maximum score initially attributed to a new course is three. Programmes are ultimately rated on a scale of 1-7, where scores 6 and 7 are reserved for programmes with doctoral provision and that are assessed to be operating at an internationally comparable level of excellence. Scores 4 and 7 can only be attributed following a full periodic review, once the programme is well established.

 5.  Several funding programmes are run by CAPES for postgraduate programmes with a CAPES evaluation score of at least three. For public institutions, the Programa de Demanda Social (DS) provides funding for student grants and the Programa de Apoio à Pós-Graduação (PROAP) provides funding for the programme itself (facilities, project etc.). For private institutions, the Programa de Suporte à Pós-Graduação de Instituições de Ensino Particulares (PROSUP) provides funding for student grants. Programmes in the public and private sectors that achieve a score of 6-7 (which necessarily have a doctoral programme) can obtain further grant funding from the Programa de Excelência Acadêmica (PROEX).

 6.  The relevant guidelines for approval of new programmes for different areas always suggest that a site visit may be conducted, but not that visits are systematically a part of initial course approval. (CAPES, 2018[98])

 7.  The Mexican Programa Nacional de Posgrados de Calidad (PNPC), for example, includes a specific criterion on ‘follow-up and academic development of students’ (CONACyT, 2015[99])

 8.  Until 2013, the periodic evaluations were conducted every three years.

 9.  The h-index is an author-level metric that attempts to measure both the productivity and citation impact of the publications of a scientist or scholar. The index is based on the set of the scientist's most cited papers and the number of citations that they have received in other publications. The index can also be applied to the productivity and impact of a scholarly journal as well as a group of scientists, such as a department or university or country.

7.

Assuring the quality of higher education institutions

Like the external quality assurance systems in many other OECD and partner countries, the National System for Evaluation of Higher Education (SINAES) in Brazil evaluates both higher education institutions and individual study programmes within those institutions. Private and public institutions are subject to periodic re-accreditation, based on on-site reviews coordinated by INEP. Whereas for private institutions, re-accreditation is a pre-requisite for their continued operation, legally protected public institutions the process is essentially a formality. In both cases, the period for which re-accreditation is granted varies depending on the status and institutional quality score awarded to the institution. Institutions are also subject to annual monitoring, based on the average performance of their programmes in relation to SINAES programme-level indicators and the results of CAPES evaluations for “stricto sensu” postgraduate programmes. This chapter examines these processes and provides recommendations for the future development of the systems in place.

7.1. Focus of this chapter

The external quality assurance system in Brazil evaluates institutions as well as programmes

Like the external quality assurance systems in many other OECD and partner countries, the National System for Evaluation of Higher Education (SINAES) in Brazil evaluates both higher education institutions and individual study programmes within those institutions. While the ongoing programme-level evaluation mechanisms discussed in Chapter 5 (ENADE, programme-level indicators, on-site reviews for renewal of recognition) attract considerable public attention and absorb a large share of the resources devoted to external quality assurance, SINAES also involves periodic institutional evaluations, which inform regulatory decisions by SERES on whether or not to re-accredit institutions.

Legally, both private and federal public institutions are subject to periodic re-accreditation (recredenciamento), based on on-site reviews coordinated by INEP. For private institutions, successful re-accreditation is a prerequisite for their continued operation (although “de-accreditation” is rare). For federal public institutions, the process is essentially no more than a formality, as they systematically score three or more on a five-point evaluation scale and, in any case, cannot have their accreditation removed.

The period for which (re-)accreditation is valid varies depending on the status and institutional quality score (CI) already awarded to the institution. Universities and university centres are only re-accredited every eight to ten years, while colleges must be re-accredited at least every five years.

In addition, institutions are subject to annual monitoring, based on their average performance of their programmes in relation to SINAES programme-level indicators and the results of CAPES evaluations for stricto sensu postgraduate programmes. The weighted averages of the Preliminary Course Score (CPC), discussed in Section 5.2 and, where applicable, the scores attributed by CAPES for new and existing postgraduate programmes, discussed in Chapter 6 are used to produce an overall score for each institution called the “General Course Index” (Índice Geral de Cursos, IGC).

Institutional evaluation, including self-evaluation, has a central place in the original legislation governing the system

The wording of the legislation establishing the SINAES in 2004 recognises the central role of institutions in structuring and providing higher education, alongside their research and engagement functions, and acknowledges the importance of institutional autonomy and profile in allowing HEIs to fulfil their missions. The first article of the law establishing SINAES states:

The aim of SINAES is to improve the quality of higher education, to expand its provision, to increase institutional efficiency and effectiveness [eficácia] and academic and social impact [efetividade], and especially to promote deepening of the social commitments and responsibilities of higher education institutions, through developing their public mission, the promotion of democratic values, respect for difference and diversity, [and] the affirmation of autonomy and institutional identity. (Article 1 of Law 10 861 of 2004 establishing SINAES (Presidência da República, 2004[1]) bold added by the OECD Secretariat)

The legislation states that evaluation of the federal higher education system will involve institutional evaluation, programme-level evaluation and assessment of the performance of students through ENADE. It places institutional evaluation first and develops objectives and evaluation criteria for institutional review in more detail than for programme-level evaluation and ENADE. It specifies ten main dimensions to be taken into account in internal and external institutional evaluation processes, including the Institutional Development Plan (PDI); institutional policies; social responsibility; management; infrastructure; student support and financial sustainability. The seventh dimension listed is “planning and evaluation, especially the processes, results and effectiveness of institutional self-assessment”.

To undertake this institutional self-evaluation, the 2004 law specifies that all HEIs must create an Internal Evaluation Commission (Comissão Própria de Avaliação, CPA). This body is tasked both with coordinating all internal evaluation processes inside the institutions and transmitting institutional and programme-level information to INEP, as input to external evaluation activities. The CPA must include representatives from all sections of the academic community (different categories of staff) and social partners (“organised civil society”), and have formal independence from other management and collegiate bodies in the institution.

Internationally, HEIs have varying levels of autonomy to take responsibility for self-evaluation and quality assurance

The distinction between a) internal evaluation processes within HEIs; b) external programme evaluation and; c) external institutional evaluation, as seen in Brazil, is found in many higher education systems in OECD and partner countries. However, the extent to which systems rely on each of these three components varies considerably.

The systems of quality assurance in Ireland, England and Scotland, for example, dispense almost entirely with external programme evaluation and rely instead on internal quality assurance systems within institutions (self-evaluation), which are verified through external institutional reviews (QAA, 2018[2]; QQI, 2018[3]). Most quality accreditation activities in the diverse quality assurance landscape in the United States also involve institutional reviews, which verify internal quality processes (Hegji, 2017[4]). Quality assurance systems in many other European higher education systems, including the Netherlands, Sweden or Portugal, all include both programme and institutional review in their external quality assurance systems, alongside internal quality assurance. In all three of the latter countries there have been initiatives to move to systems based primarily or exclusively on institutional review (NVAO, 2016[5]; UKÄ, 2018[6]; A3ES, 2018[7]). In contrast, in Mexico, where there is no comprehensive and compulsory system of external quality assurance in higher education, existing external quality assurance mechanisms focus primarily on programme-level accreditation (CIEES, 2018[8]).

Overall, while external quality assurance systems in many countries may initially have included a strong focus on programme-level review, there has been a general trend among policy-makers and international bodies working in quality assurance to recommend increased institutional responsibility for quality and to focus external evaluation efforts primarily on the institutional level. This is the philosophy reflected, for example, in the current European Standards and Guidelines for quality assurance in higher education (ESG, 2015[9]), which serve as a reference for quality assurance in the 48 countries of the European Higher Education Area.

Despite its legal basis, Brazil’s current quality assurance model gives comparatively limited responsibility to institutions for assuring their own quality.

While the letter of the law governing quality assurance in higher education in Brazil accords a central role to institutional autonomy and self-evaluation, the practical implementation of the SINAES imposes a complex system external programme-level scrutiny on a three-year cycle. For institutions that perform poorly in ENADE and on the CPC, this leads to regular programme-level inspections, using prescriptive processes that limit the room for manoeuvre for institutions. For institutions that tend to perform well in relation to ENADE and the CPC, particularly universities and university centres that are only subject to institutional review every eight to ten years, on-site evaluations by external reviewers are comparatively infrequent occurrences with limited consequences.

There are few incentives for institutions in this position to develop strong internal quality assurance systems that go beyond the minimum requirements imposed by the legislation, or to promote quality enhancement internally on a continual basis. Interviews conducted by the OECD review team in several institutions suggest that Internal Evaluation Commissions (CPAs) focus primarily on ensuring compliance with SINAES rules and delivering data to INEP, rather than developing internal quality systems tailored to institutional needs or promoting innovation and quality improvements. This contrasts with the situation in many European countries and in the United States, where institutional review and evaluation of internal quality procedures form the core of external quality assurance practices.

The remainder of this chapter explores these issues, reviewing the processes currently in place in Brazil to assess the quality of individual higher education institutions, including ongoing monitoring through the General Course Index (IGC) and periodic reaccreditation reviews.

7.2. Strengths and weaknesses of the current system

Indicator-based monitoring of institutions: the General Course Index

Relevance: rationale and objectives of the current system

As discussed in Chapter 4, new private higher education institutions, and new campuses of existing private providers in municipalities outside the location of their headquarters, are required to obtain accreditation from SERES before they can start operating. Institutions undergo an on-site review by an external review commission appointed by INEP, which attributes the institutions and Institutional Score (Conceito Institucional) or CI, on a five-point scale. Institutions that receive a score of three or above receive formal accreditation, and, in the logic of SINAES, this institutional score is the official quality “grade” attributed to the institution and published on the e-MEC platform.

New public institutions are exempt from this initial accreditation process, as they are effectively accredited as part of their acts of establishment. Public, like private, institutions are formally required to undergo renewal of their accreditation through a process involving another on-site review periodically that leads to a new CI and which we examine below. However, whereas the re-accreditation process could theoretically lead to the “de-accreditation” of private HEIs, public institutions are protected by their legal status. This means re-accreditation is purely an administrative formality for public institutions. Even for private institutions in Brazil, the risk of “de-accreditation” appears to be low. Although clear data on the number of cases of institutional “de-accreditation” have not been made public by SERES, the OECD review team understands only a handful of private institutions have last accreditation in the last decade.

All institutions with operational courses are subject to the cycle of programme-level evaluation through ENADE, on the basis of which INEP calculates the Preliminary Course Score (CPC), discussed in Chapter 5, by combining ENADE results with other programme indicators. Once three cohorts of students have graduated (over three years) and, depending on the subjects in their programme profile, potentially been subject to three cycles of ENADE, INEP calculates another composite indicator: the General Course Index. This “Index”, also on a scale of one to five, is calculated based upon:

The average CPCs of the last three-year period (in which all subjects have been subject to a round of ENADE), for the programmes that have been evaluated in the institution, weighted by the number of enrolments in each of the programmes included in the calculation;

The average of the evaluation score of the stricto sensu postgraduate programmes awarded by CAPES in the last available evaluation round, converted to a compatible scale and weighted by the number of enrolments in each of the corresponding postgraduate programmes;

The averaged (enrolment-weighted) sum of scores from undergraduate and stricto sensu postgraduate programmes (INEP, 2017[10]).

The General Programmes Index (IGC) provides a single, synthetic, and comparative indicator of institutional performance. At the time of its creation, the IGC was conceived a means to allow the Ministry of Education “to identify the most precarious institutions and focus its attention on them” (Schwartzman, 2013[11]). In this sense, it mirrors the function of the CPCs in renewal of programme recognition.

Effectiveness: division of responsibilities

The methodology used to calculate the IGC, as well as the calculation and presentation of results (in e-MEC), are the responsibility of INEP. Institutions bear no responsibility in this process, apart from participation in the administration of ENADE and in the CAPES assessments of postgraduate programmes, and in reporting administrative data to INEP – via the CPA - used in the calculation of the IGC.

Effectiveness: use and effects of the IGC

The IGC score is used by external bodies and the media in reporting about the quality of higher education in Brazil, including in a well-known institutional ranking published annually by the Folha de São Paulo, one of Brazil’s leading newspapers (Folha de S.Paulo, 2018[12]). Notwithstanding its original purpose, the IGC is widely perceived as a visible public signal of institutional quality that that institutions themselves feature in advertising. It likely also bears upon the equity performance of publicly listed for-profit firms. One private university states on its website, for example:

The Universidade Positivo (UP) has been rated, for the sixth time running, the best private university in Paraná State, with a score of 4 in the Índice Geral de Cursos (IGC), which goes from 1 to 5. Revealed last Monday (27) by the Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira (INEP) [and] the Ministry of Education (MEC), the IGC is the official indicator of the quality of higher education institutions in Brazil. (UP, 2018[13])

The real signal value of the IGC as a quality indicator for consumers is, perhaps, quite limited. While IGC scores range, in principle, from one to five (after rounding), scores of one are virtually unknown, and nearly all scores cluster at values of three and four. In 2016, 93% of universities and 96% of university centres received scores of three or four (INEP, 2017[14]). Setting aside the validity or reliability of the IGC, it is clear that its discriminating power for non-college institutions is low.

Although the reputational effects of IGC can be important, it is not an indicator that is likely to have an impact of how institutions understand and manage the quality of the education that they provide. The IGC does not introduce new performance information for institutional leaders. Rather, it is a (weighted) summation of already-existing programme-level information. As such, it replicates the measurement problems of the CPC.

Although the IGC may have been created, like the CPC, as a means for public authorities to identify weak institutions, it is not mentioned in the most recent secondary legislation providing implementation rules for SINAES (Presidência da República, 2017[15]; MEC, 2017[16]). Unlike the CPC, it does not currently play a direct role in the process of regulation of higher education institutions. The relevant legislation states that all institutions must undergo an on-site visit for institutional re-accreditation.

Institutional reviews: re-accreditation

Relevance: rationale and objectives

Entirely separate from the assessment-based IGC – and potentially at odds with it – are the on-site institutional reviews that are periodically undertaken for the re-accreditation of higher education institutions. The same process is also used for changes in the institutional status - transformação de organização acadêmica - of colleges to university centres, or university centres to universities. The process of institutional re-accreditation generates a new institutional score (Conceito Institucional) or CI, that replaces the CI attributed during initial accreditation as the most recent quality score for the institution. The CI score is used by SERES to determine whether the institution is permitted to continue to award recognised degrees, and when the next cycle of reaccreditation must take place. For university institutions the duration of reaccreditation may range between five and ten years, depending upon their CI score. For colleges and university centres, the range is from three to five years, as shown in Table 7.1.

Table 7.1. Duration of accreditation and re-accreditation
Type of institution | Institutional score required | Duration of accreditation
Colleges and University Centres | CI 3 | 3 years
CI 4 | 4 years
CI 5 | 5 years
Universities | CI 3 | 5 years
CI 4 | 8 years
CI 5 | 10 years
Source: MEC (2017) Regulatory Ordinance No 1, 3 January 2017 establishing duration of the validity of regulatory acts for accreditation and re-accreditation of HEIs (MEC, 2017[17])

Effectiveness: division of responsibilities

Re-accreditation, like accreditation, is the joint responsibility of higher education institutions, INEP, and SERES. However, as higher education institutions are fully operational, the institution’s Internal Evaluation Commission (CPA) plays an active role in preparing an institutional evaluation report, based on consultations with academic staff and management. INEP is responsible for organising the on-site review process, and SERES with taking regulatory action of the basis of evaluation information provided by the review.

Effectiveness: quality indicators used and generated

The Institutional Score (CI) that results from the review process is generated based upon the evaluation commission’s scores for up to 50 indicators set out in the relevant INEP evaluation instrument (INEP, 2017[18]). The focus of these indicators, structured into the same five axes as for accreditation, is principally the institutional development plan (30%), the institution’s infrastructure (30%), and its management policies (20%).

Table 7.2. Number and weight of indicators for institutional reaccreditation and change of institutional status
Axis | Number of indicators | Weight
Planning and institutional evaluation | 5 | 10
Institutional development | 7 | 30
Academic policies | 12 | 10
Management policies | 8 | 20
Infrastructure | 18 | 30
TOTAL | 50 | 100 
Source: OECD calculations based upon INEP evaluation instruments. (INEP, 2017[18])

Like the other on-site review processes (such as recognition), the re-accreditation review process is focused on input and process, rather than outputs or performance, and reviewers are responsible for scoring qualitative indicators on a five-point scale. The review team’s categorisation of the indicators in the evaluation instrument is shown in Table 7.3. Given that the process of re-accreditation necessarily focuses on institutions that are already operating, with graduating students and graduates, there is scope to include greater consideration of outputs (graduates and evidence of the learning outcomes) and outcomes (graduate destinations) in the institutional assessments at this stage.

Table 7.3. Type of indicators used in the re-accreditation process
Total number of indicators | 50
Total input | 35
Total process | 15
Total output | 0
Source: OECD calculations based upon INEP evaluation instrument (INEP, 2017[18])

The current evaluation instrument for institutional re-accreditation devotes comparatively little attention (in terms the number of indicators and judgement criteria) or weight to assessment of the internal evaluation capacity of institutions. The first “axis” of the assessment framework focuses on institutional planning and evaluation activities does contain five indicators relating to internal evaluation process and the quality of the internal evaluation report produced by the CPA. Many of the factors identified in the individual judgement criteria for these indicators would appear highly relevant for the assessment of internal quality management practices, although these factors are not developed and explained in detail. However, as these issues are embedded in few individual indicators in an axes that contributes only 10% of the overall institutional score, the quality of internal evaluation capacity does not currently play a major role in whether an institution is re-accredited or what institutional score they receive.

Effectiveness: use and effects

Owing to the schedules for re-accreditation highlighted above, the CI score is not calculated and reported on an annual basis, but rather with a periodicity that may range from three to ten years. In light of its infrequency, and perhaps because it is not linked to student outcomes as observed in ENADE, the CI score appears to function solely as a regulatory input, and not as a public signal of institutional quality. In the course of stakeholder meetings with the OECD review team, the CI score was not identified as a measure to which institutions managed or adapted their performance. The ongoing monitoring of the Institutional Development Plan through internal evaluation processes (self-evaluation) was cited as an important feature of the re-accreditation process.

However, there was variation in the extent to which it was considered to have generated useful reflection within institutions or contributed to the development of a genuine quality culture. Some CPAs reported that their HEI found the self-evaluation activities to be a compliance activity in which few colleagues wished to participate. Other institutions found the obligation to create development plans and undertake a structured analysis of institutional performance spurred useful quality discussions that would have otherwise not have occurred.

7.3. Key recommendations concerning institutional review

In the view of the OECD Review team, the processes of institutional quality assurance that result from periodic re-accreditation and regular reporting of IGCs need fundamental improvements. Based on the analysis above, the team makes the following recommendations:

1. Reduce the period of re-accreditation for universities and university centres

Universities in some of the best-regarded higher education systems in the world must undergo external institutional reviews every four, five or six years. This is the case in the United Kingdom, the Netherlands and Sweden, for example. The current eight or ten-year accreditation periods for universities and university centres mean these institutions have few incentives to develop robust institutional quality mechanisms and problems in institutional quality management may go undetected for long periods. Instead of the current system, institutions with demonstrated internal quality capacity could be rewarded through dispensation from some or all aspects of programme-level review, subject to successful reaccreditation on a five or six-year cycle (see below).

2. Reduce the weight attached in institutional re-accreditation reviews to input and process indicators that measure basic supply conditions for higher education

There is scope to rebalance the weights attributed to the evaluation indicators used at the stage of institutional re-accreditation, away from inputs and towards processes and outputs. A first aspect of this is to remove indicators relating that measure basic supply conditions for higher education, such as infrastructure and equipment and general management policies. The availability of suitable infrastructure to supply each undergraduate programme is verified through the programme-level recognition and renewal of recognition processes, while the most general institutional policies are unlikely to change – or need to change - considerably over time. It is therefore wasteful to devote resources to re-evaluating and re-scoring these kinds of variable through the re-accreditation review. The inclusion of these indicators also reduces the proportional weight attributed to factors that are important to verify in re-accreditation, such as educational results and institutional performance.

3. Increase the weight attributed to outputs and outcomes in periodic institutional assessment

Evidence about educational results and institutional performance is neglected in the current system of institutional re-accreditation. While processes of accreditation cannot take into account programmatic and institutional performance, re-accreditation can – but does not. Institutions should be able to graduate most students who begin their studies and they should do so in a timely way. Those who graduate should be able to find employment, preferably in fields related to their area of study – and most certainly so if their studies have a career orientation – whether accounting, civil engineering, or nursing.

SINAES was based in a view that quality assurance could proceed through coordinated and complementary processes – including institutional self-assessment, detailed on-site institutional reviews carried out by peer reviewers, and through the implementation of learning assessments (ENADE) and use of related indicators. The processes of evaluation that have evolved are not complementary to one another. On-site reviews and performance indicators generate information about institutional quality that is either incommensurable or contradictory.

Quantitative programme and institutional indicators should ideally focus on the outputs and outcomes of higher education, while on-site reviews conducted by peers would helpfully focus on the inputs and processes that generate the outputs and outcomes observed in indicators. For example, indicators focused on outputs or outcomes, such as graduation rates, would be complemented by an on-site review process that examines the conditions that affect variation in these rates. Thee conditions include student advice and mentoring processes; how institutions identify students at risk of falling behind or dropping out; the social or psychological; and academic support services provided to students at risk.

4. Increase incentives for institutions to take a strategic view of quality

The processes of institutional quality assurance do not encourage institutions to take a truly strategic and institution-wide view of quality. The IGC generates a score that is an aggregation of programme-level results. However, it does not generate a score that has been demonstrated to be useful in differentiating different levels of institutional performance or providing actionable feedback to institutions. Institutional Development Plans (PDIs), in their current form do not appear to provide an opportunity for institutions to take a comprehensive and strategic view of their institution, its profile, and quality of its educational programmes. Universities are, notionally, institutions that provide research-led teaching, and should be able to give an account of where and how undergraduate education is joined up to their research mission. Institutional Development Plans examined by the OECD review team, however, do not show evidence of this. It would be valuable to provide incentives to institutions to develop more meaningful PDIs with a stronger focus on how quality across a range of dimensions can be maintained and enhanced. One way to do this is to make assessment of internal quality policies and practice a much bigger part of the re-accreditation process, through greater weighting in the relevant evaluation instrument for on-site reviews.

5. Move to a system where institutions that can demonstrate strong internal quality assurance capacity and a proven record of delivering quality can accredit (authorise and recognise) their own programmes

Finally, processes for demonstrating institutional quality do not permit higher education institution to demonstrate that they have the capacity to take care of quality, and should be authorised to act as self-accrediting organisations, and should be permitted to create, revise, and eliminate programmes on their own initiative – as happens in other higher education systems in the world. The process of re-accreditation – specifically, the resulting CI score – changes the periodicity of institutional reviews, but it does not alter the level of responsibility that institutions are permitted to exercise. If account for institutional quality is to be joined up to institutional responsibility for the quality of programmes, it will need to be a very different and more robust process than at present. Examples of such differentiated models – where some institutions are subject to programme-level review and others are accorded self-accrediting status on the basis of rigorous institutional review - exist in other systems of higher education and could serve as inspiration for Brazil.

References

[7] A3ES (2018), Acreditação e Auditoria - A3ES, gência de Avaliação e Acreditação do Ensino Superior (A3ES), http://www.a3es.pt/en/node/83 (accessed on 18 November 2018).

[8] CIEES (2018), Evaluación de programas, Comités Interinstitucionales para la Educación Superior, https://ciees.edu.mx/ (accessed on 18 November 2018).

[9] ESG (2015), Standards and Guidelines for Quality Assurance in the European Higher Education Area (ESG).

[12] Folha de S.Paulo (2018), Ranking Universitário Folha - 2017, Folha de S.Paulo, http://ruf.folha.uol.com.br/2017/ranking-de-universidades/ranking-por-ensino/ (accessed on 17 November 2018).

[4] Hegji, A. (2017), An Overview of Accreditation of Higher Education in the United States, Congressional Research Service, Washington, D.C., http://www.crs.gov (accessed on 17 November 2018).

[14] INEP (2017), Inep divulga Conceito Preliminar de Curso e Índice Geral de Curso de 2016 (INEP reveals the Preliminary Course Score and General Course Index for 2016), INEP 24 November 2017, http://portal.inep.gov.br/artigo/-/asset_publisher/B4AQV9zFY7Bv/content/inep-divulga-conceito-preliminar-de-curso-e-indice-geral-de-curso-de-2016/21206 (accessed on 17 November 2018).

[18] INEP (2017), Instrumento de Avaliação Institucional Externa - Presencial e a distância - Recredenciamento Transformação de Organização Acadêmica (External institutional evaluation instrument - Classroom-based and distance - Re-accreditation and change of academic status), Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira, Brasília, http://download.inep.gov.br/educacao_superior/avaliacao_institucional/instrumentos/2017/IES_recredenciamento.pdf (accessed on 11 November 2018).

[10] INEP (2017), Nota Técnica No. 39/2017/CGCQES/DAES (Methodology for calculating the Índice Geral de Cursos - IGC), Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira, Brasília, http://download.inep.gov.br/educacao_superior/enade/notas_tecnicas/2016/nota_tecnica_n39_2017_cgcqes_daes_calculo_igc.pdf (accessed on 18 November 2018).

[16] MEC (2017), Portaria Nº 23 de 21 de dezembro de 2017 - sobre o exercício das funções de regulação, supervisão e avaliação de instituições de educação superior e de cursos superiores (Ordinance 23 of 21 December 2017 regarding the exercise of the functions of regulation, supervision and evaluation of HEIs and programmes), Diário Oficial da União - Imprensa Nacional, http://www.imprensanacional.gov.br/materia/-/asset_publisher/Kujrw0TZC2Mb/content/id/1284670/do1-2017-12-22-portaria-n-23-de-21-de-dezembro-de-2017-1284666-1284666 (accessed on 18 November 2018).

[17] MEC (2017), Portaria Normativa Nº 1, de 3 de janeiro de 2017 Estabelece os prazos de validade para atos regulatórios de credenciamento e recredenciamento das Instituições de Educação Superior. (Ordinance No. of 3 January 2017, establishing duration of validity of regulatory acts for accreditation and re-accreditation of HEIs), http://www.in.gov.br/autenticidade.html, (accessed on 14 November 2018).

[5] NVAO (2016), Assessment framework for the higher education accreditation system of the Netherlands, Nederlands-Vlaamse Accreditatieorganisatie (NVAO), The Hague, https://www.nvao.com/system/files/procedures/Assessment%20Framework%20for%20the%20Higher%20Education%20Accreditation%20System%20of%20the%20Netherlands%202016_0.pdf (accessed on 17 November 2018).

[15] Presidência da República (2017), Decreto Nº 9.235, de 15 de dezembro de 2017 - Dispõe sobre o exercício das funções de regulação, supervisão e avaliação das instituições de educação superior e dos cursos superiores de graduação e de pós-graduação no sistema federal de ensino. (Decree 9235 of 15 December 2017 - concerning exercise of the functions of regulation, supervision and evaluation of higher education institutions and undergraduate and postgraduate courses in the federal education system), http://www.planalto.gov.br/ccivil_03/_Ato2015-2018/2017/Decreto/D9235.htm (accessed on 10 November 2018).

[1] Presidência da República (2004), Lei no 10 861 de 14 de Abril 2004, Institui o Sistema Nacional de Avaliação da Educação Superior – SINAES e dá outras providências (Law 10 861 establishing the National System for the Evaluation of Higher Education - SINAES and other measures), http://www.planalto.gov.br/ccivil_03/_ato2004-2006/2004/lei/l10.861.htm (accessed on 10 November 2018).

[2] QAA (2018), Reviewing Higher Education, The Quality Assurance Agency for Higher Education, https://www.qaa.ac.uk/reviewing-higher-education (accessed on 18 November 2018).

[3] QQI (2018), Institutional Reviews, Quality and Qualifications Ireland, https://www.qqi.ie/Articles/Pages/Institutional-Reviews07.aspx (accessed on 18 November 2018).

[11] Schwartzman, S. (2013), “Uses and abuses of education assessment in Brazil”, PROSPECTS, Vol. 43/3, pp. 269-288, http://dx.doi.org/10.1007/s11125-013-9275-9.

[6] UKÄ (2018), Guidelines for the evaluation of first-and second-cycle programmes, Universitetskanslersämbetet (Swedish Higher Education Authority), Stockholm, http://www.uka.se (accessed on 17 November 2018).

[13] UP (2018), UP é considerada melhor universidade privada do Paraná pelo sexto ano consecutivo (UP considered the best private university in Paraná for the sixth year running), Universidade Positivo, https://www.up.edu.br/institucional/up-e-considerada-melhor-universidade-privada-do-parana-pelo-sexto-ano-consecutivo (accessed on 17 November 2018).

8.

Governance of external quality assurance

Effective systems of external quality assurance for higher education require effective governance. International experience points to the necessity of three key characteristics in the design of institutions for the governance and implementation of quality assurance. First, quality external assurance bodies need to be independent of government and the demands of party politics, and of the higher education sector itself. Second, responsible authorities must have sufficient resources - financial, human, and intellectual - to meet properly their responsibilities. Third, quality assurance bodies should have carefully developed procedures for engaging with the broader society that higher education systems have a responsibility to serve. This chapter briefly examines the governance and implementation of quality assurance in Brazil in light of these considerations and provides recommendations for the further development of system governance.

8.1. Focus of this chapter

Effective systems of external quality assurance for higher education require effective governance. International experience points to the necessity of three key characteristics in the design of institutions for the governance and implementation of quality assurance. First, quality external assurance bodies need to be independent of government and the demands of party politics, and of the higher education sector itself (ESG, 2015[1]; INQAAHE, 2016[2]). Second, responsible authorities must have sufficient resources - financial, human, and intellectual - to meet properly their responsibilities. Third, quality assurance bodies should have carefully developed procedures for engaging with the broader society that higher education systems have a responsibility to serve (CHEA, 2016[3]). Below, we briefly examine the governance and implementation of quality assurance in Brazil in light of these considerations.

8.2. Strengths and weaknesses of the current system

SERES, INEP and CAPES together regulate, evaluate and supervise the federal higher education system

An elaborate system of quality assurance has evolved in Brazil, in which responsibility for quality assurance rests principally with the Ministry of Education (MEC) and, in particular, the Ministry’s regulatory department, the Secretariat for Regulation and Supervision of Higher Education (SERES), and executive agency for evaluation, INEP. In its evaluation of academic postgraduate programmes, CAPES, which is also answerable to MEC, focuses on a specific sub-sector of activity operating within an institutional landscape much of which is ultimately regulated and evaluated by SERES and INEP¹.

INEP is a semi-autonomous federal institute (literally an “autarchy”, Brazilian administrative terminology), linked to MEC and bears responsibility for planning, coordinating, collecting, and analysing evidence upon which the quality assurance of undergraduate education rests. This includes planning and managing the collection of data for the census of higher education institutions; designing and supervising, and analysing the results of the ENADE (and ENEM); planning and managing on-site evaluations; and developing and managing the indicators that inform and support the regulatory work of government, such as the CPC and IGC.

The evidence of institutional and programme quality produced by INEP’s evaluation work provides the evidentiary basis upon which SERES supervises and regulates the federal system of higher education, both federal public and private institutions. SERES recommends the accreditation, re-accreditation and termination of accreditation of higher institutions to the National Council of Education, the deliberations of which are subsequently sanctioned by the Minister of Education.

CONAES formally supervises the evaluation work of INEP

Under the provisions of SINAES, the National Commission for Evaluation of Higher Education (CONAES) is responsible for proposing and assessing the procedures for the evaluation of institutions, courses and students; for approving the list of courses to be assessed by ENADE, and, more generally, for advising on the articulation federal and state education systems. CONAES is composed of representatives of INEP, CAPES, SERES; representatives of student, teacher and higher education administrative personnel associations; and five external members of recognised expertise appointed the Minister of Education.

This system has strengths…

There are important strengths to the governance and implementation of quality assurance in Brazil. For example,

INEP is recognised internationally as a leading public agency for educational assessment. Its wide experience with large-scale assessment and its capacity to manage data collection systems provides the nation’s higher education quality assurance system with a high level of competence.

CONAES has succeeded in attracting experts to its council, and through them has been able to mobilise higher education research from across the nation to inform the further development of SINAES.

The basic legitimacy and integrity of the quality assurance system is widely accepted across the higher education system, by public and private institutions alike, and by representatives of academic staff and the administrators and owners of higher education institutions.

In the course of its implementation, SINAES has used a range of evaluation techniques – including self-assessment, peer review, and external review grounded in student assessment - that has been widely welcomed.

Some higher education institutions in Brazil now closely monitor the experience of their students and their readiness to participate in external assessments. Others are making efforts to use compulsory self-assessment and peer review processes as opportunities for improvement, and to engage broadly their university community in the assurance of quality.

…but the current system of governance faces three main challenges

Nonetheless, there are three fundamental challenges facing the institutions of quality assurance that merit attention and improvement. First, the design of quality assurance institutions creates conflicting responsibilities for the Ministry of Education. MEC establishes, funds, and steers the federal university system, through its Secretariat for Higher Education (SESu). At the same time, it is responsible, through SERES and, indirectly, INEP, for evaluating their performance and for regulatory actions concerning the programmes they offer. These conflicting responsibilities lead the nation’s higher education institutions, especially its private institutions, to view the Ministry as a champion of one sector rather than a neutral arbiter among all. As one representative of a private higher education institution told to the review team, “For the people at MEC, the federal universities are their children, and we are their bastards.”

Second, while CONAES is responsible for providing guidance and feedback on the functioning of SINAES, it is not properly resourced and organised to do so. CONAES does not have its own professional staff or a dedicated budget, and lacks a capacity to undertake the sort of detailed and sustained analytical work that is needed to evaluate how SINAES is working. Instead, it depends upon the input of implementing bodies whose work it is to supervise and guide, most especially INEP. This dependence is exacerbated by the participation of the implementing bodies on the council itself. It lacks sufficiently wide input – from professional bodies, employer associations, and other centres of government - to take into account the broader social responsibilities of higher education.

Finally, in most higher education systems, responsibility for promoting and sharing quality improvement practices lies with bodies outside of government - with associations that represent sub-sectors (such as research, confessional, or polytechnic universities), and with bodies that represent professional groups within higher education institutions, including institutional research, curriculum design, assessment, and quality assurance. Examples of such bodies with such a role in other OECD countries include the German Rectors’ Conference (HRK, 2018[4]), Universities UK (UUK, 2018[5]) and the National Association of Universities and Higher Education Institutions in Mexico (ANUIES, 2018[6]). The review found few examples of the engagement of equivalent bodies in Brazil in research, advocacy, and training in support of quality improvement, and little attention on the part of public authorities to their potentially important role.

In addition to these three structural issues, there is the question of the coverage of the federal quality assurance system. As noted earlier, the systems for external quality assurance of HEIs and undergraduate programmes analysed in this report apply only to private HEIs and federal public HEIs. State and municipal public institutions – which account for almost 10% of enrolment - are not subject to SINAES, but rather to state-level regulatory and quality assurance rules. Although this situation reflects the constitutional distribution of competences in the Brazilian state, which allow considerable autonomy to states and municipalities, it leads to a fragmented system and means there is no single national benchmark of higher education quality. A single quality reference framework would make external quality assurance for higher education more transparent and understandable for students and their families.

8.3. Key recommendations concerning governance

To respond to the challenges outlined above, we believe that four related policy choices merit consideration.

1. Create an independent quality assurance agency

To address the conflicting responsibilities of MEC – or indeed any future ministry responsible for higher education - Brazilian authorities should consider creating an independent quality assurance body that stands outside the Ministry, in line with practice in many OECD and partner countries. This agency would take the lead in implementing the reformed system of quality assurance proposed in this report. Good international models of bodies with strong legal, financial, and administrative independence exist. In systems with a similar legal tradition to Brazil, such agencies include, for example, Portugal’s Agency for the Assessment and Accreditation of Higher Education (A3ES).

The work to design and create any new agency for quality assurance in Brazil will need to address some key questions:

Which existing functions should be transferred to the new agency? In principle, the new agency would combine the evaluation functions coordinated by INEP’s higher education evaluation directorate (DAES) and the regulatory and supervisory roles of SERES. The changes to the overall model of regulation, evaluation and supervision proposed in this report – such as increased focus on institutional review, reduced numbers of programme-level reviews, a reformed ENADE and a new indicator dashboard - will affect requirements for staff in different roles. The advantages and disadvantages of creating specific evaluation units for different sets of disciplines (natural sciences, social sciences etc.) should be considered. Such units, integrated within the agency, could potentially allow evaluation to be better tailored to individual disciplines and work more closely with the discipline-specific CAPES evaluations.

Should some tasks be devolved to decentralised offices in the states? The current system of quality assurance in the federal higher education system is highly centralised, with all evaluation and regulation activities coordinated from Brasilía. Devolving responsibility to regional departments might theoretically allow a more differentiated approach to quality assurance, with better consideration of the large regional differences in Brazil. However, in the view of the OECD team, distinct quality assurance procedures in different parts of the country would risk creating a two- (or multi-)tier system and undermining national recognition of quality standards. It could be possible, however, to establish regional offices to house professional inspectorates to undertake inspection of infrastructure and institutional management, freeing academic peer reviewers to focus on assessment of academic performance, potentially remotely (see above). The costs of the current system of peer review and the potential costs of a permanent inspectorate would need to be assessed in detail.

How should the new agency be funded? The current system of external quality assurance in Brazil is funded by a combination of public resources (paying the salaries of public servants, for example) and fees paid by institutions for evaluation activities. Quality assurance agencies in a number of systems, including the Portuguese example mentioned above, are funded primarily through fees from institutions. To ensure efficient use of public resources, this should be the long-term aim in Brazil. A thorough analysis will be required to determine the costs of a new agency and the level of fees needed to finance its operation.

The OECD team recognises that there is an existing proposal to create a National Institute for the Supervision and Evaluation of Higher Education (INSAES), that was introduced as a draft bill in Congress in 2012 (Congreso Nacional, 2012[4]), but not pursued. This initiative effectively also proposed a merger of the functions of SERES and INEP, but was criticised for its potential cost and limited added value. The OECD team believes that a new agency would be the most effective way to implement a reformed system of external quality assurance. The reforms proposed in this report are vital to improve the effectiveness and efficiency of the system and any future agency must be designed to operate as efficiently as possible and with limited direct public subsidy.

2. Strengthen CONAES

To ensure that the quality assurance agency has an advisory council that brings a wide social vision to its work, CONAES could take on this responsibility, after substantially modification. CONAES would be a council holding fixed and staggered terms to ensure their independence of government, and encompass balanced representation from students, public and private sector employers, instructors from public and private higher education institutions, higher education administrators, leading researchers, and the senior policy official in MEC with responsibility for taking a comprehensive view of higher education.

3. Restructure the government departments that are responsible for higher education

MEC – or any future ministry responsible for higher education - can support the improvement of quality assurance by restructuring its responsibilities for higher education. This could entail creating a post for a principal policy officer who takes a comprehensive and strategic view of the entire Brazilian higher education system – which the Ministry presently lacks. Units organised along sectoral lines, for example, could support the work of a senior official. These might include groups responsible for (a) federal universities; (b) private universities; (c) technical higher education; and (d) coordination with state and municipal higher education institutions. This scheme of organisation would benefit the nation’s quality assurance system by supporting a strategic and comprehensive vision for the higher education system, by clarifying the role of private provision within the system, and by encouraging continued differentiation of institutions and policies.

4. Incentivise the development of expertise in quality assurance in sector organisations

In monitoring and evaluating the nation’s quality assurance system, a reconstituted quality assurance agency and advisory council (i.e. CONAES) should focus on supporting the development of quality enhancing organisations outside of government. For example, it could support collaboration among state and national bodies of institutional evaluation offices (CPAs), so they share experiences of quality management and improvement practices with one another.

5. Explore how a reformed external quality assurance system could also apply to state and municipal institutions

A single system of external quality assurance applying to all higher education institutions in the country would be more transparent for students and the public than that current co-existence of a large federal system and individual systems for state and municipal institutions in each state. The federal and state authorities, working with the higher education sector, should explore how – and under what conditions - a reformed federal system of quality assurance could be applied to state and municipal institutions, while respecting the distribution of competences enshrined in the constitution of the Union.

References

[6] ANUIES (2018), Asociación Nacional de Universidades e Instituciones de Educación Superior, Asociación Nacional de Universidades e Instituciones de Educación Superior, http://www.anuies.mx/# (accessed on 18 November 2018).

[3] CHEA (2016), The CIQG International Quality Principles: Toward a Shared Understanding of Quality, Council for Higher Education Accreditation , Washington, DC, http://www.chea..

[7] Congreso Nacional (2012), Projeto de lei: cria o Instituto Nacional de Supervisão e Avaliação da Educação Superior - INSAES, e dá outras providências., Secretaria de Assuntos Parlamentares, http://www.planalto.gov.br/ccivil_03/Projetos/PL/2012/msg398-31agosto2012.htm (accessed on 30 November 2018).

[1] ESG (2015), Standards and Guidelines for Quality Assurance in the European Higher Education Area (ESG).

[4] HRK (2018), The HRK - German Rectors' Conference, https://www.hrk.de/hrk-at-a-glance/ (accessed on 18 November 2018).

[2] INQAAHE (2016), INQAAHE Guidelines of Good Practices 2016 - revised edition, International Network for Quality Assurance Agencies in Higher Education (INQAAHE), Barcelona, http://www.inqaahe.org/.

[5] UUK (2018), Universities UK, https://www.universitiesuk.ac.uk/ (accessed on 18 November 2018).

Note

 1.  CAPES regulates all academic postgraduate provision in Brazil, including in public State and municipal universities and university centres. SERES and INEP only regulate institution in the Federal higher education system.

Annex A.

Review team

María José Lemaitre is Executive Director of the Centro Interuniversitario de Desarrollo (CINDA), an international organisation dedicated to promoting effective higher education policy, based in Santiago, Chile. A sociologist and education specialist, María José has been responsible for the development and implementation of quality assurance processes in higher education in Chile. She has chaired the Accreditation Agencies International Network (INQAAHE) and the Iberoamerican Network for Quality Assurance (RIACES) and has served in the Advisory Council of the Spanish Quality Assurance agency (ANECA) and in the International Commission for the Council for Higher Education Accreditation of the United States (CHEA). She has provided advice on higher education and quality assurance issues to governments, higher education institutions and bilateral and multilateral organisations in many regions of the world.

Pedro N. Teixeira is Associate Professor at the Department of Economics (University of Porto) and Director of CIPES (Centre of Research on Higher Education Policy). He was previously Vice Rector at the University of Porto, responsible for the area of Academic Affairs and Teaching and Learning. Pedro is special adviser on higher education to the President of Portugal and a member of the evaluation panels for the Institutional Evaluation Programme of the European University Association (EUA) and the Reviews of the European Association for Quality Assurance in Higher Education (ENQA). He holds a PhD in Economics (Exeter University, UK), a masters in the Economics of Higher Education at CHEPS (University of Twente, The Netherlands), and a bachelor’s degree in Economics (University of Porto). His research interests focus on the economics of higher education, notably on markets and privatisation, and the development of human capital as a research programme.

Thomas Weko is a senior analyst at the OECD specialising in higher education. He coordinated the review. He worked on OECD’s Tertiary Education for the Knowledge Society in 2005-2006, and subsequently served in the US Department of Education, first as Associate Commissioner for Postsecondary, Adult, and Career Education at the National Center for Education Statistics, and then as Director of the Policy and Program Studies Service. He previously worked at the Washington State Higher Education Coordinating Board, at the US Government Accountability Office, and as a university professor.

Simon Roy is an analyst at the OECD, with a background in international higher education policy. He joined the OECD in 2017 from the European Commission, where he was a policy officer in DG Education and Culture's higher education policy team. During his time at the Commission, Simon worked on analysis and policy cooperation activities involving EU Member States and partner countries, including in the fields of higher education funding, performance measurement and graduate skills and employability. He coordinated preparation of the Commission's most recent Communication on higher education, focused on EU cooperation in the field and published in May 2017.

Annex B.

Schedule for the review team’s mission to Brazil

Monday, 5 March, Brasília

9:00-10:00 | Ministry of Education (MEC) Executive Secretary (Vice-Minister)

10:15-11:15

MEC - Secretariat for Higher Education (SESU)

Secretary

General Coordinator for Legislation on Higher Education

Director for the Development of Federal Institutions of HE

General Coordinator for Higher Education Policy

11:30 – 12:30

MEC - VET Secretary (SETEC)

Secretary

Director of VET Policy and Regulation

General Coordinator of VET Regulation and Supervision

13:30 – 14:30

MEC - HE Regulation and Supervision Secretary (SERES)

Secretary

Director for Higher Education Supervision

Director for Regulatory Policy

Director for Higher Education Regulation

14:45 – 15:45 | Science and Technology Development National Council (CNPq) President
16:00 - 16:50 | Anísio Teixeira National Institute for Education Research and Innovation (INEP) President
17:00 - 18:00 | INEP Director for Higher Education Assessment
Tuesday, 6 March, Brasília
8:00 – 8:40 | INEP - Enade General Coordination (CGENADE) General Coordinator for ENADE
8:50 - 9:30 | INEP – General Coordination of Higher Education Institutions Assessment (CGACGIES) General Coordinator
9:40 -10:20 | INEP – General Coordination for Quality Control in Higher Education (CGCQES) General Coordinator
10:30 – 12:30 | National Commission for Higher Education Assessment (CONAES) President and Members
14:30 – 15:00 | Graduate Education Coordination (CAPES) President
15:00 – 16:00 | CAPES – Assessment Directorate Director
16:00 – 16:40 | CAPES – General Coordination for Assessment and Monitoring Coordinator
16:40 – 17:20 | CAPES – General Coordination for Support to Graduate Activities Coordinator (CGAP) Coordinator
17:20 – 18:00 | CAPES – General Coordination to Monitor and Assess Professional Masters (CGNE) Coordinator
Wednesday, 7 March, Brasília
10:00 – 11:30 | University Centre of Brasília (UniCEUB) Senior administration
11:45 – 12:30 | Northern University of Paraná (Unopar)
14:00 – 15:00 | National Association of Directors of Federal Institutions of Higher Education (ANDIFES)
15:00 – 17:40 | University of Brasília (UnB) Senior Administration Teaching staff
18:00 – 18:40 | Association of Private Higher Education Institutions (ANUP)
Thursday, 8 March, Recife

9:00 – 12:10

Federal University of Pernambuco (UFPE)

Senior Administration

Teaching staff

Students

14:30 – 18:00

Maurício de Nassau University Centre (UNINASSAU)

Senior Administration

Teaching staff

Students

Friday, 9 March, Recife
9:00 – 10:00 | Association of the owners of private institutions (ABMES) President Vice-president
10:15 – 12:30 | Federal Institute for Education, Science and Technology of Pernambuco (IFPE) Senior Administration Teaching staff
15:15 – 17:30 | Catholic University of Pernambuco (UNICAP) Senior Administration Teaching staff
Monday, 12 March, São Paulo

9:00 – 11:30

University of São Paulo (USP)

Senior Administration

Teaching staff

Students

14:00 - 15:00 | The Foundation for Research Support of the State of São Paulo (FAPESP) President, Director for the Technical and Administrative Council Scientific Director
15:10 – 16:30 | São Paulo Industry Association School (SENAI)
16:45 – 18:00 | Private University Association for the Development of Higher Education (ABRAES) President Executive Director
Tuesday, 13 March, São Paulo
9:00 – 12:20 | Anhanguera University/Kroton Senior management Teaching staff
14:00 – 15:00 | Association of private institutions in São Paulo (SEMESP)
15:15 – 16:45 | Getúlio Vargas Foundation - FGV
17:00 – 18:30 | Municipal University of São Caetano do Sul (USCS) Senior management
Wednesday, 14 March, São Paulo
9:00 – 10:00 | Brazilian Bar Association (OAB) Representative
10:10 – 11:00 | Brazilian Society for the Advancement of Science (SBPC) Representatives
11:10 – 12:00 | National Association of Graduate Studies and Research in Social Sciences (ANPOCS) Representatives

This work is published on the OECD iLibrary, which gathers all OECD books, periodicals and statistical databases.

Visit www.oecd-ilibrary.org for more information.

Contact Us • Alerts

Follow us on:

ORGANISATION FOR ECONOMIC CO-OPERATION AND DEVELOPMENT

The OECD is a unique forum where governments work together to address the economic, social and environmental challenges of globalisation. The OECD is also at the forefront of efforts to understand and to help governments respond to new developments and concerns, such as corporate governance, the information economy and the challenges of an ageing population. The Organisation provides a setting where governments can compare policy experiences, seek answers to common problems, identify good practice and work to co-ordinate domestic and international policies.

The OECD member countries are: Australia, Austria, Belgium, Canada, Chile, the Czech Republic, Denmark, Estonia, Finland, France, Germany, Greece, Hungary, Iceland, Ireland, Israel, Italy, Japan, Korea, Latvia, Lithuania, Luxembourg, Mexico, the Netherlands, New Zealand, Norway, Poland, Portugal, the Slovak Republic, Slovenia, Spain, Sweden, Switzerland, Turkey, the United Kingdom and the United States. The European Union takes part in the work of the OECD.

OECD PUBLISHING, 2, rue André-Pascal, 75775 PARIS CEDEX 16