External review of the Netherlands Quality Agency (NQA)

by the European Association for Quality Assurance in Higher Education (ENQA)

Annex I: TERMS OF REFERENCE

July 2022

1. Background and context

Netherlands Quality Agency (NQA) is a quality assurance agency based in Utrecht, in the centre of the Netherlands. NQA focuses on providing services to (mainly) institutions of higher professional education. As a(n) (external) quality assurance agency, NQA particularly organizes and co-ordinates assessments of degree programmes on the basis of the formal accreditation framework that has been established by the relevant authorities, the Ministry of Education, Culture & Science and the Netherlands-Flemish Accreditation Organization (NVAO).

NQA originates from the Netherlands Association of Universities of Applied Science (in Dutch: ‘Vereniging Hogescholen’). In accordance with the amended Act on Higher Education and Scientific Research (2002), assessments of (existing) degree programmes were to be conducted by independent quality assurance agencies. Subsequently, NQA was formally founded December 2003. Since the foundation NQA has performed over 1200 assessments of degree programmes (associate degree, bachelor, master) of institutions of higher professional education. The results of these assessments are laid down in a report that institutions/educational programmes use to get re-accredited by NVAO.

Assessments are the solid key objective of the activities of NQA. In addition, the company offers a variety of consultancy and training services. Clients of these services are mainly institutions of higher professional education, although also some other (educational) institutions use these services of NQA. On a yearly base these services are just 5% tot 10% of our sales volume.      

This application of NQA for the ENQA membership focuses on the assessment (audit/evaluation) activities. A thorough (ENQA) review of this cannot be done without a proper view of the Dutch system of higher education and the system of external quality control. An extensive description will be part of the NQA self-assessment report (SAR). In the meantime we would like to refer to a description by EP Nuffic that gives an introduction to these two elements:

Higher Education in the Netherlands

Higher education in the Netherlands is offered at two types of institutions: research universities and universities of applied sciences. Research universities include general universities, universities specialising in engineering and agriculture, and the Open University. Universities of applied sciences include general institutions as well as institutions specialising in a specific field such as agriculture, fine and performing arts or teacher training. Whereas research universities are primarily responsible for offering research-oriented programmes, universities of applied sciences are primarily responsible for offering programmes of higher professional education, which prepare students for specific professions. These tend to be more practice oriented than programmes offered by research
universities.
 

In this binary, three-cycle system, bachelor’s, master’s and PhD degrees are awarded. Short-cycle higher education leading to the associate’s degree is offered by universities of applied sciences. Degree programmes and periods of study are quantified in terms of the ECTS credit system.

System of external quality assurance

A guaranteed standard of higher education, and alignment with the Qualifications Framework for the European Higher Education Area, is maintained through a system of legal regulation and quality assurance, in the form of accreditation. The Ministry of Education, Culture and Science is responsible for legislation pertaining to education. The agriculture and public health ministries play an important role in monitoring the content of study programmes in their respective fields. Quality assurance is carried out through a system of accreditation, administered by the Accreditation Organisation of the Netherlands and Flanders (NVAO).

According to the Dutch Higher Education Act, all degree programmes offered by research universities and universities of applied sciences must be evaluated according to established criteria. These evaluations are mostly carried out by external quality assurance agencies like NQA.  Programmes that meet the criteria are accredited: i.e. recognised for a period of six years. Only accredited programmes are eligible for government funding; students receive financial aid and graduate with a recognised degree only when enrolled in, and after having completed, an accredited degree programme. All accredited programmes are listed in the Central Register of Higher Education Study Programmes (CROHO).
 

Since January 2011, the Netherlands has a renewed accreditation system. The process described above still applies, but beginning in 2011, higher education institutions can request the NVAO to conduct an ‘institutional quality assessment’ to determine the extent to which the institution is capable of guaranteeing the quality of the programmes it offers. Programmes offered by institutions that receive a positive evaluation still have to be accredited, but the accreditation procedure takes less time and is not as extensive. The latest renewed accreditation system was introduced in 2018 with smaller adjustments. The SAR of NQA will describe these newest adjustments. 

NQA has been a member of ENQA since 21 June 2018 and is applying for renewal of membership.

2. Purpose and scope of the evaluation

This review will evaluate the extent to which NQA (the agency) fulfils the requirements of Parts 2 and 3 of the Standards and Guidelines for Quality Assurance in the European Higher Education Area (ESG). Consequently, the review will provide information to the ENQA Board to aid its consideration of whether the membership of NQA should be reconfirmed.

2.1 Activities of the agency within the scope of the ESG

To apply for ENQA membership, this review will analyse all of the agency’s activities that fall within the scope of the ESG, e.g., reviews, audits, evaluations or accreditations of higher education institutions or programmes that relate to teaching and learning (and their relevant links to research and innovation). All activities are reviewed irrespective of geographic scope (within or outside the EHEA) or whether they are obligatory or voluntary in nature.

The following activities of the agency must be addressed in the external review:

Assessments (audits/evaluations) of degree programmes of institutions for higher education as a(n) (external) quality assurance agency.

Assessment of international degree programmes

3. The review process

The review will be conducted following the methodology of ENQA Agency Reviews. The process is designed in line with the Guidelines for ENQA Agency Reviews.

The review procedure consists of the following steps:

• Formulation of, and agreement on the Terms of Reference for the review between NQA and ENQA (including publishing of the Terms of Reference on ENQA’s website[1]);
• Nomination and appointment of the review panel by ENQA;
• Self-assessment by the agency, including the preparation and publication of a self-assessment report;
• A site visit of the agency by the review panel;
• Preparation and completion of the final review report by the review panel;
• Scrutiny of the final review report by ENQA’s Agency Review Committee;
• Publication of the final review report;
• A decision from the ENQA Board on ENQA membership;
• Follow-up on the panel’s recommendations to the agency, including a voluntary progress visit.

3.1 Nomination and appointment of the review team members

The review panel consists of four members: one or two quality assurance experts (at least one of which is currently employed by an ENQA member agency), an academic employed by a higher education institution, a student member, and eventually a labour market representative (if requested). One of the members will serve as the chair of the review panel, and another member as a review secretary. For ENQA Agency Reviews at least one of the reviewers is an ENQA nominee (most often the QA professional[s]). At least one of the reviewers is appointed from the nominees of either the European University Association (EUA) or the European Association of Institutions in Higher Education (EURASHE), and the student member is always selected from among the ESU-nominated reviewers. If requested, the labour market representative may come from the Business Europe nominees or from ENQA. An additional panel member may be included in the panel at the request of the agency under review. In this case, an additional fee to cover the reviewer’s fee and travel expenses is applied.

The panel will be supported by the ENQA Review Coordinator who will monitor the integrity of the process and ensure that ENQA’s requirements are met throughout the process. The ENQA staff member will not be the secretary of the review and will not participate in the discussions during the site visit interviews.

Current members of the ENQA Board are not eligible to serve as reviewers.

ENQA will provide the agency with the proposed panel composition and the curricula vitarum of the panel members to establish that there are no known conflicts of interest. The reviewers will have to agree to a non-conflict of interest statement that is incorporated in their contract for the review of this agency.

3.2 Self-assessment by the agency, including the preparation of a self-assessment report

The agency is responsible for the execution and organisation of its own self-assessment process and must adhere to the following guidance:

• Self-assessment is organised as a project with a clearly defined schedule and includes all relevant internal and external stakeholders;
• The self-assessment report is expected to contain:

1. a brief description of the HE and QA system;
2. the history, profile, and activities of the agency;
3. a presentation of how the agency addresses each individual standard of Parts 2 and 3 of the ESG for each of the agency’s external QA activities, with a brief, critical reflection on the presented facts;
4. opinions of stakeholders;
5. reference to the recommendations provided in the previous review and actions taken to meet those recommendations;
6. a SWOT analysis;

• reflections on the agency’s key challenges and areas for future development.
• All the agency’s external QA activities (as defined under section 2.1) are described and their compliance with the ESG is analysed in the SAR.
• The report is well-structured, concise, and comprehensive. It clearly demonstrates the extent to which the agency performs its tasks of external quality assurance and meets the ESG.

The self-assessment report is submitted to the ENQA Secretariat, which has two weeks to carry out a screening. The purpose of a screening is to ensure that the self-assessment report is satisfactory for the consideration of the panel. The Secretariat will not judge the content of information itself but rather whether or not the necessary information, as outlined in the Guidelines for ENQA Agency Reviews, is present. If the self-assessment report does not contain the necessary information and fails to respect the requested form and content, the ENQA Secretariat reserves the right to ask for a revised version within two weeks.

The final version of the agency’s self-assessment report is then submitted to the review panel a minimum of eight weeks prior to the site visit. The agency publishes the completed SAR on its website and sends the link to ENQA. ENQA will publish this link on its website as well.

3.3 A site visit by the review panel

The review panel will draft a proposal of the site visit schedule which must be submitted to the agency at least six weeks before the planned dates of the visit. The schedule is to include an indicative timetable of the meetings and other exercises to be undertaken by the review panel during the site visit, the duration of which is usually 2,5 days. The approved schedule must be given to the agency at least one month before the site visit to properly organise the requested interviews.

In advance of the site visit (ideally at least two weeks before the site visit), the panel will organise an obligatory online meeting with the agency. This meeting is held to ensure that the panel reaches a sufficient understanding of:

• The specific national/legal context in which the agency operates;
• The specific quality assurance system to which the agency belongs;
• The key characteristics of the agency’s external QA activities.

The review panel will be assisted by the ENQA Review Coordinator during the site visit. The review coordinator will act as the panel’s chief liaison with the agency, monitor the integrity of the review process and its consistency, and ensure that ENQA’s overall expectations of the review are considered and met.

The site visit will close with a final debriefing meeting in which the panel outlines its general impressions and provides an overview of the judgement on the agency’s ESG compliance. The panel will not comment on whether or not the agency would be reconfirmed membership with ENQA.

3.4 Preparation and completion of the final evaluation report

Based on the review panel’s findings, the review secretary will draft the report in consultation with the review panel. The report will follow the purpose and scope of the review as defined under sections 2 and 2.1. It will also provide a clear rationale for the panel’s findings concerning each standard of Parts 2 and 3 of the ESG.

A draft will first be submitted to the ENQA Review Coordinator who will check the report for consistency, clarity, and language, and it will then be submitted to the agency – usually within 10 weeks of the site visit – for comment on factual accuracy and grave misunderstandings only. The agency will be given two weeks to do this and should not submit any additional material or documentation at this stage. Thereafter, the review panel will take into account the agency’s feedback on possible factual errors and finalise and submit the review report to ENQA.

The report should be finalised within three months of the site visit and will normally not exceed 40-50 pages in length.

3.5. Publication of the report and a follow-up process

The agency will receive the review panel’s report and publish it on its website once the Agency Review Committee has validated the report. The report will also be published on the ENQA website together with the statement of the Agency Review Committee validating external review reports by assessing the integrity of the review process and checking the quality and consistency of the reports. Importantly, during this process, and prior to final validation of the report, the Agency Review Committee has the option to request additional (documentary) evidence or clarification from the review panel, review coordinator or the agency if needed. The review report will be published on ENQA website regardless of the review outcome.

As part of the review’s follow-up activities, the agency commits to react on the review recommendations and submit a follow-up report to ENQA within two years of the validation of the final external review report. The follow-up report will be published on the ENQA website.

The follow-up report may be complemented by an optional progress visit to the agency performed by two members of the original panel (whenever possible). The visit, which normally takes place 2-3 years after the verification of the final external review report (and after submission of the follow-up report), aims to offer an enhancement-oriented and strategically driven dialogue that ordinarily might be difficult to truly integrate in the compliance-focused site visit. The progress visit thus does not have the objective of checking the agency’s ESG compliance or how the agency has followed up on the recommendations, but rather provides an arena for strategic conversations that allow the agency to reflect on its key challenges, opportunities, and priorities. Should the agency not wish to take advantage of this opportunity, it may opt out by informing the ENQA Review Coordinator about this.

4. Use of the report

ENQA will retain ownership of the report. The intellectual property of all works created by the review panel in connection with the review contract, including specifically any written reports, will be vested in ENQA.

To apply for ENQA membership, the agency is also requested to provide a letter addressed to the ENQA Board outlining its motivation for applying for membership and the ways in which the agency expects to contribute to the work and objectives of ENQA during its membership. The ENQA Board will take a decision on the ENQA membership of the agency based on the final review report, the application letter, and the statement from the Agency Review Committee. The decision on membership will be published on ENQA’s website.

5. Indicative schedule of the review

Agreement on Terms of Reference - July 2022

Appointment of review panel members - September 2022

Self-assessment completed - 24 October 2022

Screening of SAR by ENQA Review Coordinator - October/November 2022

Preparation of the site visit schedule and indicative timetable - December 2022

Briefing of review panel members - December 2022

Review panel site visit - Second half of February 2023

Draft of review report and its submission to ENQA Review Coordinator for verification of its compliance with the Guidelines - April 2023

Draft of review report to be sent for a factual check to the agency - May 2023

Agency statement on the draft report to the review panel (if necessary) - May 2023

Submission of the final report to ENQA - June 2023

Validation of the review report by the Agency Review Committee - June 2023

Publication of report - June/July 2023

Decision on ENQA membership by the ENQA Board - September 2023

[1] The agency is encouraged to publish the ToR on its website as well.