Follow Us:

Roles and Challenges of Large-Scale Assessments of Learning

During the recent capacity development workshop organized by the Network on Education Quality Monitoring in the Asia-Pacific (NEQMAP) from 23 to 26 September 2014 in Bangkok, Thailand on the topic of “Introduction to Large-Scale Assessments of Learning”, we spoke to three experts on the implementation of large-scale assessments of learning, the role of these assessments and challenges in their implementation.

(from left to right) Dr. John Cresswell, Dr. Suman Bhattacharjea and Dr. Steffen Knoll as session facilitators at the workshop. ©UNESCO/A. Tam

The experts in question, Dr. Suman Bhattacharjea, Research Director at the Annual Status of Education Report (ASER) Centre in India, Dr. John Cresswell, Deputy Research Director of the International Surveys Program at the Australian Council for Educational Research (ACER), and Dr. Steffen Knoll, Senior Research Analyst at the International Association for the Evaluation of Educational Achievement (IEA) in Germany, all served as facilitators during the workshop. 

Q. With your experience in working on large-scale assessments, how do you think the results of the surveys or large-scale assessments of learning can influence policy making or improve the quality of learning in countries?

Dr. Bhattacharjea: We have been conducting ASER survey since 2005 in India, and this has reached over 600,000 children from almost every rural district. Currently, ASER is the only annual source of information on children’s learning outcomes available in India. The ASER model is built around four key elements. First, it is a rapid one-on-one assessment of basic reading and math ability of children in the age group 5-16. The second element of the assessment is household-based, in order for the survey to include all children – those enrolled in government schools, those in private schools but also those who do not attend school regularly. Third, this assessment is conducted by ordinary citizens, which promotes a broader debate over schooling and learning in society. The last element is the simplicity of the model as a tool, which enables stakeholders including policy-makers, teachers and parents to understand the learning achievement of children and to help them develop their abilities. 

We believe that ASER has helped to place quality issues in the centre of the education agenda in India. For example, the chapter on education in the current Five Year Plan has a strong focus on ensuring learning outcomes and on regular assessment. Last year, all states were directed to conduct annual learning assessments to ensure basic reading and math abilities of children in early grades. 

Dr. Cresswell: Large-scale assessments such as PISA and the Trends in International Mathematics and Science Study (TIMSS) enable countries to compare their students’ strengths and weaknesses with those of other countries. In the case of Australia where there are government and non-government schools, the results have allowed a valid comparison of learning outcomes between the various sectors. Moreover, they have a unifying effect on the different education systems and have played an important role in providing stimulus for a national approach to curriculum.

Dr. Knoll: The assessment results help countries to explore the educational situation and then to decide on a policy. Data collected on student characteristics, achievement, attitudes, background etc. and further analysis on factors related to student learning from teachers, schools and parents allow countries to draw a very rich picture about the situation of a part of their education system. Participation in such studies on a regular basis opens the opportunity to measure trends over time and to determine whether various interventions are having their desired effect. 

In Germany, Ministries of Education of the 16 German states agreed to develop educational standards as to what students should be able to achieve at the end of compulsory schooling in grade 9. The results in TIMSS 1995 and PISA 2000 certainly had an influence on that decision. Furthermore, standards for centralized upper secondary exams are now on their way and will provide equal opportunities for students when applying for university, which is not easily accomplished in a federal system.

Q. What are the main challenges in translating assessment results into policy suggestions, changes or improvements? How would you suggest addressing these challenges?

Dr. Bhattacharjea: Developing countries face two enormous challenges: no consensus of the concept of “learning” and measurement. It is very difficult to challenge people’s assumptions about what should be measured. For instance, in India, the tendency is to design grade-level assessments, because it seems obvious that if children are in a given grade, we should be testing them at that level. It is only by looking closely at ground realities that one realizes that in the country most children are far below grade level. This is probably true of many other developing countries as well, and implies that in order to be useful for policy, assessments should include items that are at least one or two grade levels below the grade the child is actually studying in. In order to translate into policy suggestions, assessments need to establish what children can do as well as what they cannot do.

Dr. Cresswell: One of the main challenges in translating assessment results into policy changes is to ensure the engagement, over a long period of time, of political leaders. To tackle this challenge, the ways of reporting and disseminating assessment results need to be planned so that maximum impact is achieved both within political circles and with the public. In this regard, different reports need to be produced with different target audiences in mind.

Dr. Knoll: Large-scale assessments which can measure student performance on an internationally comparable scale have a critical role in identifying possible factors that are associated with students’ learning achievement. Countries need to investigate the factors through the analysis of the assessment results considering the educational context of the country. Intervention studies, for example with a design including both pre-test and post-test, or longitudinal studies are then needed to make sure that changes in teacher training, teaching, school resources and parental involvement will lead to better learning and performance of the students. If that can be confirmed, policy conclusions can be drawn and implemented at national or sub-national level.

Q. What are some of the important factors that developing countries should think about in order to successfully implement national and international large-scale assessments?

Dr. Bhattacharjea: For the successful implementation of large-scale assessments, it is important to begin with a set of questions that can assist in identifying potential determinants of learning outcomes. Based on them, a set of assessment tools and processes should be chosen; therefore we can analyze the findings and use them to improve children’s learning outcomes. We need to learn and plan how to use the data before implementing the assessments. Otherwise, producing assessment data that nobody uses is an enormous waste of resources.

Dr. Cresswell: Countries need to make a long-term commitment to the implementation of national and international large-scale assessments, including financial commitment. To that end, countries are required to establish a national centre for the implementation and to ensure that all stakeholders including teachers and parents understand the aims and roles of the assessments.

Dr. Knoll: When implementing national and international large-scale assessments, countries should first appoint a department/ministerial unit that will be held accountable for the project. A scientific coordinator appointed by the department then needs to form a network that provides support and advice from institutions and experts in the areas of policy, curriculum and data analysis in the country. At the same time, a team to accomplish all studies including sampling, data analysis and report writing has to be established. The results should then be reviewed in light of the national context, and first policy conclusions can be formulated, prioritized and then investigated further. Some results, especially pointing at relationships between student learning and associated factors, need to be investigated further by using intervention studies.

Q. From what you have observed and learned during this workshop, what do you think that NEQMAP can do to support developing countries in improving large-scale assessments of learning and their use?

Dr. Bhattacharjea: It is very important to emphasize the fact that there is no “one size fits all” assessment model. The model needs to be implemented based on context, assumptions, situations and purposes. NEQMAP can help to explore issues such as what different models can provide and in what situations they are useful in more depth. Meanwhile, for developing country contexts, the first vital question is whether pen and paper tests are the place to begin, or whether oral assessments are more relevant to the context of children.

Dr. Cresswell: NEQMAP provides members with a very useful platform for gaining and sharing information about the best ways to carry out assessments and providing access to experts who have long experience in the area. It can also consider the feasibility of supporting countries by providing advice on different ways to fund their participation in large-scale assessments. 

Dr. Knoll: This workshop was an excellent starting point not only for learning from different countries and sharing their needs, challenges and concerns in implementing large-scale assessments, but also for providing solutions for improvement. For some developing countries, funding is one of the key issues in implementing the assessments. To address this issue, NEQMAP may be able to offer information and practical knowledge. The network can also play an important role in bringing different stakeholders together, collecting the needs of each country and helping build up relationships between stakeholders and countries.

For more information, please contact Ramya Vivekanandan [r.vivekanandan(at)unesco.org] at the Education Policy and Reform Unit (EPR).


Written by:  Ryoko Suzuki [r.suzuki(at)unesco.org]


Related Links:

• Capacity development workshop "Introduction to Large-scale Assessments of Learning"

• Network on Education Quality Monitoring in the Asia-Pacific (NEQMAP)

• Annual Status of Education Report (ASER) Centre

• Australian Council for Educational Research (ACER)

• International Association for the Evaluation of Educational Achievement (IEA)



31.10.2014