Follow Us:

Technology-based assessment: Challenges and solutions

© Flickr/Michael Surran

21.02.2014

By Jean-Paul Reeff & Heiko Rölke, German Institute for International Educational Research (DIPF)        

The OECD Programme for the International Assessment of Students’ Competencies (PISA) (www.oecd.org/pisa/) has substantially shaped education policies and practices across the world. While PISA in its beginnings was a paper & pencil assessment only, very early discussions showed already the need for a computer-based - or in a more general sense - a technology-based assessment (TBA). However, the implementation of a technology-based assessment in a large-scale international study faced several serious obstacles. These obstacles become more substantial when trying to translate the expertise and the technology to the context of daily educational practice (Reeff, 2007).

This short article highlights three key obstacles, and the solutions that had been developed over time and are continually been refined and improved.

Firstly, there was no single delivery platform for educational assessments that would have fitted all relevant needs in the beginnings of PISA. Developing and maintaining such a platform proves to be a complex and costly endeavor, hard to finance within a study like PISA, and certainly beyond the budgets of single schools. Early R&D efforts in Luxembourg (en.wikipedia.org/wiki/Tao_platform ) yielded a prototype of an open-source platform (TAO), and that was later considered as a possible candidate to be used in large-scale assessments (LSA). A substantial investment of the German government into both the platform and related TBA activities speeded up its development and enabled its use in PISA 2009 for the Electronic Reading Assessment. Through further PISA cycles and its use in other studies (e.g. PIAAC) TAO developed into a mature platform to be used both in large-scale surveys and school-based assessments (www.tao.lu).

Secondly, there was little or no opportunity for user to develop authentic assessment items. While TAO encompasses all relevant facets of an entire assessment workflow (e.g. www.tao.lu/resources/tutorials/tao-platform-screenshots ), it offers very limited scope or assistance with regard to item authoring capacities. In order to overcome these shortcomings the TBA group at the German Institute for International for Educational Research (www.dipf.de/en/research/research-themes/technology-based-assessment ) developed a first prototype of an authoring system (ItemBuilder, tba.dipf.de/en/software/software/cba-itembuilder/cba-itembuilder-1 ) that facilitated the implementation of the PISA 2009 ERA items, both in terms of item development and the interface and integration with TAO platform. The fundamental idea of the ItemBuilder is to provide an easy-to-use authoring tool to enable test developers, teachers and other professionals developing computer-based items in WYSIWYG-manner without the help of programmers. Todays’ classes of items “include, but are not limited to: web environment simulations (including browser, search engines etc.), desktop application(s) simulation, complex problem solving items, and automata simulation (e.g. ticket automata, ATMs)” (Rölke, 2013) and, as an ongoing development, audio and video recording. The tool is free of charge for non-commercial projects; a more general open-access/open-source license is currently under discussion.

Finally, and above all in an LSA context, specific attention needs to be given to translation and adaptation issues. In studies like PISA and PIAAC, items are developed in English, but then have to be translated/adapted into one or more national languages in the countries participating in the study. In complex multi-media assessments, this task requires even more resources than in a paper-and pencil environment. Teachers who want to implement a technology-based classroom assessment and use existing items from other countries may face similar problems. In order to facilitate translation and adaptation processes, DIPF, in close cooperation with cApStAn (www.capstan.be), developed a specific workflow as well as supporting tools to fit into the TAO-ItemBuilder working model (Upsing et al., 2011).

Summary

In response to the need in international large-scale assessments, concepts and tools for complex multi-media-type assessments have been developed, continuously improved and validated in multiple national and international studies. The tools are open-source and/or free of charge for non-commercial educational and research purposes, and tangible for use in school and classroom settings outside the international studies.

References:

  • Reeff, J.P. (2007). Technische Lösungen für ein computer- und internetbasiertes Assessment-System. In: Bundesministerium für Bildung und Forschung (BMBF): Möglichkeiten und Voraussetzungen technologiebasierter Kompetenzdiagnostik. Bildungsforschung, Band 20. Berlin, 2007. (https://www.bmbf.de/pub/band_zwanzig_bildungsforschung.pdf )

  • Rölke, H. (2012). The ItemBuilder: A Graphical Authoring System for Complex Item Development. In T. Bastiaens & G. Marks (Eds.), Proceedings of World Conference on E-Learning in Corporate, Government, Healthcare, and Higher Education 2012 (pp. 344-353). Chesapeake, VA: AACE. (www.editlib.org/p/41614)

  • Upsing, B., Gissler, G., Goldhammer, F., Rölke, H., Ferrari, A. Localisation in International Large-Scale Assessments of Competencies: Challenges and Solutions. The International Journal of Localisation, Vol.10, Issue 1 (www.localisation.ie/sites/default/files/publications/Vol10_1UpsingGissleretAl.pdf)

Authors

Dr. Jean-Paul Reeff works as an independent consultant and as a senior consultant for the German Institute for International Educational Research (DIPF) in the field of innovation management and international cooperation. He holds degrees in psychology, physics and computer sciences, and a PhD in experimental psychology. reeff@dipf.de      

Heiko Rölke is the head of the Technology Based Assessment (TBA) group at DIPF, where he works as a senior software architect and manages several national and international assessment projects. In recent years, Heiko Rölke has designed and developed important parts of the computer-based item development and delivery for PISA 2009, PIAAC, and PISA 2012, amongst several smaller-scale studies. rölke@dipf.de