Follow Us:

Using indicators to assess impact of ICT in education

Below is an attempt to synthesise several countries’ efforts and initiatives in conducting research to evaluate the use of ICT in education, and, for some countries, the development of ICT indicators.

Among these countries are:

Australia, Canada, CIS and Balkan countries, India, Indonesia, Japan, Korea, Republic of, Malaysia, New Zealand, The Philippines, Singapore, Slovenia, South Africa, Thailand, The United Kingdom, The United States, Uzbekistan and Viet Nam.

The development or potential development of indicators in the region follows three paths:

  • Ministries of Education or the Government set up Task Forces or committees, to take charge of developing indicators measuring the use and impact of ICT in education
  • The vision and goals of the ICT programme serve as basis for the formulation of ICT indicators
  • The impact of the use of ICT in education is determined through surveys and research and thus, serves as basis for the formulation of national indicators

While the government of Korea has “Cyber Korea 21” and the “White Paper”, the government of Japan has “eJapan Priority Programme” and Europe has the “eEurope” initiative. These programmes are all aimed at improving the ICT sector and carry with them built-in indicators for measuring the achievement of these goals.

Today, most countries include ICT integration, either in their national policies or in laws pertaining to the education sector. In Australia, at the federal level, the Commonwealth government has set goals for schools in relation to ICT development. The government wants students to leave school as confident, creative and productive users of new technologies, particularly ICTs, as well as understanding the impact of those technologies on society(http://www.curriculum.edu.au/mceetya/nationalgoals/natgoals.htm). Schools are expected to integrate ICT into their operations. In Thailand, the national Education Act of 1999 formed the core of education reform towards ICT use in education, as guided by three principal strategies: value-added, equity and quantum-jump (http://www.unescobkk.org/education/ict/resources/policy/ap/Learntec-Tongyoo.pdfLearntec-Tongyoo.pdf). Similarly, the Law on Education in Uzbekistan outlined the inclusion of ICT in the education sector. Meanwhile, the Philippines Department of Education has also formulated policies for the use of ICT. The same trend is seen in Indonesia, Malaysia, Uzbekistan, Vietnam and others, where the national government sets goals for ICT integration in education.

Kinds of indicators developed, based on visions or goals and surveys
Within the region, the Republic of Korea is the most advanced in terms of indicator development on the use of ICT in education. The Ministry of Education and Korean Educational Research and Information Service (KERIS - http://www.keris.or.kr/english) have formulated and used a set of indicators, categorised into support, input, utilisation or output. The support indicators include training hours given to teachers and principals, budget allocation, the year plan and incentives for teachers. The input indicators consist of the ratio between computers and students, Internet connectivity and speed, educational software and the number of applications used. The utilisation indicators include subjects using multimedia, the percentage of classes using a Web board or computer labs, the percentage of teachers joining ICT associations and discussion forums, student usage of computers and usage of the internet. Output indicators include teachers and students having an email address, a homepage and ICT certificates and students also having completed a 32 hours ICT course.

In Australia, the Ministerial Council of Education, Employment, Training and Youth Affairs (MYCEETA) is currently looking into the use of performance measures for student achievement relating to new technologies. Taking the National Goals for Schooling as the starting point, the NEPMT commissioned a project in 2000 to develop key performance measures for monitoring the ICT knowledge and skills of Australian school students. The outcome of this process was the report Monitoring Progress towards the National Goals for Schooling: Information and Communication Technology (ICT) Skills and Knowledge. Basically, the report describes the context of learning ICT skills and knowledge as an outcome of schooling; identifies and describes performance measurement approaches, definitions and sources of data currently used in relation to schooling, research and national and international reporting on ICT skills and knowledge; evaluates the ICT measures currently in use in the context of the National Goals for Schooling in the 21st Century; proposes a strategy for monitoring the profile of ICT skills and the knowledge of school students; and identifies further work required to establish an operational monitoring programme.

There have also been several initiatives undertaken to measure the impact and use of ICT in education in different States of Australia. The trends show that these indicators are increasingly focusing on qualitative indicators, measuring teacher and student outcomes. For example, the University of Sydney looked into whether student-centred learning took place, along with student improvement in high-order thinking skills. As far as the teachers are concerned, the indicators assess changes in teaching practice and improvement in the ability to use emerging technologies. However, connectivity remains a strong indicator in Australia, focusing on the number of teachers with a notebook, the computer-student ratio, bandwidth available in schools and the type of Internet connection.

In Malaysia, the creation of the Smart School System serves as benchmark for ICT integration in schools (http://www.mdc.com.my/msc/flagship/ss.html). The Malaysian Smart School programme is aimed at reinventing learning institutions, in terms of teaching practices and school management, preparing children for the Information Age. Some of the initial proposed indicators by the ICT project are both quantitative and qualitative. They focus on the use of ICT in curriculum, in pedagogy, in assessment and materials. The quantitative indicators include decreased drop out rates and recorded achievement gains each year, among others, while the qualitative indicators include caring and competent teachers and a broad, flexible curriculum that caters to the differing needs and abilities of students.

In the Philippines, the Senate Committee on Education, Arts and Culture commissioned SEAMEO-INNOTECH to conduct the national survey to profile the ICT capabilities of elementary and secondary schools in the Philippines. SEAMEO-INNOTECH was responsible for formulating the indicators/ items on the survey (http://www.seameo-innotech.org/frames.html). These include the existence of an enabling environment, the computers skills of school personnel, the presence of computers in schools, instructional or academic use of computers, ICT infrastructure and Internet connectivity. The Department of Science and Technology, in collaboration with the Department of Education, has also conducted a survey, not only looking at the infrastructure and hardware support indicators, but also with regard to how computers had been used and integrated into the curriculum, as well as in teaching and learning.

In Thailand, initial use of indicators through a survey included Internet connectivity in schools, the computer-student ratio, usage of ICT in schools and school administration and connection to SchoolNet as main issues. The annex found at the end of this document also shows a variety of potential indicators which fall under their ICT project’s three principles/strategies, namely, value-added, equity and quantum-jump strategies. A committee was formed to concentrate on formulating indicators to measure the impact of ICT use in schools.

Uzbekistan, through its Ministry of Public Education, has carried out monitoring and evaluation through the 14 Provincial Departments of Education and 15 In-service Training Institutes. Under these, indicators have been examined, such as the availability of computers in schools, the number of students using computers, the number of computer classes, integration of ICT in subjects and teacher training. It was reported that the evaluation of student computer ability and performance is difficult, as different computers are being used throughout the country.

In India, the ICT programme statement of goals brought about potential indicators that include the presence of MIS and software, ICT in textbooks, ICT in computer subjects and in-service and pre-service training of teachers.

Though Indonesia has not developed its set of indicators, its ICT in education programme goals and objectives also imply that the following indicators can be potentially useful: student’s understanding of the benefits of ICTs - disadvantages as well as challenges; student’s use of ICT knowledge in acquiring, processing, arranging, distributing and keeping information; and student’s application of knowledge, skills and attitudes in designing information technology systems and solving problems relating to ICT.

Similarly, while Viet Nam has not developed its indicators, its ICT goals imply indicators in the following areas: ICT as a subject in general education, number of trained teachers and establishment of ICT departments in universities to train teachers, provision of computer labs in schools and the application of ICT in school management.

In Singapore, as the process of IT integration in schools reaches a considerable level of maturity, the question being asked is: How do we integrate IT in schools so learning opportunities are optimised? Research undertaken by the National Institute of Education and Ministry of Education, entitled “Effective Integration of IT in Singapore Schools” (http://eduweb.nie.edu.sg/projects/ITintegration/), focused on the general pedagogical practices and socio-cultural policies of IT integration in Singapore schools, not just the innovative and best practices. The indicators used included infrastructure and resources, school IT culture, staff development, teacher use and student use. The research sought answers to the following questions or indicators of a qualitative nature:

  • What are the pedagogical practices of teachers & students that promote or hinder the integration of IT?
  • What are the roles of the human participants, activities and tools in such an environment?
  • How does a change in the curriculum promote a culture that facilitates integration of IT?
  • How does a change in the mode of assessment affect integration of IT?

The UNESCO Institute of Information Technology in Education (http://www.iite.ru) based in Moscow undertook a survey of the use of ICT in education in Baltic and CIS countries, making use of the following indicators: a) Official documents on ICT usage in education; b) ICTs in curricula - as a separate subject or to support other subjects; c) equipment (computer-student ratio, computer availability in classrooms; multimedia system and LAN connection; d) software (percentage of schools with DOS and Windows and using software for teaching subjects, and software designed by local and foreign experts); e) global communication means (percentage without Internet access; with limited Internet access – only e-mail; access via dial-up or dedicated line, and those with web pages); and f) personnel development - percentage of elementary school teachers, subject teachers, and teachers of Informatics and EI administration who have undertaken the computer literacy course, for certain hours during a certain time frame, and the computer skills of elementary school teachers, subject teachers and teachers of Informatics and administration.

Standards for technology use
What kinds of changes in knowledge, skills and competencies are required from teachers and students in the use if ICT? In order to serve as a basis for developing knowledge, skills and competencies in the use of ICT, as well as in comparing outcomes of ICT use against goals set, few countries have standards for technology use.

In Australia, however, the Victoria government has a curriculum standards framework (CSF) developed by the Victorian Curriculum and Assessment Authority, which identifies what students should know and be able to do in eight key learning areas from preparatory year to year 10. In New South Wales, the Government has foreshadowed the introduction of an external Computer Skills Assessment for all years 6 and 10 students to determine their knowledge, skills and understanding in the use of ICT. Western Australia has a Competency Framework for Teachers and in Queensland these competencies are identified in Minimum Standards for Teachers-Learning Technology (http://education.qld.gov.au/curriculum/ learning/technology/ mst_int.html#introduction). Queensland also produced a document ICTs for Learning: School Information Kit 2002–2003, which details benchmarks established around teaching, learning and the curriculum; learning and development; ICT infrastructure; connectivity; ICT support; and innovation. Furthermore, the development of indicators will be based on the following: a) definition of content domain, including functional skills in a cross curricular context, manageability in its complexity, delineates between Year 6 and Year 10 in terms of the curriculum context and the hierarchy of skills identified; and b) the development of a scale of competence, including basic and functional skills, as well as complex thinking, and facilitating the reporting of a range of achievement.

The United States and Europe utilise a set of standards for the technology use of students, teachers and school administrators. The International Society for Technology in Education (ISTE) came up with the National Education Technology Standards (NETS) Project (http://cnets.iste.org/) the primary goal of which is to enable stakeholders in PreK-12 education to develop national standards for the educational uses of technology that will facilitate school learning. (See attached for more details on the ISTE NETS standards.) The fact that the other countries did not indicate any existing standards that can monitor the use of technology reflects the lack of qualitative indicators. Many countries merely focus on providing the technology that is making computers and Internet access available.

Levels of integration
In most of the countries researched on, the studies have been limited to ICT indicators used in the primary and secondary level. There are, however, instances where other levels are included, such as the case of Republic of Korea, where university level is included and in Slovenia, pre-school. In the United States, those in physical education and special education subjects in the primary and secondary levels were excluded. Japan is noteworthy, as its ICT policy includes even special schools for the blind and the disabled. Australia incorporates those in non-formal education in their studies. This situation tells us something about the issue of equity in the use of ICT in education. There are times when the government policy itself can serve as an aid or barrier to equity in access.

ICT infrastructure, hardware and connectivity

The number of computers or Internet connections in schools dominates statistics. In fact, only six countries specified indicators pertaining to other ICT technologies, such as telephone lines, fax machines, LCD projectors and the like. These countries are Australia, India, New Zealand, Republic of Korea, Thailand and the Philippines.

Perhaps, future studies should also look into other types of ICTs in addition to computer-based and Internet. In doing so, there should be a clear definition of what ICT is. The following definitions can serve as a guide:
Information Technology (IT) is the term used to describe the items of equipment (hardware) and computer programs (software) that allow us to access, retrieve, store, organise, manipulate and present information by electronic means. Personal computers, scanners and digital cameras fit into the hardware category. Database storage programs and multimedia programs fit into the software category.

Communication technology (CT) is the term used to describe telecommunications equipment, through which information can be sought and accessed, for example, phones, faxes, modems and computers
Information literacy is the combination of knowledge, understanding, skills, and attitudes that students need to fully contribute as members of society in the information age. When students become information literate, they develop an ability to select, interpret, evaluate, manipulate and present information.

Teacher’s use of ICT
Studies on how teachers use ICT are also included in most ICT integration programmes. Many looked into how teachers have used ICT in teaching specific subjects; into how they have employed Internet resources in teaching; how they have used email in communicating with students and other teachers; and whether they have developed their own homepages. In Australia for example, teachers make use of self-paced learning materials on ICT use in CD-format. In other countries, preference is given to teacher applicants who are already computer literate, such as in the case of the Philippines. In Korea and Australia, teachers are provided with PCs.

Targets of indicators surveys
Respondents for the different surveys in the US, New Zealand, the UK, South Africa and Europe usually include teachers. Students are included in surveys to determine their perceptions towards ICT, such as in theUK and South Africa. In other cases, school heads are taken as respondents, such as in the Philippines and New Zealand. It was not clear from the results in other countries, such as Canada, Korea, Japan and Slovenia who their respondents were.

Types of indicators
While some countries use qualitative in addition to quantitative indicators, their use is obviously limited. Quantitative data usually includes ICT infrastructure and connectivity, consisting of hardware, and the physical networks that connect the computers locally and globally. Common indicators used for measuring or determining ICT infrastructure include:

  • Availability of computer hardware; ratios of computer/student, computer/classroom, computers/teacher; type of computers (standalone, multimedia in network)
  • Availability of connectivity and bandwidth of the computers. Type of bandwidth for connections includes high-speed, broadband, wireless.

However, it is equally important to examine indicators that will show how ICTs have been used not only as a basic operational tool, but also as a communications tool, which promotes the development of creativity, interactivity, collaborative learning, critical thinking and problem-solving.

There are countries that are already attempting to include attainment and ICT impact in indicators. Canada included “obstacles to fuller use of information technologies” in its list of indicators (http://www.cmec.ca/stats/pceip/1999/Indicatorsite/english/pages/page19e.html), while in Europe, “teacher confidence in the use of ICT, change in teaching methods and desirable ICT skills” is assessed (http://www.eurydice.org/Documents/TicBI/en/FrameSet.htm). In New Zealand, “obstacles faced by teachers in using ICT, factors that encourage use of ICT, student’s positive views about technology activities in school” are among the qualitative indicators.

South Africa (http://education.pwv.gov.za/teli2/research/ICT%20audit/ICTaduit.htm) attempts to measure practical, foundational and reflexive competencies of educators, along with how well students enjoy ICT-related activities and learner’s perception of how the use of computers improves learning and attitudes.

Meanwhile, the UK (http://www.dfes.gov.uk/statistics/DB/SBU/b0296/) looks at teacher confidence in ICT use and the benefits of ICT in subjects, while in the US, most qualitative indicators pertain to the teacher’s perceived value of internet use relative to the teacher’s educational background, school level taught, location of access etc.

The Republic of Korea includes output indicators, teachers and students now having email addresses and a homepage to deliver information and exchange school work and ideas, as well as more participation in ICT extra curricular activities (http://www.icce2001.org/cd/pdf/p12/ KR115.pdf).

Research in Japan, the Philippines, Slovenia and other countries show that these countries are still concentrating on providing ICT infrastructure, that is, making the technology available in their respective countries, rather than measuring impact and effectiveness.

Qualitative indicators mentioned in the research:

  • How much students think they have improved
  • Change in teaching methods
  • Desirable ICT skills
  • Factors that encourage telecommunications use
  • Students who had positive views about technology activities in schools
  • Teachers agreeing that telecommunications technologies can enhance learning and teaching
  • How much learners think they have improved in various activities
  • Teacher confidence in the use of ICT
  • Development of practical, foundational and reflexive competencies
  • Impact of computers on educators and learners
  • Barriers to computer related activities

Some indicators appear to be unique or were only mentioned for some countries. In Europe, for example, ICT indicators include the ICT policy and strategy of each country in the CIS and Balkan study, looking at Denmark, Sweden, Finland, Norway and the UK. These indicators measure objectives of each country in national ICT projects, along with the schedule of implementation.

In the Republic of Korea, the establishment of an education network (Korea Education Network) serves as one of the ICT indicators for higher education. Korea also uses the establishment of a networked research system (Research Information Service System) for higher education as another indicator.

The establishment of a network can be considered a quantitative indicator in the sense that it is a physical phenomenon, where hardware or portals/platforms are provided for networking in a digital form to take place. However, networking can also be a qualitative indicator if the focus is on developing competence in disseminating knowledge and information quickly and to large numbers of people. The level or quality of networking among organisations can also be used as a qualitative indicator.

Furthermore, most of the countries give themselves a deadline or a yearly achievement target. For example, in the Republic of Korea, the targets for the year 2000 are to provide PCs and other infrastructure to schools, for 2002, the establishment of a Nationwide Education Administration Information System and for 2005, the setting up of the Integrated Human Resources Information Network.

The use of ICT in the school system in the Asia-Pacific Region, as in the other Regions of the world, is widespread and continuously growing. Many believe that ICT will empower teachers, transforming teaching and learning processes from being highly teacher-dominated to student-centered. Arguably, the quality of student learning will improve tremendously, as use of ICT will create opportunities for student to develop their creativity, problem-solving abilities and other higher-order thinking skills.

It appears though that while ICT has revolutionised business, industry and entertainment, indicators to prove that it has had the same effect on student outcomes are lacking as yet. Moreover, ICT has brought with it its own problems, such as the oft-mentioned “digital divide,” exacerbating the problem of access to ICT between rich and poor, male and female and between and among teachers and school administrators. Moreover, as countries continue to invest in ICT for use in education, drawing financial resources from a variety of sources, including the private sector and bilateral funding agencies, there is even greater need for performance indicators to monitor the use and effects of ICT, demonstrating accountability to these various funding sources and to the public.

While these problems are known to our policy makers and educators, information on the extent of these problems remains scant, due to lack or absence of monitoring and evaluation systems concerning the use of ICT in schools and its impact on teaching and learning. There is an urgent need, therefore, for a monitoring and evaluation (M and E) system, if current efforts to make ICT use an integral part of the education system are to succeed. Such M and E systems should start with formulating a set of indicators of ICT use and impact in education.

This effort is not new in other parts of the world, such as in the USA, Europe, the UK, Canada, Australia, New Zealand, and the like. (A listing of ICT indicators developed in these countries is given as Appendix I to this paper.) However, in most of the Asia-Pacific countries, there has been little systematic work in the use of ICT indicators in the field of education, except that of the Republic of Korea and soon, Australia.

 

Given that Asia-Pacific countries differ widely in regard to the scope and variety of use of ICT in education, it would be unrealistic and inappropriate to attempt to formulate a uniform set of indicators that can be used to frame data collection for ICT in education projects. Rather, it is better to arrive at a consensus on common core indicators that can be used regardless of the ICT utilisation stage which a country has reached. Important criteria to be observed in formulating these core indicators would include local relevance, reliability and robustness.

Assessment and evaluation methods

In most studies, traditional or standard assessment methods were used. Except in a primary school in Sunnyside Elementary School, Putnam, Washington, assessment rubrics were designed for specific projects, lessons, and/or classroom experiences, in addition to standard assessment tools. In this study, parents were also involved in the assessment process, preparing reports of their children’s progress.

The ways that technologies are being used in schools change the teacher’s role from that of technology-as-teacher to technology-as-partner in the learning process. As students increasingly use technologies as learning tools, they will produce technology-based artifacts - student-constructed knowledge bases. These knowledge bases are rich, multi-modal indicators of what students have learned. Moreover, as learning becomes more meaningful, so it becomes more authentic and more complex. Thus, authentic assessment and other non-standard assessment methods, such as rubrics, need to be designed to capture meaningful learning experiences of students brought about by these new technologies.

Unfortunately, not too many teachers have acquired the competencies to conduct authentic assessments for student learning, using learning portfolios and rubrics for performance evaluation. The need for such learning assessment competency becomes even more urgent as educators move away from the behaviourist and objectivist perspective of learning to that of a more constructivist view.

Using the type of indicators used by countries as basis for categorising them based on the extent and impact of ICT and using the input-process-output model, the results would be:

Country Level* Quantative Indicator Qualitative Indicator
Australia 3 X X
CIS and Balkan countries 2 X  
Canada 3 X X
Europe (Denmark, Sweden, Finland, Norway, UK) 3 X X
India 2 X  
Indonesia 1 X  
Japan 2 X  
The Republic of Korea 3 X X
Malaysia 2 X  
New Zealand 2 X X
The Philippines 1 X  
Slovenia 1 X  
South Africa 3 X X
Thailand 2 X  
The UK 3 X X
The USA 3 X X
Uzbekistan 2 X  
Vietnam 1 X  

*level 1 – includes input indicators only
  level 2 – includes input and process indicators
  level 3 – includes input, process and output indicators