Follow Us:

Section One: Some Basic Approaches and Guidelines to Developing Indicators

Defining indicators

Indicators are signs or manifestations of something. In evaluation studies, indicators are used as evidence or signs by which we can assess or evaluate materials, methods, an intervention, a programme or a project. Indicators are measuring devices. They define concepts in terms of the measurements and data it is possible to collect and analyse. They define what data to collect and at what time intervals.

 

Steps in developing indicators

To develop good indicators, we need first to have a clear vision of what is to be achieved and measured. The first requirement for a systematic development of indicators for ICT is to identify the results, objectives, outputs and key concepts of the ICT project, as part of the evaluation system plan. The basic approach involves four steps as follows:

  • Identify what is to be measured
  • Develop trial measures
  • Assess each trial indicator, using agreed-on criteria
  • Select the best indicators for a specific project

 

The process of developing indicators involves a combination of activities, such as brainstorming, multi-stakeholder discussions and being clear about definitions, criteria, goals and priorities. More focused and cost-effective monitoring and evaluation of ICT use in education is expected. Indicators are based on assumptions about what is relevant - they are expressions of value or of what is desirable. Thus, it is important that researchers and evaluators involve stakeholders in developing the indicators to ensure that the data collected will respond to their information needs.

The Actions Required in Each Step

Step

Action

Step 1: Identify all concepts to be measured, especially project objectives and outputs

• Review all concepts, objectives, results, and output statements given in the programme or project work plan to clarify them and get agreement

• Be clear about what type of change is implied (a situation, state, condition, attitude, behaviour). What role can ICT perform in reforming the educational system? Will the state of schools change from no access to more connectivity? Will teaching methods change/improve as a result of ICT use? Will a teacher’s attitude become more favourable to the use of ICT in teaching? Will learning become more participatory, interactive, and reflective?

• Clarify whether the outcome sought is an absolute change, a relative change, or no change. Will ICT be introduced in all subjects or just a few major ones? Will ICT be introduced in all schools around the country or in just selected pilot schools? Only in primary or secondary or all levels of education? Is ICT used as a subject or used in subject teaching?

• Specify where and when the change is expected (what target group, what location, and in what time frame) (this identifies the appropriate unit of analysis). Will the results only occur among teachers or students or both? Is change expected to occur all over the country or just a few geographic regions? Only in primary or secondary schools? Will the change begin to show only after the project is completed or are changes occurring during the different phases of project implementation which need to be measured?

• Determine the relationship between project activities and their outputs or objectives (are these outputs or objectives direct or indirect?)

Step 2: Develop a list of possible (trial) indicators

• Think of possible alternative indicators for each concept, objective, and output, without being too restrictive. Come up with a shopping list first which can be prioritized later. One attribute or object can have three to five indicators. For example, if the object is to assess teacher development, the following indicators can be considered: number of hours teachers have been trained in ICT; whether training is basic or advanced; teacher-computer ratio; use of Internet and e-mail by teachers; use of ICT in teaching science; or improvement in teaching methods; and impact of ICT use on teacher advancement.

• Conduct internal brainstorming sessions to develop indicators that emanate from internal project/programme needs and ideas/thinking of staff and those involved

• Consult stakeholders and other experts to validate or/and expand the set of indicators developed internally from the perspectives of partners and other groups in order to ensure that their needs and requirements are being met by the indicators.

• Try to borrow from other projects and studies. If there are indicators which have already been tested and found useful and workable and have universal application, then these indicators can also be included in the list.

Step 3: Assess each trial indicator against criteria

• Establish an agreed set of criteria for indicators (see Table 2)
• Use a scoring scale (1–5) to determine the usefulness of each trial indicator (but be flexible and use your own judgment) based on each criteria

• Use a scoring scale to also assess how easy/difficult it is to obtain data for the indicator

Step 4: Select the best indicators for this project

• Consider each indicator on its merits against the criteria

• Consider the mix of indicators to construct a robust set that is consistent and complementary in terms of data-collection methods and time frames. The indicators chosen should support each other, not cancel each other out.

• Avoid having too many indicators (it may indicate that the objectives and outputs are not clearly defined). The ideal number will depend on the complexity of the object being assessed. For example, if the object is ICT access and connectivity, three to four indicators dealing with presence of computers in schools; student-computer ratio; schools with Internet connection will suffice. But if the object being assessed is improvement in teaching and learning, then more qualitative indicators may be needed. The World Bank recommends not having more than three indicators for each object or element.

• Be prepared to update your indicators — the best indicators may change as projects develop (one common change occurs after using input indicators at first and then realizing that output indicators are what is needed). For example, indicators dealing with access and connectivity may be important during the initial stage of ICT development but as the project progresses, indicators dealing with teaching and learning outcomes may become equally useful.

Criteria for Assessing Indicators
In developing indicators, criteria are needed against which their value and importance can be measured. Criteria are needed in order to guard against biases and differences in opinions and attitudes of different evaluators and developers of indicators.

The following criteria can be applied in assessing potential indicators. These same criteria were used in the Consultative Workshop for Developing Performance Indicators for ICT in Education jointly sponsored by UNESCO ASIA and Pacific Regional Bureau for Education and the Southeast Asian Ministers of Education Organization (SEAMEO INNOTECH) conducted last 28 – 30 August 2002 in Quezon City, Philippines.

Criterion

Description

 Direct measure
  • Indicator is intuitively understood (high face validity).  Almost everyone involved in the assessment exercise looks unanimously at some indicators as easily acceptable and recognizable as useful
  • Indicator is a direct measurement, rather than a proxy that depends on assumptions for its validity. No other indicators can substitute for it.
  • Indicator is supported by a body of research. Many research findings support its validity, relevance and usefulness

 Objective
  • Indicator is unambiguous about what is being measured. It only produces one absolute result or outcome
  • Different people will collect comparable data based on the indicator
  • Definition remains stable over time, so change can be measured
  • Indicator is unidimensional (measures only one thing at a time)
  • Indicator can be quantitative or qualitative, as long as it is clearly and consistently defined and interpreted. A quantitative indicator measures changes in numbers while qualitative indicator measures changes in opinion and behaviour. For example, a quantitative indicator may measure the number of hours teachers have been trained in ICT, while a qualitative indicator looks at how the ICT training has changed knowledge, attitude and skills of teachers. Qualitative indicators are useful only to the extent that they are specifically detailed, standardized and made comparable, as, for example, a check-off list of attributes (hence, non-quantitative) to be measured by recording absence or presence of the repetitive attributes.

Valid
  • Is a particular measure or set of measures a valid indicator of the concept that it is supposed to represent? For example, is a count of number of computers in school a valid measure for access? To answer this question, one needs to assess the degree of correlation between the measure and the concept. Are they very closely related or remotely related? “Total man-hours of hands-on use of computers” is more closely related to the concept of access than a count of number of computers in school. Possession of computers by the school does not necessarily mean that pupils have access to them as they can be locked up and made accessible only during teacher-supervised session.

 Adequate
  • Either by itself or with a minimal companion set of indicators, the indicator provides reasonable confidence that it accurately measures the attribute
  • Object is to have as few indicators as possible per attribute (should be three or fewer) — more is not necessarily better
  • Number of indicators will depend on the complexity of the object, or what is being measured

 Quantitative
  • Quantitative indicators are more objective than qualitative ones, but to arrive at more meaningful associations and relationships in results, qualitative indicators should balance the quantitative ones.
  • Qualitative indicators should be adequately specified to be objective and consistent

 Disaggregated
  • The more disaggregated the indicator, the more easily data can be manipulated to answer questions not anticipated at the outset. Changes happen also because of many extraneous factors. In order to explain more fully why some unanticipated changes occur, some factors need to be isolated and included in the analysis. These could include age, gender, location, educational attainment, occupation, etc. which may have strong influences in the use of ICT by teachers, for example.

 Practical
  • Data can be collected at reasonable cost, given their utility
  • Data are available and can be collected at suitable time intervals. For example, some data are accessible and usable only within a short period of time e.g., number of computers in school, while others may require inordinate amount of time from conception to operation, e.g., learning achievement test.
  • Data can be readily collected in various projects for comparison
  • Consider the methods of collection as well as the collection method shapes the data collected

Reliable
  • Indicator is reliable within the context of the evaluation purpose and resources
  • Data-collection process is consistent across different time and space scales, using comparable methods and sampling procedures Indicator is based on representative data

Sustainable and replicable
  • These criteria should be applied in relation to the intention of whether it is to be conducted only once or repeatedly at a regular intervals, i.e., annually. The first time will be costly, as it includes investment cost for developing the methods, instruments, equipment, training, organization, etc. while the cost will decline for the following rounds. But there will be operational costs in terms of staff, logistics, supplies, materials, etc. So, is the funding one time or will there be a regular budget to sustain the system?

Methods for Collecting Indicators

Below are some methods for collecting indicators that various countries have adopted in their data collection activities.

  1. The most popular method is the use of a survey questionnaire farmed out to a representative sample of schools heads, teachers, and students. Use of survey questionnaires can be made on a regular basis (e.g., every year, every two years) or on a one-shot basis. ICT surveys done more regularly do have an advantage over those done only once. The former can track the behaviour of the indicators over time. Improvements in inputs, processes and outputs, if any, can be observed over time. Weaknesses in ICT implementation can be remedied in due time; new strategies in ICT implementation can be carried out, thus increasing the probability of success of ICT programmes and projects. The one-shot survey is more appropriate for summative assessment in which we are more interested in finding out, for instance, the longer-term impact of ICT use. Collection of data can be undertaken through mediated, i.e., someone conducts interviews, and unmediated (where respondent fills out the survey and sends it in), It is important to differentiate between data collection methods, especially if data is meant to be utilized to draw conclusions through comparison. The use of handhelds/PDA in data collection is also now a growing trend.
  2. Where the telephone system in a particular country is well developed, such that most of the homes and schools have access to a telephone and the cost of placing long distance calls is reasonably low, telephone interviews are a more cost-efficient method of collecting data on indicators. For example, EURYDICE, Basic Indicators on the Incorporation of ICT into European Education Systems Project, conducted Eurobarometer surveys over the telephone in 2001, covering a representative sample of school heads and teachers in each EU country. The questions put to the school heads were mainly concerned with their school ICT equipment and facilities, while those asked of the teachers included the use of computers and the Internet with their pupils (time spent using them, frequency of their use, reasons for not using them, etc) and how ICT had changed the way they taught.
  3. Where there is a saturation of Internet connectivity in the country, that is, most of the homes, schools and offices are connected to the Internet, use of questionnaires posted on the websites of the institution/agency conducting the surveys can be an efficient method of collecting data on indicators. Communication by e-mails between the respondent schools, school heads, teachers and even students will facilitate data gathering. Data entered into the web-based questionnaire can be automatically uploaded to the institution/agency’s computer server located thousands of miles away. However, there is a potentially huge selection bias here, as only the most technically competent teachers may respond via the Internet.
  4. A combination of various assessment methods can also be considered. In Australia, the national monitoring of ICT skills and knowledge of students in Year 6 (aged 10-12 years) and Year 10 (aged 14-16 years) will be done by means of three-yearly sample assessments, commencing in 2005. Student outcomes and performance in different years in a particular domain is the focus of measurement. The assessment methods will include a mix of tests on paper and on line and performance assessment tasks with teacher assessment. The study by Singapore to measure the effective integration of ICT in schools indicated that to gather accounts by various groups and individuals in the learning environment, both qualitative and quantitative methods were drawn upon, such as observations of IT and non-IT based lessons, face-to-face interviews with principals and IT-coordinators, group interviews with students and teachers, questionnaires for teachers and students and samples of students' work.
  5. Use of ready-made Web-based tools such as EnGuage, a new Web-based framework developed by NCREL in the USA which serves as a tool set designed to help schools and school districts use technology effectively for learning, teaching and managing and will help answer the following issues: What value does technology bring to the schools? How can our schools ensure a return on these investments? And why does technology work in some schools and not in others? Another example is the School Technology and Readiness (STAR) Chart which provides another web-based self-evaluation tool, providing schools with the information they need to better integrate technology. With STAR Chart online, multiple-choice questionnaires can be completed that will provide not only a measurement of any particular school’s technology and readiness, but also serve as a benchmark against which every school can assess and track its own progress.

The method of data collection for ICT indicators will then vary from country to country depending on the spread of telecommunication use such as the telephone and the Internet. It will also depend on whether the existing monitoring and evaluation systems in the Ministries of Education can still accommodate and/or integrate into their existing data base system additional data on the use and impact of ICT in the schools, including additional variables based on ICT indicators. It will also depend on whether the staff can still carry on additional tasks of monitoring and evaluating the impact of ICT use in education. This also complicates drawing conclusions from comparing data collected in different manners in different countries.