Source: READ Annual Report (2013, p. 39) (NLSA = National Large Scale Assessment, ILSA = International Large Scale Assessment)
Abstract: In the field of comparative education, there is a growing amount of research on how education policy agendas are formed at the transnational level and how these influence policy making in individual countries. This article focuses on learning outcome assessment that has gained prominence as a transnational agenda, constructed within a network of actors sharing a common belief in the necessity of measuring learning outcomes as a key precondition for improving education quality. This agenda is clearly guiding the activities of the READ program, funded by the Russian Government and implemented by the World Bank in eight developing countries during 2008-2013. The case of Ethiopia as one of the READ countries is analysed on the basis of the READ reports and the ESDP and GEQIP documents. The analysis shows both noticeable influences that can be traced back to the transnational agenda and some policy positions that reflect national prioritization. The discussion section raises the question how standardised measurement of learning outcomes – which is becoming a global imperative through the EFA process – may become a straitjacket to teaching and learning.
Keywords: assessment, learning outcome, READ program, Ethiopia
This article is an outgrowth from two strands of the author´s research: previous and ongoing analyses on assessment of learning outcomes as a transnational agenda (Piattoeva & Takala, forthcoming; http://www.uta.fi/edu/en/research/projects/evalpolitics.html), and a longstanding interest in educational development in Ethiopia (for Ethiopia as part of comparative analyses see Takala, 1998 & Takala, 2009, and as the focus of analysis Martin, Oksanen & Takala, 2000). The latter perspective has recently been deepened by his role as supervisor of dissertations of Ethiopian PhD students both at the University of Tampere (a finalised dissertation is Adamu, 2014) and at the College of Education, Addis Ababa University.
In the field of comparative education, there is a vast and growing amount of research on how education policy agendas are formed at the transnational level and how these influence policy making in individual countries (e.g., Chabbott, 2002; Mundy, 2007; Steiner-Khamsi & Waldow, 2012). In this literature, one can distinguish between two perspectives. One focuses on how development agencies can influence policy decisions through framing policy discussions and related technical advice, and through prioritizations that materialize in projects and sector programs (e.g., King, 2004; Samoff, 1999). This perspective can be further refined by analysing how local technocratic elites of developing countries may act in alliance with the external agencies (Gould & Ojanen, 2003). The other perspective is more open to also identifying how national policies may modify or reject parts of the transnational agenda (e.g., Baker & LeTendre, 2005; Takala, 1998; Takala, 2008).
The globally defined Education-for-All by 2015 target is an example of strong external influence on national policy-making, obligating countries to build their education policies and sector plans around this target. The EFA agenda has contributed to a policy consensus between the donor agencies and developing country governments: enrollment in primary education is to be increased at a maximum possible pace, special attention is to be given to gender equality and to other dimensions of socio-cultural disadvantage, and the quality of education is to be improved. For countries which have participated in the Education for All Fast-Track Initiative/Global Partnership for Education, external influence has become tangible in the “indicative framework,” which defines target parameters for resource allocation and cost-efficiency in the education sector (World Bank 2004). But even in this case, the actual importance of such prescriptions is found to be variable in different countries (Cambridge Education, 2010).
For the purpose of this article, it is important to note that any generalizing analysis on the influence of the World Bank and bilateral donor agencies on national education policies is misguided. First, this influence is obviously variable across countries and related to their degree of dependency on external funding. It is also variable in different policy issues: for example, constructing and equipping science laboratories in schools and universities is closely tied to external funding, whereas the content of the primary school curriculum and the language of instruction are more immune to external interventions.
An intriguing finding related to the topic of this article has emerged from previous studies conducted by the author together with colleagues. In interviews with representatives of the Ministry of Education and donor agencies in Mozambique, one question concerned the origin of the Sector-Wide Approach (SWAp) in the education sector. The Mozambican respondents maintained that because of their accumulated negative experience with the project mode of external assistance, they devised and decided upon the SWAp and then “sold it to the donors.” By contrast, on the donor side the interviewees definitely saw the same process as having been initiated, conditioned and steered by the donors (Takala & Marope, 2003). A subsequent study of the preparation process of the first Education Sector Development Program in Ethiopia found through a questionnaire survey and interviews that in the opinion of both the Ethiopian participants in this process and the donor representatives involved, the adoption of the SWAP was genuinely initiated and led by the Ethiopian Ministry of
Education (Martin, Oksanen & Takala, 2000). This corresponds to the view of Mozambican informants on the origin of SWAP in the education sector in their country, but in Ethiopia this interpretation of events was not contested by an alternative view, as was the case in Mozambique.
This article looks at the policy of educational assessment constructed as a transnational agenda and how this agenda is manifested in the activities of the READ (Russia Education Aid for Development) program. The case of Ethiopia as one of the partners of this program is analysed on the basis of the READ reports, on the one hand, and the ESDP and GEQIP documents on the other.
The transnational agenda on assessment of learning outcomes is constructed within a network of actors that does not have clear boundaries or an identifiable center of power or thought from where the agenda emanates. The network shares a common belief in the necessity of measuring learning outcomes as a key precondition for improving education quality worldwide (e.g., UNESCO Institute of Statistics, 2013; World Bank, 2013; Center for Global Development, 2013). Actors within the network consist of a wide range of multilateral organizations (e.g., the World Bank, UNESCO, OECD), regional networks (e.g., SACMEQ), private professional bodies (e.g., Brookings Institute, Educational Testing Services), academic experts and consultants.
Improvement of education quality can of course be justified by reference to any objectives set for the education system. In contrast, the typical justification expressed for the transnational assessment agenda is notably narrow and even monotonous: reference is in numerous sources made to the finding that cognitive learning achievement – rather than years of schooling per se – has a positive impact on economic growth (Hanushek & Woessmann, 2007; World Bank, 2011; Center for Global Development, 2013). Some of the references contain remarkably simplistic statements such as “a 10 % increase in the share of students reaching basic literacy translates into a 0.3 percentage higher annual growth rate for that country” (Gove & Cvelich, 2011, p.1). The corollary then becomes that a well-functioning system of assessing learning outcomes is “a key driver of economic growth and poverty reduction” (READ, 2010, p. 36), or in a sloganistic manner, “measuring for success”, which was the title given to the READ final conference in May 2014.
READ is a program funded during 2008-13 by the Russian Government and implemented by the World Bank through a Trust Fund agreement. The idea of READ was born in the context of the Education for All Fast-Track Initiative, FTI (later renamed Global Partnership for Education, GPE), which Russia joined in 2006. READ has operated in eight countries: Angola, Armenia, Ethiopia, the Kyrgyz Republic, Mozambique, Tajikistan, Vietnam and Zambia. The selection of the READ countries was determined on the one hand by their being beneficiaries of support from the Education for All Fast-Track Initiative, and on the other by either being former members of the Soviet Union or having ideological and financial link with the then USSR.
A significant feature in the READ set-up is that the beneficiary countries are receivers of external assistance to their educational development, from the Global Partnership of Education, World Bank projects and bilateral donors. In countries that are dependent on development assistance to education, the tendency towards measuring learning outcomes is also related to increased accountability demands on the donor organizations whose primary concern in the education sector was previously the growth of enrolment figures. For example, basic education assessment has become a component in the majority of World Bank financed education projects and its frequency has increased over time (Liberman & Clarke, 2012; Lockheed, 2013).
The aim of READ is to “help low-income countries to improve their student learning outcomes through the design, implementation and use of robust systems for student assessment” (READ, 2011, p.4). In addition, a large proportion of the READ funds goes to the World Bank for the development of “global products” in educational assessment. During its implementation, READ has become closely linked with SABER (System Analysis for Better Education Results), designed for the World Bank and with background work outsourced to the American Institutes for Research (READ, 2012). READ support to SABER started in 2010 and the latter has subsequently underpinned the work completed under the READ programme. In a remarkable change of terminology, the early “READ diagnoses” and other READ activities have been renamed as SABER activities (READ, 2011; 2012; 2013). It is also known that, in the African READ countries, READ is perceived as a World Bank program – quite understandably, as Russian experts have not been visible in its implementation in these countries.
The SABER student assessment framework stipulates a uniform ideal for all countries in their development of the different components of assessment systems (READ, 2011). The components included in the framework are, first, the results of both national and international large-scale assessments, which are signals that call for attempts to improve the quality of overall learning and give information on weak areas in learning outcomes that can be used in policy-decisions, such as revising curriculum content. Public high-stakes examinations are considered to be powerful means to guide the content of what happens in classrooms, and tools of classroom assessment are seen as necessary to improve pupils’ performance both in examinations and large-scale assessments.
The SABER diagnoses create a picture where countries are placed “at different points on the continuum of assessment systems,” where the terms “latent”, “emerging”, “established” “advanced” are used to mean “absence of, or deviation from, attribute”, “on way to meeting minimum standard”, “acceptable minimum standard” and “best practice”, respectively (Clarke, 2012).
SABER is a clear example of framing an education policy issue as something where “international best practice” can be identified, agreed upon and distributed through programs or projects such as READ. By 2013, baseline studies following the SABER model had been conducted in all READ beneficiary countries. The numbers indicating the extent of SABER’s global outreach are impressive: by the end of 2013, training in the use of the SABER Student Assessment Tool had been provided for over 2000 professionals and the tool had been used in more than 50 countries (READ, 2014).
The process of defining READ priorities at the country level involves consultation with national level policy makers and experts, but it is initiated, framed and finalized by World Bank staff. It begins with a baseline diagnosis on the state of the country’s assessment system, which then leads to devising actions for improvement. Against the baseline further analysis of change achieved is undertaken (World Bank, 2013; Clarke, 2012). Within the eight READ countries, the emphasis given to the various components is notably variable and depends on the preferences of the national policy makers, which in turn may be influenced, though not determined, by the SABER diagnoses (READ, 2014). It is clear that on the basis of document analysis alone it would not be possible to definitely distinguish between “home- grown” and “externally influenced” priorities. As mentioned earlier, it is also doubtful whether such a conclusion could be derived even from interview data.
The education policy priorities of the Government of Ethiopia are expressed in the consecutive Education Sector Development Programs. ESDP III, being implemented from 2005 onwards, as a program preceded any possible influence from READ, whereas within the framework of ESDP IV, covering the years 2010-2015, external support is provided by a group of agencies through the General Education Quality Improvement Program (GEQIP) (Phase I 2009-2013, Phase II 2014-2018).
In the ESDP III document, the overall goal of education sector development was partly defined in terms of economic growth and productivity, the target being to “transform Ethiopia into a middle-income country in 20-30 years”. Emphasis was also given to sustainable rural development and promotion of democracy and related civic and ethical education (Ministry of Education 2005). Sample-based assessments of learning achievements were to be carried out at Grade 4 and Grade 8. The comprehensive view of desired learning outcomes requires that assessment not only be confined to the basic cognitive competencies of literacy and mathematics, but also extend to a broader area of knowledge and skills (Ministry of Education, 2005). In addition, the document mentions strengthening of school inspection as one means to improve the quality of education (Ministry of Education, 2005).
In contrast with the ESDP, the GEQIP Phase I documents imply justifies the program by reference to the need to build human capital for economic growth. GEQIP was prepared through intense negotiations between the Ethiopian Government and the education sector donors and reflects their shared concern over the quality of primary (and secondary) education under conditions of rapid enrolment. In the area of assessment, a key objective was to monitor the ongoing revision of the curriculum (World Bank, 2008).
The results of the country-level diagnosis on the basis of the SABER framework are depicted in the following diagram:
Table 1. SABER diagnosis of the baseline situation of educational assessment
in Ethiopia in 2009 and evaluation of change 2009-2013
Source: READ Annual Report (2013, p. 39) (NLSA
= National Large Scale Assessment, ILSA = International Large Scale Assessment)
According to this diagnosis, the situation at the outset of the READ programme was quite variable between the different components, ranging from the low scores of “latent” for classroom assessment and international assessment studies, barely at the “emerging” level for national learning assessment, and a higher “emerging” level for examinations. The low score given for national assessment is noteworthy in light of the fact that such assessment at grade levels 4 and 8 was initiated in Ethiopia already in 2000, with subsequent studies carried out in 2004 and 2007 (Greaney & Kellaghan, 2008).
The diagram presented above depicts the change observed during 2009-2013 as having been significant in the national assessment component and more modest in examinations. Of these, the former still remained at the “emerging” stage, whereas examinations were deemed to have reached the borderline between “emerging” and established”. Participation in international assessments and development of classroom assessment were not activities funded by READ in Ethiopia, but “improving the teaching and evaluation methods of teachers” (classroom assessment) is among the objectives of the GEQIP program (GEQIP, 2008).
The activities supported from the READ funds have been complemented by the multi-donor funded GEQIP and support from USAID and UNICEF. Hence, external influences are aligned and intertwined a way that prevents specific judgments on “Program organization X had an influence on activity Y.” These activities include the following (READ 2011; READ 2013; READ 2014)
Notably, school inspection is not a part of the SABER framework, and the allocation of READ funds to this area is a signal of the influence of Ethiopian priorities, as expressed already in ESDP III. Such prioritization reflects a top-down approach focusing on provision and maintenance of a standard set of inputs that are expected to have a positive effect on the quality of education, rather than a pedagogical approach to supervision (READ, 2010).
The READ Annual Report for 2010 specifically mentions that there is in Ethiopia “one individual with international standard technical skills in large-scale assessment” (READ, 2011, p.18).
In the ESDP IV document the target year for transforming Ethiopia into a middle-income country is set at 2025, and from this follows an emphasis on the teaching of science and mathematics from the primary level upwards (Ministry of Education, 2010). In comparison with ESDP III, where the economic rationale was counterbalanced by civic and ethical education, ESDP IV has in this respect become more aligned with the transnational discourse on the quality of education. The same trend is evident in the section on the curriculum reform where special mention is made of the training abroad received by curriculum developers and teacher educators in science and mathematics (Ministry of Education, 2010).
According to ESDP IV, the objective of improving student learning is to be achieved “through a consistent focus on the enhancement of the teaching/learning process and the transformation of the school into a motivational and child-friendly learning environment” and through “reinforcement and better coordination of key quality inputs and processes” (Ministry of Education, 2010, p.6). Program objectives related to assessment include aligning assessment and examinations with the ongoing curriculum reform (Ministry of Education, 2010). The document further notes that in spite of improvements during ESDP III in the level of teacher qualifications and material inputs, the average level of learning shown in the national assessments has been decreasing. To reverse this trend during ESDP IV, the target is set in precise numbers: “90 % of the students at all grade levels will score at least 50 % in examinations and assessments of every subject” (Ministry of Education, 2010).
The GEQIP Phase II document gives credit to Ethiopia as one of the few countries in Africa that have institutionalized a system of national assessment (GEQIP, 2012). If we compare the ratings given in 2013 for the individual READ countries in the domain of national assessments, only Zambia is evaluated as being at a higher level than Ethiopia – in addition, Armenia is set at par with Ethiopia (READ, 2013). A critical conclusion from previous GEQIP experience is that the School Grants made available as part of the program have mainly been used “for upgrading school infrastructure, and these are not likely to make measurable improvement in learning outcomes unless these grants are used strategically” (World Bank, 2012). As regards future development of assessment, examinations and school inspection, the respective activities initiated during Phase II are to be continued and developed further (World Bank, 2012).
The analysis of the key education policy documents that present the priorities of the Government of Ethiopia and the consensus achieved with the donor community shows both sizable influences that can be traced back to the transnational agenda of educational assessment and some policy positions that reflect national prioritization. Identification of influences that could be attributed specifically to the READ program is circumscribed by the fact that the operations of the various external actors are aligned and intertwined, preventing simple judgments on “program/organization X had an influence on activity Y”.
In the recent intense discussions on the post-2015 EFA goal, a growing number of actors have become concerned with educational assessment (see King & Palmer, 2012). The actors share an interest in keeping education high on the global political agenda, and in this situation, researchers may be tempted to mute their critical voices. The explicit criticism from researchers has emphasized that the transnational agenda of assessment is fundamentally flawed because it is “seeking to achieve what is measurable, without asking the fundamental question of whether what is measurable is worthwhile, valuable and meaningful” (Sayed, 2011). One can extend this question to ask whether the values that basic education seeks to promote are derived from the goal of economic growth or from other value perspectives, such as democracy or (other) ethical principles.
The criticisms notwithstanding, at the time of this writing, has become clear that the new EFA goal for 2030 is being formulated in a manner that will lend itself to standardized measurement of cognitive learning achievement. Indicators for learning outcomes have been proposed by the Global Partnership for Education (formerly EFA Fast-Track Initiative) at a general level: “The share of students who, by the end of two grades of primary schooling, demonstrate that they can read and understand the meaning of a grade appropriate text” and “The proportion of students who, by the end of the primary or basic education cycle, are able and demonstrate understanding of an appropriate text, as defined in the national curriculum or agreed by national education experts” (Global Partnership, 2012, p.128-9). The most recent Education for All Global Monitoring Report (UNESCO, 2015) is less specific on this issue, but recommends development of a “common metric linked to national learning assessments”. Subsequently, the Framework for Action prepared for the 2015 World Education Forum states that “Learning outcomes must be well-defined in cognitive and non-cognitive domains, and continually assessed as an integral part of the teaching and learning process” (World Education Forum, 2015, p.5). Formulation of the target in the ESDP IV document such as “X % of students at a given grade will achieve at least a score of Y %” is already well suited to this purpose. As pointed out by Barrett (2011), this kind of simple information can be expected to easily catch political attention, but its value for educational development on the ground would be very limited and indirect at best.
The new EFA goal will provide a further boost to the weight of the transnational assessment activities in the political EFA discourse, as well as in the country level efforts towards EFA. The consequences for education policy-making will then be much more binding than anything that READ or a similar program can possibly have. It remains to be seen to what extent there will be space for national priorities within the globally prescribed setting and how, for example Ethiopia, might use this space in the years to come.