Reflections on Theory, Method, and Practice in Comparative and International Education[1]

 Steven J. Klees
University of Maryland
Revised, April 2008

I have been very fortunate in my professional life in many ways.  One way is that I get paid, in part, to reflect on some of the significant issues of our day.  Over the years, my views have evolved, shaped by what I have seen, discussed, experienced, read, and studied.  I wish to take the opportunity afforded by this Presidential Address to reflect on what I believe and how I came to believe it.  I hold strong views about our field of comparative and international education, views that often clash with the dominant modes of theory, method, and practice.  In this address, I wish to explain what I see as fundamental flaws in some of these dominant modes of thinking and explore what some of the alternatives to those views are.  I will begin by saying a few words about how I see the field of comparative and international education and then discuss theory, method, and practice in turn.

Comparative and International Education

            Over the past sixty years, we have seen enormous development of the field of comparative and international education (CIE), from one that was a focal point for a relatively small number of academics and practitioners to one that extends to large numbers of people engaged in grassroots organizations; nongovernmental organizations (NGOs); field projects; universities; and local, national, and international agencies. In the 1950s, CIE was a small field, relatively closed and cloistered, composed of a range of historical and cultural studies to ‘traveler’s tales.’ The 1960s were a watershed, epitomized by Harold Noah and Max Eckstein’s (1966) book, The Science of Comparative Education.  During the 1960s, the borders of comparative education came down and along with them much of the introspective discussion that had tried, rather unsuccessfully, to come up with definitions to focus and limit the field.  Instead, the borders were opened, especially to all the social sciences.  Moreover, the 1960s and 1970s saw ferment within and across the social sciences about the validity of the modernization narrative and the associated debates about the meaning and practice of development.  The resulting paradigmatic controversies also became an integral part of the CIE field. 

This openness of CIE to the theories, methods, practices, debates, and controversies from many fields continues to form the most vital part of our field today.  And it is this integral and necessary permeability that forms both a great opportunity and a great challenge to CIE.  Harold Noah once argued at a CIES meeting that the comparative advantage of comparative educators was our focus on education.  I respectfully disagreed at the time and still do.  The principal comparative advantage of comparative education is that the field is literally constituted by border crossings, and comparative educators, by necessity, roam far beyond education.[2]  In my view, no other disciplinary or professional field has such a broad, interconnected vantage point from which to view the dilemmas of our time.  Education is the anchor that focuses us, but we know – and have to know – significantly more in some ways, for example, than economists or other social scientists, even about their own fields, about the research methodologies used, about policy and practice.  For example, a comparative educator today needs to understand economics’ human capital theory, but also its complement in sociology, modernization theory, as well as the critique of both (Arnove and Torres 2007; Ginsburg and Price-Rom 2008).

            This has yielded a serious dilemma to those of us involved in university education.  I feel no other field has to develop the breadth of knowledge that CIE requires for a student to become an educated, reflective practitioner and/or researcher.[3]  Nowhere is this challenge more apparent than in deciding what core knowledge should be in CIE graduate programs.  I have spent many years at Florida State University and the University of Maryland engaged in explicit discussions with colleagues of what this core knowledge should include.  The most recent effort, a few years ago, yielded this list:  students should know something about anthropology, economics, political science, and sociology, as well as debates within and across these fields; development theory and practice; globalization; postmodernism and the other posts-; feminism/gender and development; race and ethnicity/multiculturalism/diversity/identity; current education and development issues; the nature and history of the CIE field; critical pedagogy; policy studies, analysis, and planning; environmental issues; international institutions; the role of NGOs, civil society, and social movements; participatory perspectives; community development; the history and philosophy of education; the arts and comparative literature; psychology.

This is obviously a tall order.  Of course, how much a student learns about any of these areas is variable and depends on specific decisions we make in our core required courses, of which presently the University of Maryland has three.  We do try to cover something about each topic above (except for the last three categories).  There are at least two things that give coherence to our teaching this array of topics.  One is that all issues are relevant to the central dilemma of our time (‘What to do about poverty, inequality, and development?’) and for our field (‘What is education’s role in all this?’).  A second is that a central tenet of my view of the field is that all knowledge is seriously contested.  Therefore, the debates surrounding each topic mentioned above become focal points in the curriculum..  Since the nature of the debates on each topic is similar, the core courses have a coherence that can turn out students with a remarkable ability to integrate a lot of diverse issues.

Reflections on Theory

I did not knowingly become a part of this CIE field until 1980 when I took a job in a CIE program at Florida State University.  Prior to that I had done my graduate work at Stanford University in economics and public policy, with a concentration in the economics of education, and had taught for seven years, partly in an educational policy program at Cornell and partly in a public policy program in Brazil.  During that time, I saw my main professional identity as an economist of education concerned with public policy.  However, I felt very much at home in CIE from my first exposure to it at FSU and in CIES, when I attended my first meeting in March 1981, held at FSU.[4]

One reason I felt so at home in CIE was that economics had a prominent place in it at that time.  But even more so, I felt at home because, from the 1970s onward, economics, and associated human capital and modernization perspectives were hotly contested within CIE.  And from my earliest work in this profession, as a graduate student, to my work today, teaching, writing, and engaged in practice, this contestation has remained central to my understanding of the dilemmas we face in theory and practice.  In this section, I discuss some of the key issues and occurrences that have influenced my thinking about theory, especially economic theory.

Early Misgivings

            The bulk of my graduate education focused on neoclassical economic theory – what I see as the fictional account of the efficiency and equity of free markets.  But this was the Vietnam War era, the late sixties to the early seventies, which threw even academia into social upheaval.  Thus, from the beginning I saw glimpses of competing stories, competing social explanations.  The economics faculty had John Gurley, a neo-Marxist economist, and the education faculty had Martin Carnoy and Henry Levin, two strong critics of neoclassical economics.  Many of the economics students were dissenters from the orthodoxy, and, at one point, we even got together and raised enough money to hire Joan Robinson, a famous critic of neoclassical economics, as a visiting professor for a semester.

These critics exposed me to alternative theories.  But equally influential were my early misgivings about the neoclassical story.  It became clear in my graduate school courses that neoclassical economic theory had fundamental inconsistencies that rendered it problematic at best, absurd at worst.  Chief among these inconsistencies is that in order for a market system to be efficient, you need a number of assumptions to hold (mainly, profit and utility maximization, perfect market competition, and perfect knowledge).  Yet these assumptions can never hold completely in practice.  That might be OK if coming close to these assumptions was good enough to get you close to efficiency.  However, that is not the case.  About three years into my graduate education, one of my classes spent a day on ‘Second Best Theory,’ which asks, ‘If we don’t live in the first-best world of the assumptions of perfect competition, but have, let’s say, one imperfection in an otherwise perfect world (e.g., one monopoly), what are the results?’ (Rakowski 1980).   It turns out that Second Best Theory says the results of one imperfection ripples through the market system, so the outcome is not efficient nor close to efficient.[5]  This means that Adam Smith’s dictum that the invisible hand acts in the social interest is totally false in a real world economy rife with multiple deviations from free market theory. [6] I remember one professor saying in class that we have ‘workable competition’ in the real world, but when asked what that was and how it related to the numerous benefits of perfect competition that they had spent years teaching us, he was at a loss, as are all neoclassical economists..

            From early on, it has seemed to me that the concept of efficiency, which is thoroughly embedded in the concept of perfect competition, was empty in the real world and, therefore, calls for improving educational or economic efficiency were devoid of meaning.[7]  The great feat of neoclassical economics has been to convince people that there is a vantage point separable from distributional concerns, where one could look to see if the system as a whole is better off, defined as efficiency, where the decisions to produce an array of goods and services could be made in the interests of all irrespective of how little one had.  However, if prices are not defined according to the exact dictates of free market theory, then private profitability or social rate of return analysis tell us nothing about the comparative social advantage of producing, say, more yachts instead of rice and beans, or more primary education instead of higher education.  Without the justification provided by neoclassical economic theory, all markets become just a convenience but have no legitimacy as acting in the social interest (Klees 1993).  Given the great inequalities it yields, I believe that someday our market system of wage labor will be considered as illegitimate as slavery is now considered.  The fact that the market pays someone $1.50 for a day of backbreaking labor while others get millions of dollars for their white collar labor is akin to a form of slavery for which no one takes responsibility and which is disguised by the rhetoric of freedom.[8]

 Paradigm Wars

Given my misgivings about neoclassical economic theory, my exposure to some alternative theoretical frameworks, and the contestation in the fields of education and economics in the 1970s, it is not surprising that much of my early research focused on what I came to view as the paradigm wars (e.g., Ashby et al. 1980; Klees 1986; Klees and Wells 1983).  These were not some abstract theoretical debates either.  My consulting work was focused on specific education innovations, most often in developing nations, and it was clear to me that how I evaluated any real world program or policy depended completely on what theoretical framework I adopted.  If I started from what we might call right-of-center politics, as does neoclassical theory, I got a very different picture than if I started from left-of-center politics, as does what I call political economy (see discussion below under Alternative Theories).  Therefore, what I did in most of my consulting work and research work is examine from, what I call for convenience here, Left and Right perspectives – topics such as educational technology, nonformal education, educational access and quality, vocational education, vouchers, education and work, etc.  Moreover, I found that these same underlying Left/Right debates were not limited to the fields of education and economics.  In doing work that related education to other fields of study, I started to find these same debates – in health and nursing, agricultural and rural development, communications, sociology, and other social sciences.[9]

            I saw then and still see now these paradigm debates as fundamental to our social understandings.  In the early eighties, I pointed out how I thought Mark Blaug (1975) was wrong in his famous review of the history of economics (Klees and Wells 1983).  Blaug (1975) concludes his paper by rejecting Thomas Kuhn’s (1962) understanding of paradigm change, which said that paradigm conflicts are resolved, not by internal rules of evidence, but by the nature of the social context in which these conflicts are embedded.  For Kuhn’s “external history” approach to be valid in economics, Blaug (1975, p. 431) said there must be instances of “(1) internally consistent, well corroborated, fruitful, and powerful scientific ideas which were rejected at specific dates in the history of a science because of specific external factors, or (2) incoherent, poorly corroborated, weak scientific ideas which were in fact accepted for specific external reasons.”  While Blaug (1975, p. 431) could think of “no unambiguous examples of either (1) or (2) in the history of economics,” a political economist’s response would be obvious. The best example of (1) is the general rejection of the political economics paradigm due to the domination of a capitalist system-generated worldview; and the best example of (2) is the acceptance of the completely unrealistic framework of perfect competition and efficiency because it conforms to capitalist ideology.

            One important point to make about these paradigm debates is that they were generally not direct debates.  That is, in practice, at least in the U.S., neoclassical economics has so dominated theory and practice that the vast majority of research offers only one view, containing few critiques or alternatives.[10]  Similarly, in education, despite having long-term critics like Apple, Arnove, Bowles, Carnoy, Gintis, Kelly, Kozol, Levin, Masemann, McLaren, Stromquist or Weis, the vast majority of educational researchers still do not seem to be very aware of any challenges.  There are exceptions, but still, critiques and alternatives are mostly seen only in the writings of critics because it is the critics who are forced to examine and confront dominant views.  Mainstream educational research, of course, has some internal debates, but my point is that it is not forced to engage with these fundamental critiques.[11] 

The Labor Market Segmentation Debate

In the late 1970s and 1980s, there was a significant theoretical debate in which mainstream researchers in economics and sociology were forced to engage with their critics.  The debate was over what was called labor market segmentation.[12]  In economics, this was about a challenge to the neoclassical idea that there was just one big perfect labor market in which success was determined by your individual human capital characteristics.  To the contrary, political economists and other critics of the neoclassical story saw an imperfect labor market with fractures (e.g., divisions into primary and secondary labor markets) and structures (e.g., large firms, unions, sexism, and racism) that greatly influenced whether an individual succeeded.  In sociology, this was about a similar challenge to the idea generated from the dominant structural-functionalist theory and its derivative status attainment theory that, like economics, argued that individual success was determined by individual characteristics.  To the contrary, a growing wave of critical sociologists, often sharing a conflict theory critique of structural functionalism, argued, like political economists, that success in the labor market was greatly determined by structural factors.

            I found this debate interesting for many reasons.  First, it was a major debate in both the economics and sociology fields.  Second, labor market success, in terms of measures like income, status, and mobility, is one of the most studied topics in both fields.  Third, the underlying debate was about one of the most important issues of our times, the nature of poverty and inequality.  The dominant views in sociology and economics saw labor market inequality as resulting from individual differences which led to policy recommendations to improve individuals through education and health, sometimes accompanied by a blame-the-individual-for failure attitude.  The critics saw the source of poverty and inequality as structural, less amenable to simplistic solutions and more in need of systemic reform.  Fourth, the outcome of the debate in the two fields was very different.  The debate did not really challenge neoclassical preeminence in economics but, in sociology, it overturned the dominance of structural functionalism and status attainment theory.[13] 

            The other feature of this debate that interested me is that proponents of the dominant perspective were forced to respond directly to critics.  The labor market segmentation debate was one of the rare instances where we had an actual back and forth between scholars subscribing to conflicting paradigms.  However, it was depressing to find that both sides used essentially the same, three-point argument:[14]

1.  ‘You have no theory.’ 
2.  ‘Your methods are wrong.’ 
3.  ‘Even if we were to use some of your theory and methods, the empirical results still confirm our original conclusions.’

Now these points were not empty ones – both sides could explain why they thought the opposition had flawed theory, methods, and results (though often through creating straw persons).  But nonetheless, there was something sophomoric about the debate.  Although no one went ‘Nyah, Nyah, Nyah’ you got the feeling they were close.  I do not mean this reflection on the sophomoric nature of the labor market segmentation debate to imply a ‘plague on both your houses’ conclusion.  After all, I am very much committed to one of the houses, as I explain in the next section.  And, of course, even if you accept my characterization of this debate, I will leave it to you to decide how typical this debate is.  Unfortunately, I find it very typical.  For the most part, adherents of divergent paradigms do not confront each other, but when they do, little productive engagement occurs.

Now, this may simply be the necessary result of true paradigm clash, where incommensurate theories and methods make scientific resolution of differences an impossibility. But if that is so, it raises fundamental questions about how we do research and how we make decisions.  It calls into question the scientific paradigm that putatively underlies research findings and policy choices today.  If dominant approaches to theory and method cannot guide our policy choices, then we need to broaden our purview.  We need to consider alternative theories, methods, and practices and need to embed these considerations in legitimate political processes instead of pretending they can be adjudicated through some technical, scientific search for truth.    

Alternative Theories

In economics, in most Western countries, neoclassical economics predominates.  This is true in many but not all developing countries.  This theoretical dominance is linked to the domination of capitalism (Amin, 1998).  Both have become even more entwined and enshrined over the last twenty years or so, especially with the fall of the Soviet Union.  It has become commonplace for those on the Right to proclaim that we have reached ‘the end of history,’ ‘the end of ideology’ (Fukuyama 2006). But, of course, this is not true.  The Left, as a theoretical and practical enterprise, is vibrant, very much alive and well across fields of study and across the globe.

My disenchantment with neoclassical economics, modernization theories, and related dominant approaches to understanding education and development made me search for alternative theories.  What I saw first, in some of my graduate school economics courses, was the work of U.S. political economists who drew on Marx and a group of neo-Marxists.  What I saw in my education courses with Carnoy and Levin was how they and others used this framework to understand education.  Issues of class predominated, but while I was in graduate school I read Simone de Beauvoir’s (1972) The Second Sex, and that book and a growing leftist feminist movement greatly influenced how I saw education and development.  It was clear to me that gender and patriarchy were as fundamental to understanding our social world as class and capitalism.  And it also became clear that issues of race and ethnicity added another important dimension.  This was the time I started doing research that crossed fields and found that these same critical issues raised in education and economics were raised throughout the social sciences and applied fields.

            Today, I see an even stronger and more remarkable confluence of critical perspectives than ever before.  While they cross disciplines and applied fields, these alternative theories, such as the following, are very much interrelated: dependency; world systems, critical, neomarxist, progressive economic, economic reproduction, cultural reproduction, resistance, feminist standpoint, gender and development, socialist feminist, critical race, queer, intersection, critical postmodern, poststructural, postcolonial, and critical pedagogy.  And this does not include all the related critical theories within each social science and applied field.

I am not saying that these theories offer identical perspectives, just that they share essential commonalities with respect to addressing the two major questions social theories face: ‘How do we understand our social world?’ and ‘What can we do about it?’  In terms of understanding, most fundamentally, all these theories are focused on marginalization.  They see the world as composed of systems and structures that maintain, reproduce, and legitimate existing inequalities.  From these perspectives, inequalities are not system failures, but the logical consequence of successful system functioning.  In terms of what to do, while most of these theories recognize that reproduction is pervasive, they also agree that there are serious challenges to reproduction.  There is general agreement that those challenges have two interrelated sources.  One is that the systems and structures that dominate are not monolithic but are pervaded by contradictions, such as that between the stated value of political democracy and the reality of economic authoritarianism or that between the stated value of human equality and the reality of systematic discrimination.  The other is a belief in human agency, in that oppression can be recognized and fought individually and collectively.

            While these theories do not have a name in common, I sometimes call them “critical theory,” “critical paradigm” or “political economy” and will use these terms in this presentation to refer to this intersection of these alternative theories.  And, as I see it, it is border crossing among these theoretical perspectives which forms the terrain of what we might call critical comparative education today.  Here lies an incredible comparative advantage of our field – it offers one of the few vantage points from which such an array of alternatives is studied and therefore where their intersections become visible.[15] 

Reflections on Method

In this section I focus on regression analysis, the statistical technique for uncovering the causal impact of one variable on another.  While some might see this as technical and esoteric, regression analysis is the dominant methodology for studying educational and social policy and practice and it is important that all researchers and practitioners recognize its fundamental flaws.  As a graduate student in economics, I was well-trained in these quantitative research methods.  This set of methods, involving causal modeling and estimation via some form of regression analysis, was dominant in economics at that time (and still is), and since then has become the dominant method in many social science and applied fields, even making inroads in anthropology and history.  But, even in graduate school, I had misgivings.  The theory behind regression analysis reminded me of the failure of neoclassical economic theory    If the assumptions of neoclassical economics held, then prices were accurate indicators of economic value and the invisible hand of capitalist markets worked.  However, the assumptions were such that they could never be true, and Second Best Theory was unforgiving; under real world conditions, prices utterly lose their social meaning.  In regression analysis, one also begins with a few assumptions about your model – that all variables are included, measured correctly, and their functional interrelationships accurately specified.  If the assumptions hold true, then regression coefficients are accurate estimators of causal impact.  However, in the real world there are always multiple failures of the assumptions.  Regression analysis theory does talk about the failure of one assumption at a time, but offers no guidance as to how inaccurate the resulting regression coefficients are under real world misspecification conditions.

Despite my misgivings, some of my early research used regression analyses and, to this day, I insist that students learn regression analysis and still work closely with students who use it in their research.  Nonetheless, because of these misgivings, I became very attentive to methodological debates.  Part of my focus has been on the flaws in regression analysis and part has been on alternative approaches to research methods.  In this section I discuss both. 

Anatomy of a Finding

Upon taking a position at FSU in 1981, I was fortunate enough to work with Sande Milton, a sociologist of education, who shared my interest and misgivings about these kinds of quantitative research methods.  We found ourselves paying attention to the results of regression analysis research.  In particular, we found ourselves looking at well-publicized findings in education and trying to understand on what they were based and what they really said. 

            After years of looking at particular findings, our conclusions were dismal.  We found that literatures using regression analysis research generally found little in the way of consistent findings.  Those conclusions that were reached offered no consistent information beyond what you would get from correlation data.[16]  An example was Walker and Schaffarzick’s (1974) review of the literature on the effectiveness of curricula.  After an exhaustive review of millions of dollars spent on curriculum research, they came up with two conclusions that could be supported by this research:  1) if the subject was in the curriculum, students learned it better than if it were not and 2) if the subject were emphasized, students learn it even better.  While, these are not empty conclusions, they offer little support for the benefits of a supposed scientific method.

When, rarely, a particular consistent finding that could be useful to policy is reported across studies, it often turns out that the consistency is more in the eye of the beholder than in the data.  One of the few examples touted as a consistent finding was the positive effect of textbooks on student achievement.  Three regression analysis based studies were singled out in major literature reviews (Lockheed and Verspoor 1991; Fuller 1987) as offering convincing research on this point.  When I examined each, I found fundamental flaws in the studies they cited.  For example, Lockheed, Vail, and Fuller (1986), looking at textbook use in Thailand, used a very difficult-to-interpret textbook variable, controlled for hardly any teacher or school variables, and actually found that textbooks did not matter in some of their regressions.  Stephen Heyneman and colleagues (1984), looking at textbook use in the Philippines, had no pretest measure, no school or teacher variables were included, and the dummy variable on textbook usage was very unclear.  The third study reported the results of an experiment that looked at textbook usage in Nicaragua (Jamison et al. 1981).  While, in theory, experiments might not need regression analysis controls, in practice they mostly do because experimental controls in field settings are always inadequate.[17]  Dean Jamison and colleagues controlled for a few variables but again left out almost all home and school variables.  Moreover, the textbook variable was not statistically significant in their individual data (it was in their group averages) and only showed an effect for 1^st grade not 4^th grade.  I go back to this old research because the effectiveness of textbooks is one of the few consistent findings from this input-output research, yet we see that the research is fundamentally flawed, subject to rampant misspecification and contestable interpretations.  I want to point out that these are well-known, competent researchers.  It is not my contention that they did an especially bad job as much as that good regression analysis practice is simply not possible.[18] 

The Determinants of Earnings

            While I found my observations about these methodological problems convincing, they were not very likely to be considered convincing evidence by researchers concerned with methodological issues.  So Sande Milton and I set out to develop a convincing argument using what is probably the most studied topic using regression analysis, the determinants of earnings.  One of my greatest professional regrets is that we never published the resulting paper.  One of my proudest moments is that when I sent the paper to Mark Blaug, the preeminent economist of education, he said that while he was unhappy with our conclusions, if he were editor of the Journal of Economic Literature, the top economics journal for literature reviews like this, he would publish it as is because the profession needs to discuss it.  However, Sande and I thought economists would not be open to the argument, so we wound up sending it successively to the top two sociology journals.  Both came back to us with split reviews, one saying, like Blaug, we need to publish and discuss this and the other saying that we have known about these problems since graduate school and the authors are throwing out the baby with the bathwater.  Unfortunately, Sande and I got tired of revising it (Klees and Milton 1986).

This paper was a logical follow-up to the labor market segmentation research I had been doing at the time since all of this literature used regression analysis to “prove” their side of the debate.  The argument of the determinants of earnings paper was straightforward.  The three principal conditions necessary for the regression coefficients of an “earnings function” to be accurate estimators of true causal impact are very far from being fulfilled.  First, all relevant variables that may affect earnings can never be included.  Our theories literally posit dozens of variables, and which variables are included in a particular regression study is idiosyncratic.  Examples of variables that some researchers have considered relevant are: health status, years of schooling, quality of schooling, type of schooling, cognitive ability, race, ethnicity, religion, socioeconomic status, gender, immigration status, marital status, participation in a union, job search, occupation status and differentiation, labor market segment, firm and industry characteristics, and many more.  Second, we do not know the “right” way to measure most of these variables.  Measurement is ad hoc and varies from study to study.  Third, the functional interrelationship between variables is not known.  While it is common to use the natural logarithm of income as the dependent variable, even neoclassical economists admit the basis for doing so is very weak and, in actuality, what would be needed is to specify some unknown set of complex simultaneous equations filled with variables subject to complex interactions.      

            The result of this state of affairs is endless misspecification – by necessity.  Each researcher has an almost infinite array of choices in how they specify the earnings function they estimate.  Each regression study is never a replication but always different from others in many respects.  The upshot is each regression study is idiosyncratic.  Since it is relatively easy to get significant coefficients, especially with large data sets, everyone finds their particular variable of interest to be significant.  When there is controversy, everyone finds empirical evidence to support their side of the debate.  Every segmentation theorist finds labor market segments to be a significant factor in determining earnings and other labor market outcomes, yet no neoclassical economist or structural functionalist sociologist ever does. 

Education and Agriculture Production Function Studies

This dismal conclusion that the major social science tool for empirical research may be a dead end is demonstrable in all its uses.  Perhaps the second most common use of regression analysis is in education where it is used to estimate what are called educational production or input-output functions.  The dependent variable usually studied is the score on some achievement test.  The three conditions for proper specification are again impossible to fulfill.  First, the array of potential independent variables is huge, including, for example: socioeconomic status, gender, race, ethnicity, age, homework effort, computer use in the home, previous learning, ability, motivation, aspiration, peer characteristics, teacher degree level, teacher practices, teacher ability, teacher experience, class size, school climate, principal characteristics, curriculum policies, to name a few.  Second, there is no agreement on how to measure most, if not all, of these variables.  Third, again the possible functional interrelationships are innumerable.  Contrary to the linear formulation usually run, recursive and simultaneous equation formulations with an array of interaction terms among the independent variables have been posited but little used.

            Literally hundreds of these educational production function studies have been done.  Again, with such an infinite array of specification choices, almost every study is unique and idiosyncratic.  Eric Hanushek (2004, 1979) has, over the long term, studied and summarized the results of such studies.  Not surprisingly, he and others have found inconsistent results.  However, he and the vast majority of quantitative researchers cling to the hope that improvements in models and data can eventually show some clear results.  To the contrary, I see the complete indeterminacy of this form of research built into the very assumptions on which it is based.

At one time, I thought that perhaps regression analysis could work under certain circumstances, such as when it was used to examine input-output relations that were based on physical relations more than human and social relations.  However, when I got involved in estimating agricultural production functions – the relation of agricultural output to the array of inputs that produced it (Ashby et al. 1980),  I found there was no greater ability to fulfill the conditions for proper specification here than anywhere else.  First, the array of potential agricultural inputs was enormous:  from amounts and type of seed and fertilizer used to the way in which they were applied; amount and distribution of rainfall and sunshine; farm size; the number, use, and education of farm workers; farm management practices and the education of the farmer; access to credit and extension services; and many more.  Second, there was again disagreement about how to measure inputs and outputs.  Third, there was disagreement about the nature of the functional interrelationship between them.  When I ran my data, I found that with one specification some farm practices and characteristics were significant and with a different specification others were, and there was no theoretical or practical means of choosing among a very large variety of alternative specifications, each with different results.  This is always the case in regression analysis. 

Implications

In theory, when using regression analysis, you are supposed to start with a complete model specification, and then take your data and estimate it, a one-shot deal.  Given the indeterminacy of model specification, no one does that in practice.  In his now classic article, “Let’s Take the Con Out of Econometrics,” Leamer (1983, p. 36) describes regression analysis in the real world and its consequences:

The econometric art as it is practiced at the computer … involves fitting many, perhaps thousands, of statistical models….This searching for a model is often well-intentioned, but there can be no doubt that such a specification search invalidates the traditional theories of inference.  The concepts of unbiasedness, consistency, efficiency, maximum likelihood estimation, in fact, all the concepts of traditional theory utterly lose their meaning by the time an applied researcher pulls from the bramble of computer output the one thorn of a model he likes best, the one he chooses to portray as a rose.

The practical question then becomes whether we have learned anything from all this research?  Most quantitative researchers would say they have, but I believe that such learning, if examined, would turn out to be from a subset of studies done from a perspective with which the researcher agreed.  As Leamer (1983, p. 37) put it: “Hardly anyone takes data analyses seriously.  Or, perhaps more accurately, hardly anyone takes anyone else’s data analyses seriously.”

As above, this is the case with the LMS literature.  Even within paradigms, there is no replication.  Hardly anyone ever uses anyone else’s specification without “improving” on it, arguing explicitly or implicitly that the previous study was incorrect.  These remarks do not imply that, at least within paradigms, there is no cumulative learning from one another’s arguments.  Such learning does take place.  However, the argument here suggests that regression-based causal inference is simply an excuse for theorizing but does not provide any valid evidence for it. 

I believe that this type of quantitative methodology is a dead end, no better than alchemy and phrenology, and someday people will look back in wonder at how so many intelligent people could convince themselves otherwise.  This is not a problem that better modeling and data can fix.  But I should say that I do not see the essence of the problem as quantification.  Quantifying social phenomena clearly has its limits and, at best, yields approximations (Samoff 1991).  But cross-tabulations and correlations are useful to suggest interrelationships.  As is well-known, however, any associations found may be spurious and have a myriad of alternative explanations.  The problem is that the causal relations underlying such associations are so complex that the mechanical process of regression analysis has no hope of unpacking them.  If we are interested in looking at quantitative data, I am afraid we are stuck with arguing from cross-tabulations and correlations.  This is a dismal prospect for most quantitative researchers who have spent years becoming virtuosos at data analysis and see the implications of my argument as essentially abandoning the research enterprise.  Fortunately for us, if not for them, the research enterprise is alive and well, with a myriad of alternative methodologies with which to investigate our educational and social world.

 Alternative Methods

When I went to graduate school, introduction to research methods courses focused almost exclusively on Campbell and Stanley’s (1963) examination of the design and analysis of quantitative experimental and quasi-experimental studies.  This is still true today within certain fields and university departments.  However, the past 30 years has seen a blossoming of alternative approaches to research methods, especially in education, but in other fields as well.  Education has been at the forefront of such changes, largely, in my view, because many of the changes were generated within the field of program evaluation which grew, in large part, from the educational evaluations that were mandated by the U.S. Congress in the 1960s and 1970s.  Many of those involved in evaluation fieldwork simply found that the quantitative approach to research and evaluation could not capture the experience of the programs they were studying and drew upon other traditions, such as in sociology or anthropology, or invented new approaches.  In subsequent years, these forays yielded a wide array of alternative methods for research and evaluation (Mertens, 2004)

For a number of years, I have been fortunate to teach our department’s Introduction to Research Methods course.  While any grouping of methods is somewhat arbitrary and their labeling always problematic, the course is divided in three, focusing in turn on quantitative/positivist methods, qualitative/ interpretive methods, and critical/transformative methods.  There is a large literature on the qualitative/quantitative debate.  Some argue too much has been made of it, while others, whom I agree with, argue that there are fundamental differences in outlook that need to be considered (Smith and Hesushius 1986).  Regardless, it is clear that there are lots of qualitative alternatives to quantitative experimental and regression analysis approaches, including:  case study, ethnography, grounded theory, phenomenology, narrative, and oral history, to name a few.

Additional methodological alternatives are offered by critical/transformative perspectives which come out of the array of theories in the critical paradigm discussed above.  These perspectives criticize the fundamental lack of objectivity of positivist/quantitative research and qualitative/interpretive research, arguing that there is no neutral research, and that too often such studies are done in support of dominant interests.  Critical/transformative research takes an explicit position to work in the interests of marginalized people.  This includes research under the labels of participatory, action, feminist, indigenous, critical, critical ethnography, and critical race (Denzin et al. 2007; Mertens 2004).[19]

Proponents of quantitative research recognize that some of these alternative methods exist but usually, at best, relegate them to the realm of generating ideas, not to the scientific process of building knowledge of the social world.  To the contrary, many proponents of alternative methods argue that they are as or more valid, reliable, and generalizable than quantitative.[20]  For example, Miles and Huberman (1994, p. 434) argue:

Qualitative studies…are especially well-suited to finding causal relations; they look directly and longitudinally at the local processes underlying a temporal series of events and states, showing how these led to specific outcomes, and ruling out rival hypotheses.  In effect we get inside the black box; we can understand not just that a particular thing happened, but how and why it happened.

Similarly, strong arguments are made for the transferability and generalizability of qualitative and critical research (Donmoyer 1990). 

To conclude, I make a similar point here to that I made under reflections on theory.  If the dominant methodological paradigm is bankrupt, as I argue, then there is no need for despair since there are a plethora of alternative approaches that we can use to study educational and development policy and practice.

Reflections on Practice

There are few, if any, armchair comparative and international educators.  Ours is a field focused on practice and engaged in practice.  While my principal position has been as a university professor, education policy and practice has always been a focal point of my research and of my work as a consultant.  I have worked on several dozen projects in as many countries.  Given my critique of theory and method, it is not surprising that I have been and continue to be deeply critical of educational and development policy and practice. 

Central to my critique of development practice is my critique of neoliberalism, which has dominated development discourse for over a quarter of a century.  Neoliberalism derives directly from neoclassical economic theory.  Neoliberals are fundamentalists.  They believe in a strict interpretation of the neoclassical bible of perfect competition.  They generally believe that free markets do work in the social interest and that government does not.[21]  It is neoliberals who have given us the Washington Consensus and policies designed to curtail the state, deregulate, privatize, and liberalize.  These practices, embodied in the Structural Adjustment Programs (SAPs) of the 1980s and 1990s, continue to today as part of the Poverty Reduction Strategy Process (PRSP).  The PRSP, which is supposed to orient all work by the International Monetary Fund and the World Bank, is a travesty of its intent; it is not country-owned, it does not allow for serious consultation with civil society, and its results look a lot like the bankrupt SAPs.  I believe that as a result of neoliberal-shaped globalization we have had more than a quarter century of public policies and private actions that have contributed to the further marginalization of certain individuals, groups, and nations, while privileging others (Klees forthcoming).

Not all neoclassical economists are fundamentalists.  Liberal neoclassical economists dominated in the 1960s and 1970s.  They believe in the need for strong government intervention to correct a world prone to unacceptable inequalities and market imperfections.  From a critical perspective, this is a vast improvement over a neoliberal stance but does not go far enough, as I will discuss below.

As noted above, education policy and practice for the past quarter century has been embedded in what some have called neoliberal globalization (Carnoy 1999, 2000).  In practice, this has meant that education policy, as with economic and social policy, experienced a significant and sustained shift from that of the 1960s and 1970s (Klees forthcoming).  Below, I briefly reflect on some of these policies and practices. 

Emphasizing Learning Outcomes

Of course, learning outcomes are important.  But the new wisdom, especially of the last decade or so, is that they are all that is important.  It used to be that there was a strong focus on educational inputs and processes – are there schools available and functional, are trained teachers available, are class sizes reasonable, are there sufficient books and other learning materials available, is teaching learner-centered, are students actively engaged, etc.?  Today these questions are rarely asked or answered.  Attention to learning outcomes tends to lead to massive testing programs (like the relatively useless No Child Left Behind in the United States) – and little more.  It seems that this is the era of the cheap fix.  Neoliberals want to measure learning outcomes, but no one wants to spend the money it will take to improve learning outcomes.  As a wag pointed out, the current emphasis on testing for achievement is like giving a sick patient a thermometer instead of what is needed to cure the illness.[22]   For the most part, this new emphasis on learning outcomes seems to have gotten us harmful policies like large class sizes and the hiring of untrained, paraprofessional teachers.

 Privatization, Vouchers, and Charters

Privatization has been the watchword of neoliberalism and has gotten a lot of play in education.  Voucher schemes, which spend public funds to send some students to private schools, have been tried in a number of countries with disappointing results.  Most studies show that student achievement is no higher in voucher schools than in non-voucher schools (Klees forthcoming; Levin 1998).[23]  Charter schools, which are public schools with diminished regulation, have become quite popular in the U.S., but again the research does not indicate charter school students have higher achievement than non-charter school students (Imberman 2007).

            Edison Schools, Inc., a U.S. based private company provides a case in point.  It has been hired by many school districts to manage failing schools.  External evaluations report mediocre test results, with Edison students doing no better than comparable students in public schools (Levin 2006; Bracey 2002).  However, while public schools are trying to improve student achievement, they must also be responding to the many other needs expressed by parents and communities, e.g., providing counseling, health, welfare, policing, and other services. Yet with all this pressure to serve so many needs, public schools are still able to produce an achievement gain equivalent to Edison’s – a company that was hired to focus on just one outcome, achievement.  Why would anyone think that the private sector can offer anything to improve education generally?

Vouchers and charters will generally increase inequality, as those who are more well-to-do are able to take greater advantage of them.  Vouchers and charters also leave a weakened and ignored public school behind.  Vouchers and charters are also another one of neoliberalism’s cheap fixes.  Neoliberal reforms, like privatization, have often been management and governance reforms which require little in the way of additional resources.  They have become the cheap magic bullet of the Right – you do not have to worry about providing smaller class sizes, better technologies, qualified and motivated teachers – you just change the organization structure.

Merit Pay

Merit pay – bonuses – for high-achieving teachers and schools has received periodic attention.  There is a simple logic to rewarding superior performance.  However, gains in student performance may not be due to the particular teacher or the school.  Gain scores reflect the joint effect of many individual, home, peer, and school influences.  While, in theory, some sort of regression analysis could control for other variables, in practice (as discussed earlier) we have no ability to specify the analysis correctly.  Different controls lead to rewarding different teachers and schools with no rational basis for choosing among them.  Merit pay is simply one example of what is wrong with the simplistic neoliberal policy of tying resources to output.  There is nothing wrong with the concept, but the idea of a simple connection between measurable inputs and outputs is wrongheaded (Klees forthcoming; Rothstein 2008).  If it was such a good idea, why, for example, is there not a push to have merit pay schemes for World Bank staff based on the percent by which their work reduces poverty?   Rewarding good teachers and helping problematic ones can best be achieved through an evaluation process that reflects the complexities of teaching and uses professional judgment, principally by peers (Millman and Darling-Hammond 1990). 

Higher Education

The neoliberal era has been characterized by the deprecation of higher education.  In the 1960s and 1970s, education policy paid attention to the joint expansion of primary, secondary, and higher education as essential for development goals.  In the 1980s, this changed sharply; primary education was touted, public higher education was to be cutback and privatized, and secondary education was put in a holding pattern (World Bank 1995).[24]  Ostensibly, this was based on rate of return analysis that showed higher returns to primary education.  However, these rates of return were based on both failed economic theory and flawed empirical analysis, as described earlier, thus forming an illegitimate basis for such important decisions (Klees forthcoming; Samoff 2007).  Even if you accept rate of return evidence, research in the late 1990s concluded that we had been vastly underestimating the returns to higher education by not taking into sufficient account the diffuse social benefits which neoclassical economists call externalities.  This admission of error called into question 20 years of policies that cut back higher education, harming individuals and nations, making them less competitive in the world system.  Political economists would argue, however, that it is not rates of return but the interest of the world system that led to these policies; that is, these policies had the effect of making developing countries more dependent on competing in terms of low wage labor, which was desirable in the 1980s and 1990s, but now, given the high costs of higher educated labor in richer countries, it is more useful to the world system to have developing countries compete more in this realm. 

Girls’ Education

            The Women’s Movement of the 1960s and 1970s, fueled by a confluence of liberal, neoliberal, and critical perspectives, has had far-reaching social consequences.  Among other things, more attention has been paid to women and development. While the nature of this consideration has been contested, in education, this has translated into attention to girls’ and women’s education, particularly the former (Diaw2005; Stromquist 1997).   Expanding girls’ education has been explicitly a part of Education for All (EFA) goals and the Millennium Development Goals (MDGs).  And, since 1990 when EFA was initiated, there has been some progress in increasing girls’ access to primary and secondary school (UNESCO 2003).

However, that progress has been slow and tortuous.  EFA goals on gender equity were postponed from 1995 to 2005, and then in 2005, neither EFA goals nor their equivalent in the MDGs were met – nor is it clear when they will be met.  When we go beyond access to look at quality issues, a much larger gender gap exists, as girls continue to be disadvantaged in multiple ways in the educational process.  Nelly Stromquist (1997, p. 60) discusses the need to go even further, beyond first and second level responses to access and quality, to a third level response “that challenges the ideological and material conditions that support women’s inferior position in society.”  However, progress in challenging the patriarchal nature of most societies is slow, and neoliberal policies either do not recognize the problem or avoid the issue.  For example, girls’ education as it is practiced by most development agencies is determinedly apolitical, often carefully divorcing itself from the activities of the local and national women’s movement, thus weakening its progressive potential (Stromquist et al. 2000). 

EFA and the MDGs

As with girls’ education, other original EFA goals were postponed.  Also, the nature of the goals changed (Torres 2000).  Before the World Conference on EFA in Jomtien, Thailand in 1990, World Bank staff told other participants in the EFA planning process that the focus had to be on primary education; adult education could not be included or the Bank would withdraw its support.  Nonetheless, at Jomtien, pressure by NGOs and others included attention to adult education as part of EFA goals.  In practice, however, the attention was almost exclusively focused on attaining universal primary education (UPE) and, in the Dakar meeting revisions, and in the MDGs, UPE became the centerpiece (King 2007).[25]

The original EFA goals envisioned UPE by 2000.  In the mid-1990s, it was clear this would not be achieved, and so the goal was postponed to 2015 (as it is in the MDGs).  However, it is clear today to all analysts that under current efforts, the goal will not be even close to being fulfilled in 2015, nor will any of the other MDGs concerning poverty, health, gender equity, and the environment.[26] Yes, again, as with girls’ education, you can point to some progress.  But UPE goals and promises have been made every decade since the 1960s, with some progress, yet no fulfillment. 

So what is happening here?  Hans Weiler (1984) called it compensatory legitimation; more colloquially, I see it as a form of ‘good cop, bad cop.’  Shaky and poorly-performing economies, increasing poverty and inequality, widespread conflicts, and the equivalent of structural adjustment policies everywhere, all call into question the legitimacy of the social order – this is the bad cop.  To compensate for this, actors in the world system of neoliberal globalization must introduce polices, for example, EFA and the MDGs, aimed at ameliorating some problematic conditions and thus restoring legitimacy – this is the good cop.  This argument does not question the good intentions of the proponents of these policies but does question their effects.  Simply having these policies may be sufficient for compensatory legitimation; fulfilling them, judging by past experience, seems to be less important.  This does not have to be so.  We could achieve UPE and the rest of the MDGs in a very few years if we were willing to devote the kind of resources and attention that go into current priorities, like the Iraq War. 

Alternative Practice

            It has been more than a quarter century since the Reagan/Thatcher tide resulted in a neoliberal shift around the world.  Based on ideological beliefs, without convincing supporting evidence, the array of economic and educational practices above were rapidly introduced.  Clearly, alternative practices are possible.  In education, we could pay attention to the provision of sufficient educational inputs as well as the attainment of educational outcomes and not restrict the latter to test scores.  We could focus attention on improving public schools instead of instituting various forms of privatization.  We could pay all teachers well and have a professional peer evaluation system.  We could use a vastly expanded early childhood development system that truly tried to compensate for the unequal starts that children get.  We could reinvigorate public higher education and make it the centerpiece of a postsecondary education system.  We could face head on the challenges of a sexist world and situate our attention to the education of girls within that.  And, as I said above, we could fulfill our rhetorical commitment to EFA and the MDGs.

There are two questions raised by these examples of alternative practices.  First, what will it take to make them a reality?  Second, from a critical paradigm perspective, do they go far enough (Chan 2006)?  Let us start with the latter.  More fundamental reforms of educational practice are needed.  Some examples: In education, perhaps the best developed critical alternative approach is rooted in the work of Paulo Freire and elaborated in an extensive literature on critical pedagogy (McLaren and Kincheloe 2007).  There also has been interesting work on the implications of the global environmental crisis for a radical ecological education (Orr 1991).  Given the prevalence of war and conflict, there has also been a burgeoning attention to the substantial transformations needed to make peace education an integral part of our K-16 curriculum (Lin et al. forthcoming).  Applying a human rights perspective, there has also been significant work on the right to education and a rights-based education (Klees and Thapliyal 2007; Tomasevski 2003).

From a critical perspective, we will not have successful educational reform without fundamental social and economic transformation.  This means more than a rejection of neoliberal policies and a return to the liberal, welfare state policies of the past, although, given present circumstances, at least some critics would welcome such a return.  In practice, however, those policies did not offer a serious challenge to poverty, inequality, and marginalization.  There is much analysis of what would constitute a serious challenge.  There is exciting work being done on the theory and practice of alternative forms of social and economic life under such topics as participatory and deliberative democracy (Fung and Wright 1999); economic democracy (Alperovitz 2004; Hahnel 2005); global governance (Riker 2005); feminism (Sanders 2002); human rights (OHCHR 2002); alternative forms of globalization (Cavanagh and Mander 2002); and ecological economics (Daly 1996; Daly and Cobb 1994).

From a critical perspective, transforming our education, economic, and social systems are not straightforward.  They cannot simply be accomplished with the right policy but are results of struggle (Ginsburg et al. 1991).  The contradictions inherent in today’s world system – most importantly, perhaps, the contradictions between a rhetoric of fairness and equality and a reality that is fundamentally unfair and unequal – open up spaces for action.  Critics place a lot of faith in human agency to take advantage of those spaces – human agency being the individual and collective transformative choices and actions of those who share a critical perspective.    At the individual level, for example, teachers can sometimes offer an alternative education in their classroom, or development agency staff can sometimes defy current policies.  Nongovernmental organizations (NGOs) have developed programs, principally at a local level, that offer alternative approaches to dealing with our most difficult education and development problems.  NGOs and other civil society organizations can collectively lobby to challenge neoliberal education and development policies.  And, perhaps most promising, social movements – such as the women’s movement, civil rights movement, alter-globalization movement, landless movement, Dalit movement, the human rights movement, children’s rights movement, and others – bring the power of collective action to confronting the unfairness and irrationality of pervasive marginalization. 

Resistance and agency have changed laws and constitutions.  We have seen entire governments change, as in the pervasive shift to the left in Latin America in countries like Argentina, Bolivia, Brazil, Chile, Ecuador, Nicaragua, Uruguay, and Venezuela.

Even in the U.S., the pendulum may be swinging back to the liberal side, if a Democrat gets elected President.  While this still leaves us far from a critical stance, it could open up influential global agencies like the World Bank to some alternative perspectives.   

Conclusions

Theory, method, and practice concerning educational, social, and economic issues are hotly contested.  Unfortunately, given the unparalleled dominance of neoliberal globalization, in many national and international settings, that contestation is hardly visible.  Even relatively mild forms of dissent are often met by disregard, hostility, or derision.  I remember being struck by George Psacharopoulos telling me that some of his colleagues at the Bank called Lester Thurow, the noted liberal neoclassical economist, ‘Less Thorough.’[27]

In this presentation, I have argued that dominant economic and social theories are bankrupt and need to be replaced by the array of theories that have been developed focusing on understanding and overcoming marginalization.  I have also argued that the dominant, causal modeling, quantitative approaches to research have been a dead end, and should be replaced by an array of methodological alternatives, including many that depart from a critical perspective.  Finally, I have argued that our practices have failed to achieve their stated goals; instead of leading to a more effective and equitable system, they have been reproductive of our unjust and unequal social order. 

I learned early on that you had to take sides in these debates.  One of my earliest consulting projects was looking at the effects of nonformal education and information provision to poor Guatemalan farmers in the highlands.  After our presentation of the final report, representatives of farmer groups thanked me because I was the only one who mentioned that the real problem was not education and information, but that these farmers had been marginalized by pushing them onto tiny plots of unproductive land.  Similarly, I have been thanked – and critiqued – many times for raising such unpopular political issues.  I say this not to praise myself, I have not done much, but to underline what many of you know.  There is nothing neutral about anything we do.  Every paper, every project, every talk, and every class (if you are a teacher) is embroiled in these controversies.  In the interests of progress, it should be incumbent on all of us – no matter what your position -- to recognize, acknowledge, and engage these debates.

Notes