EDUCATIONAL DESIGN RESEARCH
Chapter 3. Design Research from a Learning Design Perspective.
Koeno Gravemeijer and Paul Cobb
> This chapter elaborates on an approach to design research that is categorized as falling within the broader category of design research that aims at: creating innovative learning ecologies in order to develop local instruction theories on the one hand, and to study the forms of learning that those learning ecologies are intended to support on the other hand.
> This approach has its roots in the history of the two authors: one has a background in socio-constructivist analysis of instruction. The socio-constructivist approach was inspired by a desire for understanding; the other has done work on realistic mathematics education (RME) that is carried out in the Netherlands. The RME approach is built because of a need for educational change.
> This chapter defines what design research is by discussing the three phases of conducting a design experiment: preparing for the experiment; experimenting in the classroom, and conducting retrospective analyses.
> This book uses an experiment on statistics to illustrate the various phases in a concrete design experiment.
Phase one – preparing for the experiment
> From a design perspective, the goal of the preliminary phase of a design research experiment is to formulate a local instruction theory that can be elaborated and refined while conducting the experiment.
This local instruction theory encompasses both: provisional instructional activities, and a conjectured learning process that anticipates how students’ thinking and understanding might evolve when the instructional activities are employed in the classroom.
> From a research perspective, a crucial issue is that of clarifying the study’s theoretical intent.
> In elaborating these points, It is given three main explanations that focus on:
> Clarification how one goes about establishing the learning goals, or instructional endpoints at which one is aiming, and the instructional starting points.
> The conjectured local instruction theory that the research team has to develop.
> The theoretical intent of an experiment.
> The preparation for a classroom design experiment typically begins with the clarification of the mathematical learning goals, which in practice, largely are determined by history, tradition, and assessment practices.
> Design researchers cannot just take the goal as a given when starting a design experiment. Instead, they will have to problematize the topic under consideration from a disciplinary perspective.
> The goal of design research is not to take the currently instituted or institutionalized school curriculum as a given, and to try to find better ways to achieve the already defined goals. Instead, the research team has to scrutinize those goals from a disciplinary point of view in order to establish what the most relevant or useful goals are.
> The focus in considering of starting point is to understand the consequences of earlier instruction, not merely to document the typical level of reasoning of students in a given domain.
> The existing research literature can be useful in the starting point. To complement such a literature study, the researchers will also have to carry out their own assessments before starting a design experiment. Besides it, they can use available items and instruments. In addition to written tests, there will also be a need for other forms of assessment, such as interviews, or whole class performance assessments.
> Performance assessments reveal the consequences of the students’ prior instruction. In this case the role of the teacher is to probe the students’ understanding and reasoning, and to find out why they use particular approaches.
Local instruction theory
> A local instruction theory consists of conjectures about a possible learning process and conjectures about possible means of supporting learning process.
> The means of support encompass potentially productive instructional activities and (computer) tools as well as an envisioned classroom culture and the proactive role of the teacher.
> The researcher anticipate how students’ thinking and understanding might evolve, revise instructional activities, reconcile the need to plan in advance and flexibility when building on the students’ current understanding.
> The available research literatures are useful in providing articles that are relevant for construing local instruction theories, such as the process of students’ learning in a particular domain together with descriptions of the instructional settings, the tasks, and the tools that enabled or supported that learning.
> If available literatures related to the topic are limited, design researchers have to turn to other resources, such as curricula, texts on mathematics education, and the like. This is done in order to take ideas to construe an instructional sequence.
=> The classroom culture and the proactive role of the teacher
> Instructional designers need to focus on instructional tasks and tools as potential means of support. Besides it, they have to consider the characteristics of the envisioned classroom culture and proactive role of the teacher.
> Design researchers have to consider the nature of classroom norms and the nature of classroom discourse.
> One of the tasks of the teacher will be to establish the desired classroom culture.
> The proactive role of the teacher will include introducing the instructional activities, selecting possible topics for discussion, and orchestrating whole class discussions on these topics.
=> Theoretical intent
> One of the primary aims of a design experiment is to support the constitution of an empirically grounded local instruction theory. Another aim of a design experiment might be to place classroom events in a broader context by framing them as instances of more encompassing issues.
> A series of design experiments can serve as the context for the development of theories or theoretical frameworks that entail new scientific categories that can do useful work in generating, selecting, and assessing design alternatives.
> Defining scientific terms is more like finding and validating a new category of existence in the world, for which we may use the term ontological innovation. Examples of such ontological innovations include the framework for interpreting classroom discourse and communication, the discovery of meta representational competence, the theory of quantitative reasoning, the design heuristic of emergent modeling and RME theory in general.
> Ontological innovations can play a dual role. On the one hand, they can serve as lenses for making sense of what is happening in the complex, more-or-less real world instructional setting in which a design study is conducted. On the other hand, ontological innovations can function as guidelines or heuristics for instructional design.
> RME theory may play a similar dual role; the theory not only guides the design, but also offers a framework for interpreting the students’ learning process.
Phase two – the design experiment
> The second phase consists of actually conducting the design experiment. When all the preparation has been done, the overall endpoints are specified, the starting points defined, and a conjectured local instruction theory formulated, the design experiment can start.
> The purpose of the design experiment is both to test and improve the conjectured local instruction theory that was developed in the preliminary phase, and to develop an understanding of how it works.
> The discussion of the design experiment will consist of three main points, namely:
(1) The iterative sequence of tightly integrated cycles of design and analysis, which is key to the process of testing, improving, and understanding.
(2) The kind of data that are generated.
(3) The interpretative framework(s) one uses for interpreting classroom discourse and communication as well as for interpreting students’ mathematical reasoning and learning.
Microcycles of design and analysis
> At the heart of the design experiment lies a cyclic process of (re)designing and testing instructional activities and other aspects of the design.
> In each lesson cycle, the research team conducts an anticipatory thought experiment by envisioning how the proposed instructional activities might be realized in interaction in the classroom, and what students might learn as they participate in them.
> During the enactment of the instructional activities in the classroom, and in retrospect, the research team tries to analyze the actual process of student participation and learning. On the basis of this analysis, the research team makes decisions about the validity of the conjectures that are embodied in the instructional activity, the establishment of particular norms, and the revision of those specific aspects of the design. The design experiment therefore consists of cyclic processes of thought experiments and instruction experiments (Freudenthal: 1991)
Figure 1. Developmental research, a cumulative cyclic process
> According to Simon’s idea about mathematical teaching cycle,” a mathematics teacher will first try to anticipate what the mental activities of the students will be when they participate in some envisioned instructional activities. Then the teacher will try to find out to what extent the actual thinking process of the students correspond with the hypothesized ones during the enactment of those activities to finally reconsider potential or revised follow-up activities.
> To characterize the teacher’s thinking, Simon coined the term, “hypothetical learning trajectory,” which he described as “The consideration of the learning goal, the learning activities, and the thinking and learning in which the students might engage” (Simon 1995: 133). Then, the mathematical teaching cycle may be described as conjecturing, enacting, and revising hypothetical learning trajectories.
> The evaluation of the micro-cycles of design and analysis concern inferences about the mental activities of the students, not merely observable behavior of the students, since the goal for the design researcher is not just to find out whether the participation of the students in those particular activities results in certain anticipated behaviors, but to understand the relation between the students’ participation and the conjectured mental activities.
> In a design experiment, the micro-cycles of thought and instruction experiments serve the development of the local instruction theory. There is a reflexive relation between the thought and instruction experiments and the local instruction theory that is being developed. The conjectured local instruction theory guides the thought and instruction experiment, while the micro-cycles of design and analysis shape the local instruction theory (Figure 2).
Figure 2. Reflexive relation between theory and experiments
> Micro-cycles require that the research team engages in an ongoing analysis of individual students’ activity and of classroom social processes to inform new anticipatory thought experiments, the design or revision of instructional activities, and, sometimes, the modification of learning goals.
> A local instruction theory encompasses both the overall process of learning and the instructional activities. So, it can be observed a process of conjecturing and revising on two levels, on the level of the individual classroom sessions, and on the level of the instructional sequence as a whole.
> Decisions about the types of data that need to be generated in the course of an experiment depend on the theoretical intent of the design experiment.
> In design experiment that focuses on the development of a local instruction theory, data can be collected by some ways, such us:
- Video record all classroom sessions
- Conduct pre- and post- interviews with the students, make copies of all of the students’ work
- Assemble field notes
- Appropriate benchmark assessment items that have been used by other researchers
- Audio record the regular research group meetings
- Conduct student interviews
> A key element in the ongoing process of experimentation is the interpretation of both the students’ reasoning and learning and the means by which that learning is supported and organized.
> Design researchers have to explicate how they translate observations of events in the classroom into scientific interpretations.
> Key elements of such a (potentially revisable) interpretative framework include:
(1) A framework for interpreting the evolving classroom learning environment,
(2) A framework for interpreting student mathematical reasoning and learning mathematics.
=> Emergent perspective
> The framework that is currently used for interpreting classroom discourse and communication is the emergent perspective.
> The framework can be viewed as a response to the issue of attempting to understand mathematical learning as it occurs in the social context of the classroom. With regard to the specifics of the framework, the column headings Social Perspective and Psychological Perspective involve a focus on the classroom community and on individual students’ reasoning, respectively.
Figure 3. An interpretive framework for analyzing individual and collective activity
> Social norms refer to expected ways of acting and explaining that become the classroom level established through a process of mutual negotiation between the teacher and students.
> The psychological correlate to social norms concerns the teacher’s and students’ individual beliefs about their own and others’ roles.
> The socio-mathematical norms can be distinguished from social norms as ways of explicating and acting in whole class discussions that are specific to mathematics.
> The last social aspect of the theoretical framework concerns the mathematical practices that are established in the classroom. A mathematical practice can be described as the normative ways of acting, communicating and symbolizing mathematically at a given moment in time.
> In contrast to socio-mathematical norms that are specific to mathematics, mathematical practices are specific to particular mathematical ideas or concepts. In addition, mathematical practices necessarily evolve in the course of an experiment whereas socio-mathematical norms tend to be more stable.
=> RME theory
> RME (Realistic Mathematics Education) theory not only offers design heuristics, but may also function as an framework for interpreting student activity in terms of learning mathematics.
> RME based on the idea of Freudenthal (1971, 1973) that mathematics should primarily have the character of an activity for the students and learning mathematics should ideally be experienced as expanding one’s mathematical reality. The term reality means common sense experiences as real at a certain stage.
> Freudental also emphasize the idea on guided reinvention which a process of guided reinvention would have to ensure that mathematical activity would foster the construal of mathematics as a body of knowledge by the students.
> The goal of RME is to support students in creating a new mathematical reality. This is to be realized by guided reinvention, or progressive mathematization. Progressive mathematization refers to a mixture of two forms of mathematizing, horizontally and vertically, which refers respectively to students mathematizing subject matter from reality, or mathematizing their own mathematical activ ity (Treffers 1987).
> According to Freudenthal (1983), mathematical “thought-things,” such as concepts, tools and procedures, are invented to organize certain phenomena. The reinvention heuristic then suggests that the instructional designer should try to find situations that create the need for the students to invent the mathematical thought-things the students are supposed to construct. To find such situations, the instructional designer should analyze the relation between those mathematical thought-things, and the phenomena they organize.
> Freudenthal’s level theory also shaped the RME view on educational models. Instead of ready-made models, RME looks for models that may emerge first as models of situated activity, and then gradually evolve into entities of their own to function as models for more sophisticated mathematical reasoning (Gravemeijer: 1999).
> The RME framework might generate additional points of focus, such as the following
1.Do the students rely on their own domain-specific knowledge?
2.Do the instructional activities provide the expected traction for the students’ informal solution procedures?
3.Do the solutions that the students develop offer possibilities for vertical mathematization?
4.Do the students mathematize their own informal mathematical activities?
Phase three – the retrospective analysis
> The retrospective analyses are conducted of the entire data set collected during the experiment.
> The goal of the retrospective analyses depends on the theoretical intent of the design experiment. However, one of the primary aims is typically to contribute to the development of a local instruction theory. Other goals may concern more encompassing issues, or ontological innovations.
> Although differences in theoretical objectives are reflected in differences in the retrospective analyses, the form of the analysis will necessarily involve an iterative process of analyzing the entire data set.
> In this chapter, the retrospective analyses is described in general, and then it’s discussed analyses to develop a local instruction theory, and finally describe analyses conducted to address more general research topics.
> The data sets typically include videotapes of all classroom lessons, video-recorded individual interviews conducted with all students before and after the experiment to assess their mathematical learning, copies of all the students’ written work, field notes, and audio tapes of both the daily debriefing session and weekly project meetings.
> The challenge from data sets is to analyze this comprehensive data set systematically while simultaneously documenting the grounds for particular inferences.
> In the design experiment about statistics that is used as an example in this book, researcher team first work through the data chronologically, episode by episode, and at each point we test our current conjectures against the next episode. As a result of this first ground of data analysis, it end up with a sequence of conjectures and refutations that are tied to specific episodes. In the second phase of a retrospective analysis, this sequence of conjectures and refutations in effect becomes the data. It is while meta-analyzing these episode-specific conjectures, confirmations and refutations, that particular episodes reveal themselves to be pivotal. These are the episodes that are typically included in research reports.
Reconstructing the local instruction theory
> One of the primary aims of a retrospective analysis is to support the constitution of a revised local instruction theory. However, it is important to emphasize that the results of design experiments cannot be linked to pre- and post- test results in the same direct manner as is common in standard formative evaluation because the proposed local instruction theory and prototypical instructional sequence will differ from those that are tried out in the classroom.
> The instructional sequence will be put together by focusing on and reconstructing the instructional activities that proved to constitute the effective elements of the sequence. This reconstruction of an optimal sequence will be based on the observations and inferences made during the design experiment, complemented by the insights gained by conducting retrospective analyses.
> Although the constitution of a revised local instruction theory is primarily a reconstruction activity, the retrospective analysis may spark design ideas that go beyond those that were tried out in the classroom. These insights might in turn create the need for a new experiment, starting with a new conjectured local instruction theory.
> In this cycle, the instructional sequence was radically revised and a further design experiment was conducted.
Encompassing issues and ontological innovations
> In addition to retrospective analyses that directly aim at the reconstruction and revision of a local instructional theory, a retrospective analysis might be conducted to place classroom events in a broader context by framing them as instances of more encompassing issues.
> In ontological innovations, which might include issues such as the framework for interpreting classroom discourse and communication, meta-representational competence, quantitative reasoning or emergent modeling, the aim of the analysis is to frame events that occurred in the design experiment classroom as instances, or paradigm cases, of a broader class of phenomena. The goal is to come to understand (the role of) the specific characteristics of the investigated learning ecology in order to develop theoretical tools to be applied in the same phenomenon in other learning ecologies.
> The course of a design experiment can be characterized in terms of the learning process of the research team. This characterization is especially fitting for the construal of the local instruction theory, which encompasses two processes:
- The learning process that is inherent to the cyclic process of (re)designing and testing instructional activities and other aspects of the initial design,
- The retrospective analysis that scrutinizes, and builds on, this primary process, and looks for patterns that may explain the progress of the students.
> In relation to this learning process, the methodological norm of “trackability” is used as a criterion in ethnographic research. Smaling (1990, 1992) connects trackability with the well-known criterion of “reliability.” He notes that reliability refers to the absence of accidental errors and is often defined as reproducibility. He goes on to say, that for qualitative research this means virtual replicability. This ethnographic norm of trackability fits with Freudenthal’s conception of developmental or design research.
> Design research aims for ecological validity, that is to say, (the description of) the results should provide a basis for adaptation to other situations. One of the primary aims of this type of research is not to develop the instructional sequence as such, but to support the constitution of an empirically grounded local instruction theory that underpins that instructional sequence.
> The intent of design research is to develop a local instruction theory that can function as a frame of reference for teachers who want to adapt the corresponding instructional sequence to their own class- rooms, and their personal objectives.
> Feedback from teachers on how the instructional sequence was adjusted to accommodate various classrooms can strengthen the ecological validity significantly. Hence, it critical to have repeated trials in a variety of settings.
Developing domain-specific instruction theories
> Besides provides a means of developing local instruction theories, design research also contributes to the development of a domain-specific instruction theory, in case the RME theory. In this regard, theory development will be divided into various levels: the instructional activities (micro-theories) level, the instructional sequence (local instruction theories) level and the domain-specific instruction theory level.
> Design research has the potential to bridge the gap between theory and practice, as domain-specific instruction theory can be categorized as episteme and micro-didactic theories as phronesis. In this case, scientific knowledge is grounded in practical wisdom while simultaneously providing heuristics that strengthen practical wisdom.
Developing ways of analyzing innovations
> A related challenge is that of developing ways of analyzing innovations that make their realization in different classrooms commensurable. An analysis of classroom events structured in terms of constructs such as social norms, socio-mathematical norms, and classroom mathematical practices.
> This part of the retrospective analysis raises its own methodological issues. Then it’s focused on the issues of generalizability when discussing the importance of viewing classroom events as paradigm cases of more encompassing issues. It is this framing of classroom activities and events as exemplars or prototypes that gives rise to generalizability.
> Generalizability is closely associated with the notion of a paradigm cases, trustworthiness is concerned with the reasonableness and justifiability of inferences and assertions. This notion of trustworthiness acknowledges that a range of plausible analyses might be made of a given data set for a variety of different purposes.
Design and research
> Design research is about researching and designing.
> Design research presupposes that there is an adequately grounded basis for designing the innovative learning ecology/instructional sequence.
> The description “learning ecology,” introduced by Cobb et al. (2003), might be more adequate as it emphasizes that we are dealing with a complex, interacting system involving multiple elements of different types and levels by designing these elements and anticipating how these elements function together to support learning.
> The theoretical base for the design should incorporate general background theories such as socio-constructivism, or socio-cultural theory, domain-specific theory and theories on specific elements of the learning ecology.