Comparing evidence on the effectiveness of reading resources from expert ratings, practitioner judgements, and research repositories

View author details View Less
  • 1 Columbia University, USA
  • | 2 World Bank, USA
  • | 3 New York University, USA
  • | 4 Houston Independent School District, USA
  • | 5 UNICEF, USA
Open access
Get eTOC alerts
Rights and permissions Cite this article

Background:

Education decision makers are increasingly expected to use evidence to inform their actions. However, the majority of educational interventions have not yet been studied and it is challenging to produce high quality research evidence quickly enough to influence policy questions.

Aims and objectives:

We set out to gather evidence on the efficacy of reading resources implemented at 23 struggling elementary schools in a large, urban district in the US. The schools were at risk of closure by the state.

Methods:

For each reading resource, we searched for existing effectiveness studies and collected professional judgements by surveying practitioners. We also used an expert survey to collect judgements from three reading experts. We compared the ratings among experts and between practitioners and experts. We also compared practitioner and expert judgements to evidence summaries from research repositories.

Findings:

We found evidence summaries in research repositories for only five of 23 reading resources used in the 23 schools. Experts showed poor to good agreement on ten questions about each resource. Agreement between practitioners and experts was low with practitioners generally rating resources more positively than reading experts.

Discussion and conclusions:

Practitioners may be overly optimistic about the efficacy of educational materials while experts have difficulty assessing how well the materials serve an unfamiliar population. In the absence of rigorous evaluations of locally-implemented programmes, district decision makers can review the consistency of evidence collected from practitioners and experts, along with external research evidence, to inform actions towards supporting and guiding struggling schools.

Abstract

Background:

Education decision makers are increasingly expected to use evidence to inform their actions. However, the majority of educational interventions have not yet been studied and it is challenging to produce high quality research evidence quickly enough to influence policy questions.

Aims and objectives:

We set out to gather evidence on the efficacy of reading resources implemented at 23 struggling elementary schools in a large, urban district in the US. The schools were at risk of closure by the state.

Methods:

For each reading resource, we searched for existing effectiveness studies and collected professional judgements by surveying practitioners. We also used an expert survey to collect judgements from three reading experts. We compared the ratings among experts and between practitioners and experts. We also compared practitioner and expert judgements to evidence summaries from research repositories.

Findings:

We found evidence summaries in research repositories for only five of 23 reading resources used in the 23 schools. Experts showed poor to good agreement on ten questions about each resource. Agreement between practitioners and experts was low with practitioners generally rating resources more positively than reading experts.

Discussion and conclusions:

Practitioners may be overly optimistic about the efficacy of educational materials while experts have difficulty assessing how well the materials serve an unfamiliar population. In the absence of rigorous evaluations of locally-implemented programmes, district decision makers can review the consistency of evidence collected from practitioners and experts, along with external research evidence, to inform actions towards supporting and guiding struggling schools.

Key messages

  • Rigorous research evidence is lacking for many of the programmes and practices implemented in schools.

  • Districts can produce internal evidence on the effectiveness of reading resources by surveying practitioners.

  • Expert ratings of the same resources are generally less optimistic about their efficacy.

  • Decision makers should consider consistency across sources of evidence and relevance to local context.

Introduction

Evidence use in education

Education decision makers are increasingly expected to use rigorous evidence to inform decisions about the programmes, practices and interventions implemented in schools (National Commission on Excellence in Education, 1983; No Child Left Behind Act of 2002; Cabinet Office, 2013; Sharples, 2013; British Educational Research Association, 2014; Every Student Succeeds Act [ESSA] of 2015; US Department of Education, 2016). What counts as evidence in education and to what extent it is used are topics that have been widely studied. Weiss (1977) and other scholars since (King and Pechman, 1984; Nutley et al, 2007; Tseng, 2012; Penuel et al, 2017) demonstrated that educational research findings have been used conceptually to influence decision makers’ understanding of decision problems, symbolically or politically as tools of persuasion to justify decisions already made, and instrumentally to directly guide and shape decision making. Researchers have also documented ‘imposed’ use of research due to government funding tied to the adoption of evidence-based programmes, and ‘process’ use of research which refers to what practitioners learn from participating in evidence production (Weiss et al, 2005; Tseng, 2012).

Equally well documented are the low incidence of research use to inform education practice (Penuel et al, 2017; Gamoran, 2018; Walker et al, 2019) and the array of obstacles preventing the use of research evidence in education decision making (Whitty, 2006; Oliver et al, 2014; Farley-Ripple et al, 2018). A chief culprit is the difficulty of producing high-quality research evidence quickly or decisively enough to influence policy questions (Resnick, 2015). Additionally, the majority of educational strategies employed in schools have not yet been evaluated (OECD, 2015), much less the multitude of curricula and instructional materials that embody particular practices. Even when evidence is available, it may not meet expected methodological standards (Malouf and Taymans, 2016) or point to a policy solution, resulting in solutions that often stray beyond the existing evidence into the realm of opinion (Hanushek, 2015). Despite initial enthusiasm for the increased rigour of educational research over the past 20 years, concerns abound that evidence prioritising internal validity, for example, from randomised controlled trials (RCTs), has too often failed to generalise across school contexts and student populations (Cartwright and Hardie, 2012; Joyce and Cartwright, 2020). The perceived lack of applicability of RCT findings to local situations leads to low credibility of such evidence among educators (Finnigan et al, 2013; Mills et al, 2020).

The limited use of externally-produced rigorous evidence has led to debate among the research community about how to enhance current research designs or produce more locally-relevant research. Defendants of RCTs argue for ways in which to improve their value and generalisability. For example, Kaplan et al (2020) recommend incorporating elements of qualitative research to help shift the goal of RCTs from ‘… generalising findings to the population to generalising findings to a robust theoretical understanding of the contextualised causal mechanisms underlying the phenomenon’. A more common call is to find a balance between rigour and ‘timely relevance’ (Feuer, 2015: 99).

To address concerns about the relevance of research to local decisions, many researchers emphasise the importance of collaboration between researchers and practitioners. Penuel et al (2015) advocate for ‘boundary practices’, activities characterised as hybrids between the practices of researchers and practitioners. Penuel and Farrell (2017) assert that participation in the research process is important for utilisation of evaluations, and suggest that conducting local evaluations of programmes may produce results that are more timely, credible and responsive to the needs of school district leaders. In turn, this increases the likelihood that the decision makers will act upon the findings. Irrespective of the supply of relevant research, Farrell and Coburn (2017) argue that districts’ capacity to use research evidence depends on organisational features which include prior knowledge and expertise, communication pathways, and strategic knowledge leadership.

In addition to research evidence, other forms of information may be as or more important to education practitioners. Coburn and Turner (2011a; 2011b), Daly et al (2014); Farrell (2014); and Marsh and Farrell (2015) focus on use of data as evidence to inform action in schools. Finnigan et al (2013) found that school personnel mostly used data such as standardised test scores and grades rather than research as evidence for decision making. Local values, interests and practical circumstances may be of equal or greater importance than objective evidence (Edwards et al, 1975; Weiss, 1995; Honig and Coburn, 2008; Asen et al, 2013; Tseng and Nutley, 2014; Hollands et al, 2019; Huguet et al, 2021). Boaz et al (2019) add issues of cost, implementation and stakeholder engagement to the list of considerations.

Spencer et al (2012) argue that evidence-based practice in education requires professional judgements in addition to research evidence and a consideration of client values and context. Feuer (2015) similarly stresses the importance of practitioner knowledge, suggesting that experiential evidence, that is, professional practice and experience, is ‘essential to the construction of useful theory’ (Feuer, 2015: 97). However, the extent to which professional educator judgements regarding the effectiveness of educational programmes align with rigorous evidence produced by formal research studies does not appear to have been studied. Expert judgements can also provide evidence on effectiveness of instructional programmes, practices and personnel (Arreola, 2000; Berk, 2005, Berk and Theall, 2006) and contribute to education decision making (Clayton, 1997). Similar to practitioner judgements, it has not been established whether expert judgements align well with existing research evidence in education, or even if experts’ assessments agree with each other or with practitioners’ judgements. In the context of a school district’s efforts to use evidence to guide its response to state pressure to improve reading performance in its underperforming elementary schools, the present study investigates whether practitioner and expert judgements can serve as reliable sources of evidence on the effectiveness of reading resources, and how they compare with existing research evidence.

Setting

In 2017–18, a large, urban school district in the southern US (‘SUD’) was required by the state education agency to improve reading achievement in 23 elementary schools or risk forced closure of these schools. SUD served over 200,000 students, of which 86% were African-American or Hispanic. Almost half the students were English language learners (ELLs) and 79% were economically disadvantaged.

At the 23 struggling schools, 98% of the students were African-American or Hispanic, 32% were ELLs, 92% were economically disadvantaged, 75% were deemed at risk of dropping out of school, and only 24% met grade level in state standardised reading tests. Among the 20 school principals for whom we were able to obtain demographic information, 75% were female, 60% were Black, 30% were Hispanic, and 10% were White. They averaged 14.8 years of experience and 60% had earned a Master’s or higher-level degree. A Teacher Development Specialist (TDS) worked at each school (one school had two TDSs) observing teachers’ instructional practices, providing feedback, and coaching them on how to increase student achievement. Among the 15 TDSs for whom we were able to obtain demographic information, 93% were female, 60% were Black, 20% were Hispanic, 13% were White, and one was Other. They averaged 15.8 years of experience and 47% had earned a Master’s degree. SUD is a decentralised district: schools may implement any resources from the district’s list of approved vendors. The district office provides support for certain programmes and may encourage school uptake of specific resources by purchasing district-wide services or offering to pay for licenses, materials or professional development.

Research production and use in SUD

In 2017–18, the SUD research department consisted of 22 researchers. All have Master’s degrees and 14 have doctorates. The department conducts approximately 20 programme evaluations each year to produce local evidence on district initiatives, for example, on bilingual education, early childhood education, migrant education, state compensatory education, college readiness and success, professional development models, and the use of specific programmes or resources, including instructional software, to improve achievement. Some evaluations are mandated by state and federal funding while others are initiated by curricular departments to investigate locally relevant issues. In other cases, community partners that contribute resources and funds request evaluations to assess the benefits resulting from their investments. These evaluations are publicly available on SUD’s website and some are in the Educational Resources Information Center’s repository of educational research. In addition to preparing written reports, the researchers meet with those requesting the evaluations to present and discuss the results and how they can be applied to modify programme implementation or instruction. Results are also shared with the superintendent and, where relevant, with organisation boards and local universities. Monthly principals’ meetings and school board meetings, which are attended by community partners, are used as opportunities to share and discuss student data and evidence-based practices. SUD researchers conduct literature reviews on existing programmes and practices to provide external evidence to inform SUD’s own actions. In addition, SUD collaborates with external researchers to engage in a variety of research, development and evaluation activities such as the present study.

In order to provide timely guidance and support to the 23 elementary schools, SUD’s Office of Elementary Curriculum and Development immediately sought to identify which reading resources (reading programmes, assessments, online tools, book collections, professional development supports) were available at each school, and to assess their effectiveness at improving student reading proficiency. Our research-practice team explored a variety of options for procuring suitable evidence to inform SUD’s actions. Because SUD needed to act swiftly to avert school closures, it was not feasible to plan and execute multiple experiments or quasi-experiments. We considered quantitative analyses using SUD’s administrative student data, but two obstacles arose. First, each school used multiple reading resources with the same students, making it hard to isolate the contribution of any one resource. We examined the possibility of applying value-added regression models to investigate the relative contribution of each resource to student reading performance using an approach similar to Betts et al (2005) and Shand et al (2020). This plan met with the second obstacle which is that SUD’s schools do not routinely document which students engage with which resources.

Current study

Given the inability to conduct experimental or quasi-experimental studies to evaluate the schools’ reading resources, we explored three strategies to collect evidence on their effectiveness. We developed two surveys to elicit judgements on the effectiveness of the resources for improving reading proficiency, one from professional educators in the schools and one from external reading experts. We also searched for existing external research evidence on the effectiveness of each resource in reputable repositories of education research (Donaldson et al, 2009). We compared the experts’ judgements with each other to determine the extent to which they agreed in their assessments of each reading resource. Subsequently, we compared the expert judgements with the practitioner judgements to evaluate the consistency of assessments between the two groups. We also compared the expert and practitioner judgements with existing evidence from research repositories. We addressed three research questions:

  1. To what extent do reading experts agree with each other when rating the effectiveness of a set of reading resources?

  2. To what extent do practitioners’ ratings of reading resources implemented in their schools coincide with ratings provided by reading experts?

  3. Do ratings from experts and practitioners align with evidence summaries from external research repositories?

Methods

Searching for existing studies in research repositories

For each reading resource, we searched two US-based repositories of research on educational programmes to identify evidence summaries and studies on its effectiveness for improving reading: the What Works Clearinghouse (WWC) and Evidence for ESSA (E4E). We also searched the evidence summaries provided by the Education Endowment Fund (EEF) in the UK.

Collecting practitioners’ professional judgements

We designed a Qualtrics survey in which the TDS at each school indicated which reading resources were being used at their school. Because TDSs routinely use student assessment data to guide their coaching strategies, they were considered to be a suitable source of judgements on the effectiveness of the resources. For each resource used, the TDS answered the following prompt: ‘Please indicate how effective you believe [resource] is for improving student reading proficiency’. Response choices were: ‘hinders reading proficiency’, ‘makes no difference’, ‘improves reading proficiency a little’, ‘improves reading proficiency a lot’, or ‘not sure’.

We created individual links to the survey for each of the 23 schools and our SUD liaison emailed the links to the relevant TDSs (23 surveys total) at the end of November 2017. Reminders were sent several times to those who did not respond or who only partially completed the survey. The TDSs were also invited to complete the survey during a routine meeting at the district office in January 2018. As a result, we collected responses from TDSs at all 23 schools, which we used to create a master list of resources. A total of 23 reading resources were being used at one or more of the 23 schools. We documented how many schools used each resource and how it was rated by each TDS. We did not calculate ICCs (intra-class correlations) for the practitioners because they were each exposed to a different instantiation of the programme so a difference in rating could reflect different approaches or different levels of implementation fidelity at each school, not simply differences in judgement of the same intended programme.

Collecting expert judgements

We recruited three reading experts and, using an expert survey (Schiefelbein et al, 1998), elicited their professional opinions regarding the effectiveness of the subset of reading resources which were being used as instructional programmes. The goal in recruiting reading experts was to find individuals with expertise in effective reading instruction, based in both research and practice. The reading experts were a convenience sample of professionals familiar with both research and practice in teaching reading, but not necessarily with knowledge of the same resources. The experts were selected on the basis of having a doctorate in literacy, multiple years of experience in K-12 literacy teaching, and multiple years of experience in university-based teacher education training teachers of reading. In our judgement, these criteria are similar to those that a school district might look for in university-based experts. The first expert is the second author who had collaborated previously with other members of the author team. The other two experts were in the first expert’s professional network and were invited to participate by the first expert. The first expert is an Associate Professor of Literacy Education who studies the language and literacy development of students from linguistically diverse backgrounds, especially in urban schools, and who was formerly a middle-school teacher. A second expert is a Clinical Professor of Literacy Education who studies the effective instruction of reading in the elementary years, especially in urban settings, and who taught for over 25 years in high-poverty, public elementary and middle-school classrooms. The third expert is a former Professor of Literacy and Special Education and charter member of the Reading League who taught Special Education in public schools for 33 years.

The subset of ten resources evaluated by the experts, listed in Table 1, were those used by SUD as core reading programmes, web-based supplemental literacy programmes, guided reading programmes, or intervention programmes. We did not ask the experts to rate the other 13 resources. Eleven of these were not instructional programmes: four were assessments (Renaissance 360, Developmental Reading Assessment 2, Fountas and Pinnell Assessment Kits, Evaluación del Desarrollo de la Lectura); five were book collections (Scholastic Classroom Libraries, Scholastic Read Aloud Libraries, Scholastic Extended Bookroom, MyON, Storia); and two were professional development supports (Neuhaus Education and Writers in the Schools). Finally, two resources would have qualified as instructional programmes but were Spanish-only (Tesoros and Benchmark Spanish Leveled Libraries) and, because the experts were not fluent Spanish speakers, they could not fully evaluate their quality.

Table 1:

Reading resources rated by experts

Programme (publisher)Type of reading resource (as used in SUD)# of schools using this resource
Corrective Reading (McGraw-Hill)Guided reading/intervention programme: small group remedial8
Frog Street (Frog Street Press)Core reading programme: bilingual14
Imagine Language and Literacy (Imagine Learning)Web-based supplemental literacy programme22
Istation Reading (Istation)Web-based supplemental literacy programme3
Leveled Bookroom (Scholastic, Inc.)Guided reading/intervention programme: small group non-remedial instruction23
LitCamp (Scholastic, Inc.)Guided reading/intervention programme: small group non-remedial instruction6
Leveled Literacy Intervention (LLI) (Heinemann)Guided reading/intervention programme: small group remedial instruction1
Reading A-Z (Lazel)Guided reading/intervention programme: small group non-remedial reading2
Reading Mastery (McGraw-Hill)Guided reading/intervention programme: small group remedial8
Reading Street (Pearson Education, Inc.)Core reading programme20

For each of the ten resources, we conducted an extensive internet search to collect the information outlined in Table 2. In addition to searching the three aforementioned research repositories, we entered the name of the resource in Google’s search engine, in Google Scholar, and in the vendor website search functions. We compiled the information in a database and made it available to the experts via a Google Sheet and a shared Google Drive folder of research studies and other materials. We also provided contextual details about SUD including student demographics and performance, and how each resource was used (for example, as a guided reading programme or intervention).

Table 2:

Information about each reading resource provided to experts

Information provided
Name of resource and publisher
URLs for public websites with general information about each resource
A classification of type of resource: core reading programmes; guided reading/intervention programmes: small group non-remedial; guided reading/intervention programmes: small group remedial; web-based supplemental literacy programmes
Description of resource
Description of activities (what the resource requires students/teachers to do in the classroom)
Length of time resource has been in existence
Dosage (how often students use the resource; for how much time on each occasion; over how many weeks, months, or years)
Scope (aim of the programme and what skills students are taught)
Sequence (in what order these skills are taught)
Target audience: general Education/ELLs
Language (English/Spanish/bilingual)
Grades/levels served
Information and URLs to materials that address alignment of the resource with Common Core Standards
Information on whether the materials align with other standards
All reports/studies/evaluations/reviews found by searching the WWC, E4E, EEF, Google Scholar, Google, and the vendor’s website using the name of each resource.

Questions for the experts, listed in Table 3, were embedded in a Qualtrics survey that we designed. The questions elicited ten ratings on each of ten resources. Experts completed their review of the compiled materials and their survey responses independently within a few weeks of each other. We compared the ratings across the three experts. We also compared the experts’ responses to a similarly-worded question about expected effectiveness of each resource for students in SUD with the practitioners’ effectiveness ratings for the same resource. We expected these ratings would be correlated.

Table 3:

Intra-class correlation values for expert survey questions

Question answered by experts for each reading resourceICC valueInterpretation of ICCP-value (H_0 = 0, H_1 >0)
How clearly specified is the theory of action?0.400Fair0.170
Given the targets of the programme (for example, student population, grade level), how well aligned is the content with existing research on effective reading instruction?0.496Fair0.103
How strong is the evidence that [resource] is effective at improving reading skills?0.740Good0.010
How much implementation support appears to be available from the vendor for teachers using [resource] under typical conditions?–1.066Poor0.867
How effective do you think [resource] will be at improving reading proficiency for the target students in typical US elementary schools (that is, average demographics, average teacher qualifications and average school funding) if implemented as the publisher intends?0.420Fair0.150
How much impact would you expect [resource] to have on each reading component?*
Alphabetics0.680Good0.020
Vocabulary0.400Fair0.170
Fluency0.740Good0.010
Comprehension0.577Fair0.058
Based on the school profiles and student demographics provided, how effective do you think [resource] would be at improving reading proficiency for target students in under-served and underperforming SUD schools if implemented as the publisher intends?–0.070Poor0.520

* Response choices were: Large negative, Medium negative, Small negative, No effect, Small positive, Medium positive, Large positive.

Assessing inter-rater reliability among the experts

Response choices to each of the questions in the expert survey are ordinal, that is, they yield categorical data where the variables have ordered categories but the distances between the categories is unknown (Agresti, 2013). To assess the inter-rater reliability (IRR) among the experts, we calculated the ICC between the ten programme ratings provided by each expert for each item. Following Hallgren (2012), the ICC function is specified by three parameters. First, the ICC function depends on how coders are selected for the study: in a one-way model, a different set of coders is randomly selected from a larger population of coders for each subject (a subject is equivalent to a reading resource in our study). In a two-way model, all subjects are rated by all raters. The second determinant of the ICC function is whether agreement needs to be absolute or relative, that is, whether it is important for raters to provide scores that are similar in absolute value (agreement) or in rank order (consistency). The last determinant is whether ratings are provided by a single coder or averaged across multiple coders. We used a two-way model whereby all resources were rated by all expert raters and we estimated the consistency of ratings across coders. Higher ICC values signify greater IRR (Hallgren): an ICC estimate of 1 implies perfect agreement among raters while 0 suggests random agreement. Negative ICC values reflect systematic disagreement. ICCs of less than −1 may arise when there are three or more coders. Cicchetti (1994) provides benchmarks for ICC scores indicating that IRR is poor for ICC values below 0.4, fair for values of 0.4–0.59, good for values of 0.60–0.74, and excellent for values of 0.75–1.0.

Correlation between experts’ ratings and practitioners’ ratings

To allow for a calculation of the correlation between experts’ ratings on the effectiveness of the resources for improving reading proficiency among SUD students and practitioners’ ratings, we changed the categorical rating of effectiveness to a continuous variable by assigning values to the each of the categories as follows: hinders reading proficiency = 1; makes no difference = 2; not sure = 3; improves reading proficiency a little = 4; improves reading proficiency a lot = 5. This assumes that the distance between two adjacent categories is constant and that ‘not sure’ is neither a negative assessment nor a positive one. We used the values to calculate an average effectiveness rating for each resource across practitioners and separately across experts.

Results

Evidence from research repositories

Ten of the 23 reading resources being used in the SUD elementary schools were listed in WWC or in E4E, but qualifying studies were available for only half of these. WWC only includes studies that would qualify for the top two tiers of ESSA evidence (experimental or quasi-experimental studies) while E4E includes items that qualify for Tier 3 (correlational studies). Four resources were supported by evidence of effectiveness at improving one or more aspects of reading. For one item, the evidence was neither significantly positive or negative. All five resources appearing in at least one of the repositories with associated evidence were among the ten items evaluated by both practitioners and reading experts (see Table 4). No information on these US-developed literacy programmes was found in EEF’s evidence summaries.

Table 4:

Comparison of practitioners and experts’ ratings on the effectiveness of reading resources

ProgrammeEvidence from research repositoryNumber of ratersHinders reading proficiencyMakes no differenceNot sureImproves reading proficiency a littleImproves reading proficiency a lot
PEPEPEPEPEPE
Corrective ReadingWWC: potentially positive effects on alphabetics and fluency; no discernible effects on comprehension830%0%0%0%38%0%0%67%62%33%
E4E: strong +0.06
Frog StreetWWC: no information1430%0%0%33%36%33%36%33%29%0%
E4E: no studies met inclusion requirements
Imagine Language and LiteracyWWC: no information2230%33%0%0%45%33%27%0%27%33%
E4E: promising +0.18
Istation ReadingWWC: no information330%0%0%0%0%33%0%67%100%0%
E4E: no studies met inclusion requirements
Leveled BookroomWWC: no information2330%0%0%0%4%67%4%0%91%33%
E4E: no information
LitCampWWC: no information630%0%0%33%33%67%50%0%17%0%
E4E: no studies met inclusion requirements
LLIWWC: positive effects on general reading achievement, potentially positive effects on reading fluency; no discernible effects on alphabetics130%33%0%0%0%33%0%33%100%0%
E4E: strong +0.13
Reading A-ZWWC: no information230%0%0%33%0%33%0%33%100%0%
E4E: no studies met inclusion requirements
Reading MasteryWWC: potentially positive effects on reading achievement for ELLs830%0%0%0%25%0%0%33%75%67%
E4E: no studies met inclusion requirements
Reading StreetWWC: no information2030%33%5%0%25%0%35%67%35%0%
E4E: qualifying studies found no significant positive results

Note: P = practitioner (TDS); E = reading expert

Practitioner ratings

Table 4 summarises the number of schools using each of the ten resources for which we compared expert and practitioner ratings, and the percentage of responding TDSs who rated each resource in a particular category of effectiveness. No practitioner indicated that a resource hindered reading proficiency, but one indicated that one resource made no difference. For all ten resources, more than 50% of the TDSs believe that the resource either improves reading proficiency a little or a lot. However, for six of the resources, at least a quarter of respondents were unsure as to whether it was making any difference.

Expert ratings

The inter-rater reliability among the three experts, based on ICC values shown in Table 3, ranged from good to poor on questions regarding effectiveness of the ten resources at improving reading proficiency. The ICC value was higher for the experts’ judgements about the effectiveness of the resources for the average US student than for students at SUD. The ICC values were also higher when experts were asked about specific components of reading (alphabetics, fluency, vocabulary, comprehension) than when asked about effectiveness at improving reading proficiency overall.

Comparison between ratings from experts and practitioners

Table 4 compares the percentage of practitioners and experts who rated each of the ten resources in each of the five response categories of perceived effectiveness at improving reading proficiency. While no practitioners indicated that a resource hinders proficiency, one expert did so for three resources. Only one resource was believed to make no difference to reading proficiency by the practitioners, but three resources were placed in this category by an expert. One or more practitioners were unsure about the effectiveness of seven resources. One or more experts were also unsure for seven resources, although only four resources overlapped between the two groups of raters. In general, the practitioners were more optimistic about the extent to which the resources improved reading proficiency. Specifically, for all ten resources, at least one practitioner indicated that the resource improved reading proficiency a lot. This was the case for only four resources based on the experts’ ratings.

Correlation between experts’ ratings and practitioners’ ratings

We found the correlation between expert and practitioner ratings to be low at 0.186. Because the experts expressed low confidence in their ratings of the resources for SUD students, we additionally calculated the correlation between the experts’ ratings on the effectiveness of these ten reading resources for the average US student and the practitioners’ ratings (which, by default, applied to SUD students). This correlation was slightly negative at −0.034. We also investigated the correlation between the scores of the experts and practitioners after dropping all ‘unsure’ ratings from both sets of raters. This produced a much higher correlation of 0.63 but, obviously, for fewer resources.

Comparison of expert and practitioner ratings with research repository evidence

Evidence summaries were only available for five of the ten resources rated by both experts and practitioners. For two of these, Corrective Reading and Reading Mastery, the majority of the practitioners and experts provided ratings that were positive and consistent with the evidence summaries. For two others, Imagine Language and Literacy and LLI, the majority of the practitioners provided ratings that were positive and consistent with the evidence summaries, but one expert rated the resource as hindering reading proficiency and one indicated ‘not sure’. For the fifth resource, Reading Street, the evidence summary indicated no significant positive or negative results, but the majority of both practitioners and experts rated it as improving reading proficiency.

Discussion

Our study demonstrates the difficulty of meeting demands to use evidence in educational decisions, especially when time is of the essence. Of 23 reading resources used in the elementary schools of a large, US public school district, the two best-recognised US repositories of education research evidence provided information on only five. This finding is consistent with Chang et al’s (2021) finding that there is either no or limited evidence on the effectiveness of many items in which schools invest Title I federal funds. Furthermore, the evidence summaries provided by the repositories are based on only one or very few studies, and often conducted with populations dissimilar to SUD’s. This may help explain why practitioners are failing to select programmes and practices for which positive evidence is available in these repositories. One possible reason for the lack of studies on widely-used reading programmes is that research funders such as the US’s Institute of Education Sciences are more likely to fund studies that build theory on effective instruction using researcher-developed curricula that are not widely used and may never be published, as opposed to studies on curricula already in widespread use but for which the theory of action is less clear.

Neither is internally-produced evidence easy to come by. Accurate data on student participation in classroom-based programmes is not routinely collected by districts and schools, thereby limiting quantitative evaluations using existing data. Without the luxury of time or the data needed to conduct quantitative evaluations of SUD’s reading resources, we sought more easily available evidence on their effectiveness – practitioner and expert judgements – with the expectation that they would corroborate each other. Practitioners provided ratings on 23 reading resources, basing their judgements on their direct experiences observing the resources being used with students in their schools and their access to student assessment data. For the few resources for which external evidence was available, none of the practitioners’ judgements were contradicted by published studies. Although the reading experts were each given access to the same set of information about each of the ten reading resources, they showed varying levels of agreement in their ratings. This may be partly attributable to different starting levels of familiarity with the resources, in particular whether the expert had ever observed them being implemented in schools. The experts demonstrated fair agreement about the overall effects of each resource on reading proficiency for average US students, and fair to good agreement on individual components of reading. For two resources, the expert ratings did not comport with positive external evidence available in the research repositories. This may have been due to the fact that both repositories based their evidence summaries on only two studies, and these were conducted in contexts unlike SUD’s with student populations that were substantially less challenged.

The reading experts showed poor agreement on the overall effects of each resource on reading proficiency for students at SUD. The inconsistency in ratings for the SUD context may be due to experts focusing on different factors for the SUD population based on their experience and expertise. For example, one expert may have focused more on the percentage of ELLs, while another may have focused more on the percentage of minorities or economically-disadvantaged students. The low overall correlation between practitioner and expert ratings appears to be mostly attributable to practitioners rating the effectiveness of reading resources more favourably than experts. This optimism produced relatively limited variation in the practitioners’ ratings which, in turn, may have attenuated the correlation between practitioner and expert ratings. In sum, it appears that practitioners may be overly optimistic about the efficacy of educational materials, while experts have difficulty assessing how well the materials will serve a population with whom they are unfamiliar.

Limitations

Some of the reading resources evaluated were used in few schools so the number of practitioner ratings for these resources was correspondingly small. Ratings from more practitioners would provide a better picture of how well each resource was perceived by a range of practitioners. While these respondents were asked to comment on the resources as implemented in their schools, a fair number indicated that they were not sure whether the resource had any impact on reading proficiency. It is unclear whether ‘not sure’ was a more diplomatic alternative to ‘makes no difference’ or whether the respondents genuinely felt they had inadequate knowledge of a resource’s impact. In addition, a larger number of experts may have yielded more consistent expert judgements: Schiefelbein et al (1998) recommend as many as 20 or 25 participants for expert surveys. We lacked the resources to recruit more than three.

Applying the results to decision making

Mills et al (2020) recommend that, to increase the use of research by practitioners, researchers should address how to implement their research findings. Accordingly, we demonstrate here how education decision makers can use the results of this study to guide schools towards or away from specific reading resources. We expect a similar approach could be applied to other types of resources, given parallel data. First, for each resource, the decision makers can review the consistency across the three sources of evidence in terms of reported effectiveness at improving reading outcomes. We also recommend they consider how many of their schools are using the resource to gauge how many might need support in implementing the resource more effectively, eliminating it altogether, or switching to a different resource. A further consideration in SUD’s situation would be the feasibility of implementing promising reading resources in additional schools quickly enough to help students within the desired timeframe.

We model this approach with four examples. Corrective Reading was used in one third of the 23 schools and the majority of TDSs indicated that it improves reading proficiency a lot. All three experts indicated that the programme improves reading proficiency. The external evidence in both repositories is positive, albeit based on only one study. In sum, the evidence is reasonably consistent and positive across sources, suggesting that this resource is worth considering for adoption by other schools. However, a substantial amount of training is needed for teachers to implement Corrective Reading. This might be feasible longer-term, but perhaps not in time to help the schools immediately. Reading Mastery was also used at eight schools. The TDSs and experts mostly rated the programme positively. WWC holds one study of Reading Mastery demonstrating potentially positive effects for ELL reading achievement. Given fewer training needs and materials than Corrective Reading, it might be more feasible to adopt in additional schools. Reading Street, used in 20 schools, earned quite disparate practitioner ratings, suggesting uneven implementation that the district office could better support to improve fidelity. Istation was rated positively by all three schools using it and by two of the experts. Being a computer-based programme, it could be implemented swiftly in additional schools with digital devices and connectivity.

This type of approach in which multiple factors are considered simultaneously could be implemented more systematically using a decision-making framework based on cost-utility analysis (Hollands et al, 2019). Resources needed to implement each programme would be compared with its utility. Utility would be determined by including the various assessments of effectiveness in addition to other considerations such as feasibility of implementation or scaling, with each consideration weighted by importance to the SUD stakeholders. One advantage of this approach is that it allows decision makers to incorporate contradictory sources of evidence but assign different importance weights to each one.

Future directions

One interpretation of the low correlation found between expert and practitioner ratings is that they are providing valuable, complementary information, based on distinct sources about the materials’ efficacy. Although we cannot shed light on this hypothesis with the current data, future studies could consider an in-depth qualitative approach to better understand the bases for both groups’ evaluation process.

A possible future avenue for producing more rigorous local evidence on a timely basis would involve routinely collecting accurate data in schools on which students engage with which reading resources, in order to allow for regression analyses using value-added models. Betts et al (2005) and Shand et al (2020) demonstrate this approach for programmes or practices aiming to improve reading achievement. The information could be collected using districts’ student information systems.

Going forward, federal and state agencies could provide better information and support to help schools and districts use evidence-based practices. Boaz et al (2019) list a number of approaches for supporting evidence use in social programmes, all of which can be applied in education. Most of these involve capacity building for practitioners and policymakers to access and use evidence, often in collaboration with researchers. We additionally recommend that the US Department of Education or state education agencies survey schools to identify widely-used programmes and practices. The proposed School Pulse survey (Hart and Tromble, 2021; Schneider, 2020; 2021) could serve as a vehicle to gather the necessary information. Subsequently, they should commission studies on the effectiveness of the most common, being sure to sample a variety of student populations and contexts. We acknowledge that the value of such evidence may be limited to situations in which schools and districts are considering the adoption of new approaches. They are less likely to replace existing programmes regardless of effectiveness, due to resistance from entrenched interests (Moe, 2015) and the high costs of switching. For programmes found to be ineffective, it may be more pragmatic to identify modifications that current users can adopt to improve outcomes for students.

Conclusion

We conclude that there remains a substantial disconnect between the existing evidence base in education and the evidence needs of education decision makers (Kane, 2016; Malouf and Taymans, 2016). This applies to both external research and the data availability within education agencies that would allow for the production of local evidence. Even in a subject area as critical as reading, there is no rigorous evidence available to inform decision makers for most of the instructional resources being implemented. Until this can be rectified, a combination of professional and expert judgements regarding the effectiveness of the instructional resources, along with information about the feasibility of implementation and support needs, can provide some guidance to decision makers on where to concentrate their efforts to help students.

Research ethics statement

Institutional Review Board approval for this study was covered under protocol #17–372 from Teachers College, Columbia University, dated July 3, 2017. In addition, approval was granted by the Department of Research and Accountability at SUD on June 26, 2017.

Funding

This work was supported by the William T. Grant Foundation under Grant number 187095 awarded to Teachers College, Columbia University.

Contributor statement

FMH, YP, MJK, VRH, and MME conceptualised the study and, with AM, designed the data collection instruments. VRH collected data from the school district. MJK served as one of the reading experts and coordinated participation by the other two experts. LH investigated existing evidence and collected and prepared the materials for the experts. YP, YW, and AM analysed the data. FMH and YP drafted the initial manuscript. FMH, MJK and VRH made revisions. All Authors commented on the manuscript.

Conflict of interest

The authors declare that there is no conflict of interest.

References

  • Agresti, A. (2013) Categorical Data Analysis, 3rd edn, Hoboken, NJ: John Wiley & Sons.

  • Arreola, R.A. (2000) Developing A Comprehensive Faculty Evaluation System, 2nd edn, Bolton, MA: Anker Publishing Company, Inc.

  • Asen, R., Gurke, D., Conners, P., Solomon, R. and Gumm, E. (2013) Research evidence and school board deliberations: lessons from three Wisconsin school districts, Educational Policy, 27(1): 3363. doi: 10.1177/0895904811429291

    • Search Google Scholar
    • Export Citation
  • Berk, R.A. (2005) Survey of 12 strategies to measure teaching effectiveness, International Journal of Teaching and Learning in Higher Education, 17(1): 4862.

    • Search Google Scholar
    • Export Citation
  • Berk, R.A. and Theall, M. (2006) Thirteen Strategies to Measure College Teaching: A Consumer’s Guide to Rating Scale Construction, Assessment, and Decision Making for Faculty, Administrators, and Clinicians, Sterling, VA: Stylus Publishing, LLC.

    • Search Google Scholar
    • Export Citation
  • Betts, J.R., Zau, A. and King, K. (2005) From Blueprint to Reality: San Diego’s Education Reforms, San Francisco, CA: Public Policy Institute of California.

    • Search Google Scholar
    • Export Citation
  • Boaz, A., Davies, H., Fraser, A. and Nutley, S. (2019) What Works Now: Evidence-informed Policy and Practice, Bristol: Policy Press.

  • British Educational Research Association (2014) The role of research in teacher education: reviewing the evidence, Interim report of the BERA-RSA Inquiry. British Educational Research Association, https://www.thersa.org/globalassets/pdfs/reports/bera-rsa-interim-report.pdf.

    • Search Google Scholar
    • Export Citation
  • Cabinet Office (2013) What Works: Evidence Centres for Social Policy, London: Cabinet Office.

  • Cartwright, N. and Hardie, J. (2012) Evidence-based Policy: A Practical Guide to Doing it Better, Oxford,: Oxford University Press.

  • Chang, Y., Hollands, F. M., Holmes, V.R., Shand, R., Evans, P., Blodgett, R., Wang, Y. and Head, L. (2021) Challenges to finding evidence for ESSA: a case study of Southern Urban District, Paper presented at the Annual Meeting of the American Educational Research Association.

    • Search Google Scholar
    • Export Citation
  • Cicchetti, D.V. (1994) Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology, Psychological Assessment, 6(4): 28489. doi: 10.1037/1040-3590.6.4.284

    • Search Google Scholar
    • Export Citation
  • Clayton, M.J. (1997) Delphi: a technique to harness expert opinion for critical decision‐making tasks in education, Educational Psychology, 17(4): 37386. doi: 10.1080/0144341970170401

    • Search Google Scholar
    • Export Citation
  • Coburn, C.E. and Turner, E.O. (2011a) Research on data use: a framework and analysis, Measurement: Interdisciplinary Research and Perspective, 9(4): 173206. doi: 10.1080/15366367.2011.626729

    • Search Google Scholar
    • Export Citation
  • Coburn, C.E. and Turner, E.O. (2011b) Putting the ‘use’ back in data use: an outsider’s contribution to the measurement community’s conversation about data, Measurement: Interdisciplinary Research and Perspective, 9(4), 22734. doi: 10.1080/15366367.2011.634653

    • Search Google Scholar
    • Export Citation
  • Daly, A.J., Finnigan, K.S., Jordan, S., Moolenaar, N.M. and Che, J. (2014) Misalignment and perverse incentives: examining the politics of district leaders as brokers in the use of research evidence, Educational Policy, 28(2): 14574. doi: 10.1177/0895904813513149

    • Search Google Scholar
    • Export Citation
  • Donaldson, S.I., Christie, C.A. and Mark, M.M. (eds) (2009) What Counts as Credible Evidence in Applied Research and Evaluation Practice?, Thousand Oaks, CA: Sage.

    • Search Google Scholar
    • Export Citation
  • Edwards, W., Guttentag, M. and Snapper, K. (1975) A decision theoretic approach to evaluation research, in E. Struening and M. Guttentag (eds) Handbook of Evaluation Research, Vol 1, Thousand Oaks, CA: Sage, pp 13981.

    • Search Google Scholar
    • Export Citation
  • Farley-Ripple, E., May, H., Karpyn, A., Tilley, K. and McDonough, K. (2018) Rethinking connections between research and practice in education: a conceptual framework, Educational Researcher, 47(4): 23545. doi: 10.3102/0013189X18761042

    • Search Google Scholar
    • Export Citation
  • Farrell, C.C. (2014) Designing school systems to encourage data use and instructional improvement: a comparison of school districts and charter management organizations, Educational Administration Quarterly, 51(3): 43871. doi: 10.1177/0013161X14539806

    • Search Google Scholar
    • Export Citation
  • Farrell, C.C. and Coburn, C.E. (2017) Absorptive capacity: a conceptual framework for understanding district central office learning, Journal of Educational Change, 18(2): 13559. doi: 10.1007/s10833-016-9291-7

    • Search Google Scholar
    • Export Citation
  • Feuer, M.J. (2015) Evidence and advocacy, in M.J. Feuer, A.I., Berman and R.C. Atkinson (eds) Past as Prologue: The National Academy of Education at 50. Members Reflect, Washington, DC: National Academy of Education, pp 95101.

    • Search Google Scholar
    • Export Citation
  • Finnigan, K.S., Daly, A.J. and Che, J. (2013) Systemwide reform in districts under pressure: the role of social networks in defining, acquiring, using, and diffusing research evidence, Journal of Educational Administration, 51(4): 47697. doi: 10.1108/09578231311325668

    • Search Google Scholar
    • Export Citation
  • Gamoran, A. (2018) Evidence-based policy in the real world: a cautionary view, Annals of the American Academy of Political and Social Science, 678(1):18091. doi: 10.1177/0002716218770138

    • Search Google Scholar
    • Export Citation
  • Hallgren, K.A. (2012) Computing inter-rater reliability for observational data: an overview and tutorial, Tutorials in Quantitative Methods for Psychology, 8(1): 2334. doi: 10.20982/tqmp.08.1.p023

    • Search Google Scholar
    • Export Citation
  • Hanushek, E.A. (2015) Education’s double helix, in M.J. Feuer, A.I., Berman and R.C. Atkinson (eds) Past as Prologue: The National Academy of Education at 50. Members Reflect, Washington, DC: National Academy of Education, pp 30912.

    • Search Google Scholar
    • Export Citation
  • Hart, N. and Tromble, K. (2021) Data must lead the way to reopening our schools, 9 February, https://www.nextgov.com/ideas/2021/02/data-must-lead-way-reopening-our-schools/171947/.

    • Search Google Scholar
    • Export Citation
  • Hollands, F., Pan, Y. and Escueta, M. (2019) What is the potential for applying cost-utility analysis to facilitate evidence-based decision making in schools? Educational Researcher, 48(5): 28795. doi: 10.3102/0013189X19852101

    • Search Google Scholar
    • Export Citation
  • Honig, M.I. and Coburn, C. (2008) Evidence-based decision making in school district central offices: toward a policy and research agenda, Educational Policy, 22(4): 578608. doi: 10.1177/0895904807307067

    • Search Google Scholar
    • Export Citation
  • Huguet, A., Coburn, C.E., Farrell, C.C., Kim, D.H. and Allen, A.R. (2021) Constraints, values, and information: how leaders in one district justify their positions during instructional decision making, American Educational Research Journal, 58(4): 71047. doi: 10.3102/0002831221993824

    • Search Google Scholar
    • Export Citation
  • Joyce, K.E. and Cartwright, N. (2020) Bridging the gap between research and practice: predicting what will work locally, American Educational Research Journal, 57(3): 104582. doi: 10.3102/0002831219866687

    • Search Google Scholar
    • Export Citation
  • Kane, T.J. (2016) Connecting to practice: how we can put education research to work, Education Next, 16(2): 8088.

  • Kaplan, A., Cromley, J., Perez, T., Dai, T., Mara, K. and Balsai, M. (2020) The role of context in educational RCT findings: a call to redefine ‘evidence-based practice’, Educational Researcher, 49(4): 28588. doi: 10.3102/0013189X20921862

    • Search Google Scholar
    • Export Citation
  • King, J.A. and Pechman, E.M. (1984) Pinning a wave to the shore: conceptualizing evaluation use in school systems, Educational Evaluation and Policy Analysis, 6(3): 24151. doi: 10.3102/01623737006003241

    • Search Google Scholar
    • Export Citation
  • Malouf, D.B. and Taymans, J.M. (2016) Anatomy of an evidence base, Educational Researcher, 45(8): 45459. doi: 10.3102/0013189X16678417

    • Search Google Scholar
    • Export Citation
  • Marsh, J.A. and Farrell, C.C. (2015) How leaders can support teachers with data-driven decision-making: a framework for understanding capacity building, Educational Management Administration and Leadership, 43(2): 26989. doi: 10.1177/1741143214537229

    • Search Google Scholar
    • Export Citation
  • Mills, K.J., Lawlor, J.A., Neal, J.W., Neal, Z.P. and McAlindon, K. (2020) What is research? Educators’ conceptions and alignment with United States federal policies, Evidence & Policy, 16(3): 33758.

    • Search Google Scholar
    • Export Citation
  • Moe, T.M. (2015) Vested interests and political institutions, Political Science Quarterly, 130(2): 277318. doi: 10.1002/polq.12321

  • National Commission on Excellence in Education (1983) A nation at risk: the imperative for educational reform, Elementary School Journal, 84(2): 11330. doi: 10.1086/461348

    • Search Google Scholar
    • Export Citation
  • Nutley, S.M., Walter, I. and Davies, H. (2007) Using Evidence: How Research Can Inform Public Services, Bristol: Policy Press.

  • OECD (Organisation for Economic Co-operation and Development) (2015) Education Policy Outlook 2015: Making Reforms Happen, Paris: OECD Publishing.

    • Search Google Scholar
    • Export Citation
  • Oliver, K., Innvar, S., Lorenc, T., Woodman, J. and Thomas, J. (2014) A systematic review of barriers to and facilitators of the use of evidence by policymakers, BMC Health Services Research, 14(1): 112. doi: 10.1186/1472-6963-14-1

    • Search Google Scholar
    • Export Citation
  • Penuel, W.R. and Farrell, C. (2017) Research-practice partnerships and ESSA: a learning agenda for the coming decade, in E. Quintero-Corrall (ed) The Social Side of Reform, Cambridge, MA: Harvard Education Press.

    • Search Google Scholar
    • Export Citation
  • Penuel, W.R., Allen, A.R., Coburn, C.E. and Farrell, C. (2015) Conceptualizing research-practice partnerships as joint work at boundaries, Journal of Education for Students Placed at Risk, 20(1–2): 18297. doi: 10.1080/10824669.2014.988334

    • Search Google Scholar
    • Export Citation
  • Penuel, W.R., Briggs, D.C., Davidson, K.L., Herlihy, C., Sherer, D., Hill, H.C., Farrell, C. and Allen, A.R. (2017) How school and district leaders access, perceive, and use research, AERA Open, 3(2): 117.

    • Search Google Scholar
    • Export Citation
  • Resnick, L. (2015) The National Academy of Education’s role in bridging research and practice, in M.J. Feuer, A.I., Berman and R.C. Atkinson (eds) Past as Prologue: The National Academy of Education at 50. Members Reflect, Washington, DC: National Academy of Education, pp 32528.

    • Search Google Scholar
    • Export Citation
  • Schiefelbein, E., Wolff, L. and Schiefelbein, P. (1998) Cost-effectiveness of education policies in Latin America: a survey of expert opinion, Inter-American Development Bank, https://publications.iadb.org/publications/english/document/Cost-Effectiveness-of-Education-Policies-in-Latin-America-A-Survey-of-Expert-Opinion.pdf.

    • Search Google Scholar
    • Export Citation
  • Schneider, M. (2020) Introducing operation reverse the loss, 26 October, https://ies.ed.gov/director/remarks/10-26-2020.asp.

  • Schneider, M. (2021) The road to learning recovery, 16 March, https://ies.ed.gov/director/remarks/3-16-2021.asp.

  • Shand, R., Pan, Y., Leach, S., Nayyer-Qureshi, S., Hollands, F., Yan, B., Dossett, D., Wang, Y., Head, L. and Chang, F. (2020) Program evaluation with administrative data: new applications, promise, and challenges for value-added models, Poster session at AEFP Conference, Fort Worth, TX.

    • Search Google Scholar
    • Export Citation
  • Sharples, J. (2013) Evidence for the Frontline, London: The Alliance for Useful Evidence.

  • Spencer, T.D., Detrich, R. and Slocum, T.A. (2012). Evidence-based practice: a framework for making effective decisions, Education and Treatment of Children, 35(2): 12751. doi: 10.1353/etc.2012.0013

    • Search Google Scholar
    • Export Citation
  • Tseng, V. (2012) The uses of research in policy and practice and commentaries, Social Policy Report, 26(2): 124. doi: 10.1002/j.2379-3988.2012.tb00071.x

    • Search Google Scholar
    • Export Citation
  • Tseng, V. and Nutley, S. (2014) Building the infrastructure to improve the use and usefulness of research in education, in K.S. Finnigan and A.J. Daly (eds) Using Research Evidence in Education: From the Schoolhouse Door to Capitol Hill, vol 2, Cham, Switzerland: Springer Science and Business Media, pp 16375, https://www.academia.edu/4926069/Research_Evidence_in_Education_From_the_Schoolhouse_Door_to_Capitol_Hill.

    • Search Google Scholar
    • Export Citation
  • US Department of Education (2016) Non-regulatory guidance: using evidence to strengthen education investments, https://www2.ed.gov/policy/elsec/leg/essa/guidanceuseseinvestment.pdf.

    • Search Google Scholar
    • Export Citation
  • Walker, M., Nelson, J., Bradshaw, S. and Brown, C. (2019) Teachers’ Engagement with Research: What do we Know? A Research Briefing, London: Education Endowment Foundation.

    • Search Google Scholar
    • Export Citation
  • Weiss, C.H. (1977) Research for policy’s sake: the enlightenment function of social research, Political Analysis, 3(4): 53145.

  • Weiss, C. (1995) The four ‘I’s of school reform: how interests, ideology, information, and institution affect teachers and principals, Harvard Educational Review, 65(4): 57193. doi: 10.17763/haer.65.4.05530845t676w50h

    • Search Google Scholar
    • Export Citation
  • Weiss, C.H., Murphy-Graham, E. and Birkeland, S. (2005) An alternate route to policy influence: how evaluations affect D.A.R.E., American Journal of Evaluation 26(1): 1230. doi: 10.1177/1098214004273337

    • Search Google Scholar
    • Export Citation
  • Whitty, G. (2006) Education(al) research and education policy making: is conflict inevitable?, British Educational Research Journal, 32(2): 15976. doi: 10.1080/01411920600568919

    • Search Google Scholar
    • Export Citation
  • Agresti, A. (2013) Categorical Data Analysis, 3rd edn, Hoboken, NJ: John Wiley & Sons.

  • Arreola, R.A. (2000) Developing A Comprehensive Faculty Evaluation System, 2nd edn, Bolton, MA: Anker Publishing Company, Inc.

  • Asen, R., Gurke, D., Conners, P., Solomon, R. and Gumm, E. (2013) Research evidence and school board deliberations: lessons from three Wisconsin school districts, Educational Policy, 27(1): 3363. doi: 10.1177/0895904811429291

    • Search Google Scholar
    • Export Citation
  • Berk, R.A. (2005) Survey of 12 strategies to measure teaching effectiveness, International Journal of Teaching and Learning in Higher Education, 17(1): 4862.

    • Search Google Scholar
    • Export Citation
  • Berk, R.A. and Theall, M. (2006) Thirteen Strategies to Measure College Teaching: A Consumer’s Guide to Rating Scale Construction, Assessment, and Decision Making for Faculty, Administrators, and Clinicians, Sterling, VA: Stylus Publishing, LLC.

    • Search Google Scholar
    • Export Citation
  • Betts, J.R., Zau, A. and King, K. (2005) From Blueprint to Reality: San Diego’s Education Reforms, San Francisco, CA: Public Policy Institute of California.

    • Search Google Scholar
    • Export Citation
  • Boaz, A., Davies, H., Fraser, A. and Nutley, S. (2019) What Works Now: Evidence-informed Policy and Practice, Bristol: Policy Press.

  • British Educational Research Association (2014) The role of research in teacher education: reviewing the evidence, Interim report of the BERA-RSA Inquiry. British Educational Research Association, https://www.thersa.org/globalassets/pdfs/reports/bera-rsa-interim-report.pdf.

    • Search Google Scholar
    • Export Citation
  • Cabinet Office (2013) What Works: Evidence Centres for Social Policy, London: Cabinet Office.

  • Cartwright, N. and Hardie, J. (2012) Evidence-based Policy: A Practical Guide to Doing it Better, Oxford,: Oxford University Press.

  • Chang, Y., Hollands, F. M., Holmes, V.R., Shand, R., Evans, P., Blodgett, R., Wang, Y. and Head, L. (2021) Challenges to finding evidence for ESSA: a case study of Southern Urban District, Paper presented at the Annual Meeting of the American Educational Research Association.

    • Search Google Scholar
    • Export Citation
  • Cicchetti, D.V. (1994) Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology, Psychological Assessment, 6(4): 28489. doi: 10.1037/1040-3590.6.4.284

    • Search Google Scholar
    • Export Citation
  • Clayton, M.J. (1997) Delphi: a technique to harness expert opinion for critical decision‐making tasks in education, Educational Psychology, 17(4): 37386. doi: 10.1080/0144341970170401

    • Search Google Scholar
    • Export Citation
  • Coburn, C.E. and Turner, E.O. (2011a) Research on data use: a framework and analysis, Measurement: Interdisciplinary Research and Perspective, 9(4): 173206. doi: 10.1080/15366367.2011.626729

    • Search Google Scholar
    • Export Citation
  • Coburn, C.E. and Turner, E.O. (2011b) Putting the ‘use’ back in data use: an outsider’s contribution to the measurement community’s conversation about data, Measurement: Interdisciplinary Research and Perspective, 9(4), 22734. doi: 10.1080/15366367.2011.634653

    • Search Google Scholar
    • Export Citation
  • Daly, A.J., Finnigan, K.S., Jordan, S., Moolenaar, N.M. and Che, J. (2014) Misalignment and perverse incentives: examining the politics of district leaders as brokers in the use of research evidence, Educational Policy, 28(2): 14574. doi: 10.1177/0895904813513149

    • Search Google Scholar
    • Export Citation
  • Donaldson, S.I., Christie, C.A. and Mark, M.M. (eds) (2009) What Counts as Credible Evidence in Applied Research and Evaluation Practice?, Thousand Oaks, CA: Sage.

    • Search Google Scholar
    • Export Citation
  • Edwards, W., Guttentag, M. and Snapper, K. (1975) A decision theoretic approach to evaluation research, in E. Struening and M. Guttentag (eds) Handbook of Evaluation Research, Vol 1, Thousand Oaks, CA: Sage, pp 13981.

    • Search Google Scholar
    • Export Citation
  • Farley-Ripple, E., May, H., Karpyn, A., Tilley, K. and McDonough, K. (2018) Rethinking connections between research and practice in education: a conceptual framework, Educational Researcher, 47(4): 23545. doi: 10.3102/0013189X18761042

    • Search Google Scholar
    • Export Citation
  • Farrell, C.C. (2014) Designing school systems to encourage data use and instructional improvement: a comparison of school districts and charter management organizations, Educational Administration Quarterly, 51(3): 43871. doi: 10.1177/0013161X14539806

    • Search Google Scholar
    • Export Citation
  • Farrell, C.C. and Coburn, C.E. (2017) Absorptive capacity: a conceptual framework for understanding district central office learning, Journal of Educational Change, 18(2): 13559. doi: 10.1007/s10833-016-9291-7

    • Search Google Scholar
    • Export Citation
  • Feuer, M.J. (2015) Evidence and advocacy, in M.J. Feuer, A.I., Berman and R.C. Atkinson (eds) Past as Prologue: The National Academy of Education at 50. Members Reflect, Washington, DC: National Academy of Education, pp 95101.

    • Search Google Scholar
    • Export Citation
  • Finnigan, K.S., Daly, A.J. and Che, J. (2013) Systemwide reform in districts under pressure: the role of social networks in defining, acquiring, using, and diffusing research evidence, Journal of Educational Administration, 51(4): 47697. doi: 10.1108/09578231311325668

    • Search Google Scholar
    • Export Citation
  • Gamoran, A. (2018) Evidence-based policy in the real world: a cautionary view, Annals of the American Academy of Political and Social Science, 678(1):18091. doi: 10.1177/0002716218770138

    • Search Google Scholar
    • Export Citation
  • Hallgren, K.A. (2012) Computing inter-rater reliability for observational data: an overview and tutorial, Tutorials in Quantitative Methods for Psychology, 8(1): 2334. doi: 10.20982/tqmp.08.1.p023

    • Search Google Scholar
    • Export Citation
  • Hanushek, E.A. (2015) Education’s double helix, in M.J. Feuer, A.I., Berman and R.C. Atkinson (eds) Past as Prologue: The National Academy of Education at 50. Members Reflect, Washington, DC: National Academy of Education, pp 30912.

    • Search Google Scholar
    • Export Citation
  • Hart, N. and Tromble, K. (2021) Data must lead the way to reopening our schools, 9 February, https://www.nextgov.com/ideas/2021/02/data-must-lead-way-reopening-our-schools/171947/.

    • Search Google Scholar
    • Export Citation
  • Hollands, F., Pan, Y. and Escueta, M. (2019) What is the potential for applying cost-utility analysis to facilitate evidence-based decision making in schools? Educational Researcher, 48(5): 28795. doi: 10.3102/0013189X19852101

    • Search Google Scholar
    • Export Citation
  • Honig, M.I. and Coburn, C. (2008) Evidence-based decision making in school district central offices: toward a policy and research agenda, Educational Policy, 22(4): 578608. doi: 10.1177/0895904807307067

    • Search Google Scholar
    • Export Citation
  • Huguet, A., Coburn, C.E., Farrell, C.C., Kim, D.H. and Allen, A.R. (2021) Constraints, values, and information: how leaders in one district justify their positions during instructional decision making, American Educational Research Journal, 58(4): 71047. doi: 10.3102/0002831221993824

    • Search Google Scholar
    • Export Citation
  • Joyce, K.E. and Cartwright, N. (2020) Bridging the gap between research and practice: predicting what will work locally, American Educational Research Journal, 57(3): 104582. doi: 10.3102/0002831219866687

    • Search Google Scholar
    • Export Citation
  • Kane, T.J. (2016) Connecting to practice: how we can put education research to work, Education Next, 16(2): 8088.

  • Kaplan, A., Cromley, J., Perez, T., Dai, T., Mara, K. and Balsai, M. (2020) The role of context in educational RCT findings: a call to redefine ‘evidence-based practice’, Educational Researcher, 49(4): 28588. doi: 10.3102/0013189X20921862

    • Search Google Scholar
    • Export Citation
  • King, J.A. and Pechman, E.M. (1984) Pinning a wave to the shore: conceptualizing evaluation use in school systems, Educational Evaluation and Policy Analysis, 6(3): 24151. doi: 10.3102/01623737006003241

    • Search Google Scholar
    • Export Citation
  • Malouf, D.B. and Taymans, J.M. (2016) Anatomy of an evidence base, Educational Researcher, 45(8): 45459. doi: 10.3102/0013189X16678417

    • Search Google Scholar
    • Export Citation
  • Marsh, J.A. and Farrell, C.C. (2015) How leaders can support teachers with data-driven decision-making: a framework for understanding capacity building, Educational Management Administration and Leadership, 43(2): 26989. doi: 10.1177/1741143214537229

    • Search Google Scholar
    • Export Citation
  • Mills, K.J., Lawlor, J.A., Neal, J.W., Neal, Z.P. and McAlindon, K. (2020) What is research? Educators’ conceptions and alignment with United States federal policies, Evidence & Policy, 16(3): 33758.

    • Search Google Scholar
    • Export Citation
  • Moe, T.M. (2015) Vested interests and political institutions, Political Science Quarterly, 130(2): 277318. doi: 10.1002/polq.12321

  • National Commission on Excellence in Education (1983) A nation at risk: the imperative for educational reform, Elementary School Journal, 84(2): 11330. doi: 10.1086/461348

    • Search Google Scholar
    • Export Citation
  • Nutley, S.M., Walter, I. and Davies, H. (2007) Using Evidence: How Research Can Inform Public Services, Bristol: Policy Press.

  • OECD (Organisation for Economic Co-operation and Development) (2015) Education Policy Outlook 2015: Making Reforms Happen, Paris: OECD Publishing.

    • Search Google Scholar
    • Export Citation
  • Oliver, K., Innvar, S., Lorenc, T., Woodman, J. and Thomas, J. (2014) A systematic review of barriers to and facilitators of the use of evidence by policymakers, BMC Health Services Research, 14(1): 112. doi: 10.1186/1472-6963-14-1

    • Search Google Scholar
    • Export Citation
  • Penuel, W.R. and Farrell, C. (2017) Research-practice partnerships and ESSA: a learning agenda for the coming decade, in E. Quintero-Corrall (ed) The Social Side of Reform, Cambridge, MA: Harvard Education Press.

    • Search Google Scholar
    • Export Citation
  • Penuel, W.R., Allen, A.R., Coburn, C.E. and Farrell, C. (2015) Conceptualizing research-practice partnerships as joint work at boundaries, Journal of Education for Students Placed at Risk, 20(1–2): 18297. doi: 10.1080/10824669.2014.988334

    • Search Google Scholar
    • Export Citation
  • Penuel, W.R., Briggs, D.C., Davidson, K.L., Herlihy, C., Sherer, D., Hill, H.C., Farrell, C. and Allen, A.R. (2017) How school and district leaders access, perceive, and use research, AERA Open, 3(2): 117.

    • Search Google Scholar
    • Export Citation
  • Resnick, L. (2015) The National Academy of Education’s role in bridging research and practice, in M.J. Feuer, A.I., Berman and R.C. Atkinson (eds) Past as Prologue: The National Academy of Education at 50. Members Reflect, Washington, DC: National Academy of Education, pp 32528.

    • Search Google Scholar
    • Export Citation
  • Schiefelbein, E., Wolff, L. and Schiefelbein, P. (1998) Cost-effectiveness of education policies in Latin America: a survey of expert opinion, Inter-American Development Bank, https://publications.iadb.org/publications/english/document/Cost-Effectiveness-of-Education-Policies-in-Latin-America-A-Survey-of-Expert-Opinion.pdf.

    • Search Google Scholar
    • Export Citation
  • Schneider, M. (2020) Introducing operation reverse the loss, 26 October, https://ies.ed.gov/director/remarks/10-26-2020.asp.

  • Schneider, M. (2021) The road to learning recovery, 16 March, https://ies.ed.gov/director/remarks/3-16-2021.asp.

  • Shand, R., Pan, Y., Leach, S., Nayyer-Qureshi, S., Hollands, F., Yan, B., Dossett, D., Wang, Y., Head, L. and Chang, F. (2020) Program evaluation with administrative data: new applications, promise, and challenges for value-added models, Poster session at AEFP Conference, Fort Worth, TX.

    • Search Google Scholar
    • Export Citation
  • Sharples, J. (2013) Evidence for the Frontline, London: The Alliance for Useful Evidence.

  • Spencer, T.D., Detrich, R. and Slocum, T.A. (2012). Evidence-based practice: a framework for making effective decisions, Education and Treatment of Children, 35(2): 12751. doi: 10.1353/etc.2012.0013

    • Search Google Scholar
    • Export Citation
  • Tseng, V. (2012) The uses of research in policy and practice and commentaries, Social Policy Report, 26(2): 124. doi: 10.1002/j.2379-3988.2012.tb00071.x

    • Search Google Scholar
    • Export Citation
  • Tseng, V. and Nutley, S. (2014) Building the infrastructure to improve the use and usefulness of research in education, in K.S. Finnigan and A.J. Daly (eds) Using Research Evidence in Education: From the Schoolhouse Door to Capitol Hill, vol 2, Cham, Switzerland: Springer Science and Business Media, pp 16375, https://www.academia.edu/4926069/Research_Evidence_in_Education_From_the_Schoolhouse_Door_to_Capitol_Hill.

    • Search Google Scholar
    • Export Citation
  • US Department of Education (2016) Non-regulatory guidance: using evidence to strengthen education investments, https://www2.ed.gov/policy/elsec/leg/essa/guidanceuseseinvestment.pdf.

    • Search Google Scholar
    • Export Citation
  • Walker, M., Nelson, J., Bradshaw, S. and Brown, C. (2019) Teachers’ Engagement with Research: What do we Know? A Research Briefing, London: Education Endowment Foundation.

    • Search Google Scholar
    • Export Citation
  • Weiss, C.H. (1977) Research for policy’s sake: the enlightenment function of social research, Political Analysis, 3(4): 53145.

  • Weiss, C. (1995) The four ‘I’s of school reform: how interests, ideology, information, and institution affect teachers and principals, Harvard Educational Review, 65(4): 57193. doi: 10.17763/haer.65.4.05530845t676w50h

    • Search Google Scholar
    • Export Citation
  • Weiss, C.H., Murphy-Graham, E. and Birkeland, S. (2005) An alternate route to policy influence: how evaluations affect D.A.R.E., American Journal of Evaluation 26(1): 1230. doi: 10.1177/1098214004273337

    • Search Google Scholar
    • Export Citation
  • Whitty, G. (2006) Education(al) research and education policy making: is conflict inevitable?, British Educational Research Journal, 32(2): 15976. doi: 10.1080/01411920600568919

    • Search Google Scholar
    • Export Citation

Content Metrics

May 2022 onwards Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 56 56 17
PDF Downloads 30 30 13

Altmetrics

Dimensions