Pages: 20

This chapter provides background information about state testing and the author’s various roles in the field of large-scale educational assessment from 1989 through 2019 that will be helpful to readers as they make their way through each of the major phases of the large-scale assessment cycle. The chapter begins with an overview of the evolution of large-scale state assessment programs from the 1980s through the 2010s. That overview is followed by a timeline of the author’s roles and responsibilities serving in various capacities during that period, first at an assessment company, then in the student assessment unit at a state department of education, and for nearly two decades as a technical consultant advisor to states and other organizations across the country. The chapter concludes with a brief description of distinguishing features of the nine state assessment programs discussed throughout the book and an explanation of the organization of the book.

Pages: 20

Standard setting is the process of determining achievement level cutscores. The field has come to realize that you should not design and develop a state test without a clear understanding of the achievement standards you hope to measure and the inferences you hope to draw with that test. This chapter describes the struggles the field is still having to connect achievement standards to student performance on a test through discussion of topics such as whether achievement standards are relative or absolute; the role of agreement and disagreement in setting cutscores; the concept of building a test to a desired cutscore; the use of impact data when setting cutscores; and the validation of achievement standards and cutscores through the use of external information. The chapter begins with the review of a series of unfortunate circumstances encountered during standard setting that demonstrate some of the fundamental challenges in trying to classify student proficiency in a content area on the basis of their performance on an on-demand test. The chapter ends with a reminder that student proficiency exists outside of the test and that it is not the role of psychometricians or purpose of the test to define proficiency.

Pages: 19

Beginning with the premise that form follows function, this chapter addresses ramifications of the field’s inability to clearly define the function of state testing while attempting to design a test to serve multiple, sometimes conflicting, purposes and uses. The chapter describes the measurement-driven approach to test design that was followed and offers an education-centered alternative for consideration. The chapter closes by expressing concerns about the current level of dependence on Item Response Theory (IRT) and reliance on large, existing item banks as a source of items on state tests. A key takeaway from this chapter regarding test design is that the field is still struggling to find the answer to the question it was asking in 1989; that is, how to move beyond on-demand end-of-year tests toward more authentic and comprehensive assessment of student performance.

Pages: 22

This chapter addresses four big picture issues that reflect the changing and increasing demands that have been placed on test developers: instantiating state content standards through test items, defining achievement standards and a continuum of learning, interpreting test performance through item and test statistics, and distinguishing between bias and other factors that cause differences in student performance. These are complex issues which ultimately affect the interpretation and use of test scores, and test developers have been asked to bear the brunt of figuring out how to deal with them. In short, test developers are often responsible for high-level policy decisions that rightly belong outside of the province of the state test. By tasking test developers with these critical high-level decisions, the field may have reached a tipping point. The hands-on, painstaking process which produced a low number of flawed items in the past may have ultimately made the test development process too unwieldy for a state to continue to develop its own items for a custom state test as test items themselves and the issues surrounding them become more complex.

Pages: 18

Recent trends in state testing have heightened the importance of decisions related to Field Testing and increased their complexity. Those that are building state tests are relying on the principles of IRT holding true when using item information from an item bank to build, equate, and score tests. In practice, the validity of inferences drawn from test scores is heavily dependent upon decisions made during field testing. This chapter addresses whether the field needs to be more deliberate in the sampling of students for field testing, highlighting three issues that the field must consider when placing so much trust in item banks and IRT. The first issue includes general questions related to the use of field test items and statistics, particularly in consideration of the increased attention being paid to issues of cultural relevance, fairness, and intersectionality. A related second issue is the selection of a representative sample of students for field testing. A third issue, related to the first two, addresses the dangers in using a population of convenience (e.g., the pool of tested students in the state) as the field test sample, particularly immediately following the adoption of new standards or the implementation of a new state test.

Pages: 21

This chapter addresses three forces that will continue to have a profound impact on how and to whom state tests are administered: test security, the personalization of test administrations, and time. Increased stakes, or consequences, for schools associated with federal accountability requirements created an industry devoted to state test security and fostered a relationship between tests and test users that created a need for that industry. Requirements and the desire to include all students in state testing and to provide them with a fair testing experience led to the increased availability and use of accommodations and is leading toward increased flexibility in test administration. The combination of the call for more authentic assessment, increased emphasis on alignment to standards, and the adoption of more complex and rigorous state content and achievement standards pushed testing time to beyond its limits. A more complex time-related issue that continues to grow in importance is the relationship between time and proficiency. As the focus on providing actionable information to inform the instruction of individual students increases and state testing evolves from external and on-demand to curriculum-embedded, the pressure those forces place on the field will only increase.

Pages: 17

This chapter addresses issues related to the scoring of student responses to constructed-response items by trained human scorers that should be better understood before fully handing the task off to computers. The future of scoring is automated scoring engines. In many ways, however, the human scorers are more of a black box than the computer-based models replacing them. The first section of the chapter provides examples of issues involving scores at the extreme ends of the scoring rubric and student responses that are difficult to score, such as responses receiving "0" scores, perfect scores, and responses classified as off topic. The second section addresses issues related to reliability and the manner in which the field has incorporated the unreliability of human scoring into the system. Removing that unreliability from the system through automated scoring might result in unanticipated and unintended consequences. The final section of the chapter addresses the field's readiness for scoring in a future of state testing that will likely make widespread use of curriculum-embedded performance tasks and tasks personalized to meet the needs of individual students, and a future that places a much heavier emphasis on application skills than on the demonstration of content knowledge.

Pages: 17

The process of creating test scores and reporting scales is where tensions between decades of choices designed to maximize the technical quality and utility of group scores for institutional purposes and the inexorable trend of state assessment toward the individual, the formative, and the desire for actionable information to support instruction and learning come to a head. It is the phase in the process at which one must heed Randy Bennett’s warning about the limits to which good processes can compensate for poor or inappropriate instrumentation, and vice versa. This chapter focuses on three psychometric issues related to the disconnect between testing for institutional and instructional purposes. The first is the choice of an Item Response Theory (IRT) model, beginning with a brief discussion of the historical choice to use IRT for scoring rather than the Rasch model for theory-driven scale development. The second is the inescapable fact that, with regard to individual student scores, an observed test score will always be a single observation, a sample of 1. The third is the allure and illusion of precision in large-scale testing, particularly with regard to decisions made during standard setting, creating scales, and reporting scaled scores.

Pages: 17

Equating receives its own chapter because without equating we can have a state test, but not a sustainable state assessment program. Equating also receives its own chapter because of the unresolved challenges in equating we have faced as the design and content of state tests has become more complex and multidimensional. This chapter begins with a detour into norm-referenced testing to contrast the minimal equating needs in the NRT era with those typically encountered in state testing today; also establishing that the history of equating large-scale state tests is relatively brief. The chapter continues with a discussion of the importance of understanding the changes in student performance that equating is intended to capture. The third section of the chapter looks toward equating and the future of state testing. Longstanding difficulty in equating tests containing writing tasks is offered as an example of the challenges the field will face as state testing continues to increase in complexity. The chapter closes with a discussion of potential dangers in the current reliance on, if not blind faith in, IRT as the field shifts toward adaptive testing and returns to building pre-equated test forms by selecting items from large, pre-populated item banks.

Pages: 24

This chapter addresses technical issues related to the production and reporting of test scores. Three aspects of test score reports that have been particularly challenging in the past and are likely to take on even greater importance in the future are discussed. The chapter begins with a discussion of subscores and approaches to meet user demands to report them despite their technical shortcomings and lack of interpretability. Next is an overview of how scaled scores and achievement levels emerged as the primary vehicle for reporting test results, and how the most critical property of the reporting scale was the manner in which it supported and reinforced the state’s message about those achievement levels. Problems that emerged when states needed to expand the use of scaled scores are discussed. The chapter ends with a discussion of the field’s ongoing struggle to meet one of the most fundamental requirements of educational testing, that is, to clearly communicate information about measurement error when reporting test scores, especially test scores for individual students. That struggle is rooted in the flawed attempt to promise too much information about student achievement on the basis of an observed score on a single, on-demand test.

Pages: 20

With this chapter, the focus of the book shifts from generating test scores to making use of those test scores. Specifically, the chapter addresses previous attempts to provide context information intended to make the reported test scores more useful, some more successful than others. As Ryan (2006) noted, “the value of various score reports reflects the match between the report, the intended audience, and anticipated use” (p. 677). When state testing transitioned from norm-referenced to criterion-referenced, or standards-based, testing in the 1980s and 1990s, it remained common practice to provide external information to place the scores in a useful context for educators and policymakers. In the years since NCLB, however, reporting has been centered around achievement levels and disaggregating results by federally required reporting categories. Less attention has been paid to providing contextual information to help key stakeholders interpret and use those test results.

Pages: 24

This chapter addresses what may be the most fundamental aspect of any assessment program; that is, the interpretation and use of test results. Test results are intended to be used and the lack of guidance and support provided to stakeholders tasked with interpreting and using state test results may be the biggest flaw in the way state testing programs have operated since NCLB. Mandated uses of test scores for accountability have been questioned, but largely unchallenged. While econometricians and data scientists routinely use test scores as data to make policy-related interpretations, curriculum and instructional specialists are left on their own to make education-related interpretations. The chapter addresses one area in particular where a lack of adequate support is provided to educators; that is, the extent to which the field has allowed individual student results from a single test administration to become the focus of the interpretation and use of state test results. The field needs to do better at communicating the limitations of current state tests while at the same time actively exploring other assessment and psychometric models better suited for interpretation and use at the individual student level.

Pages: 20

No issue is more fundamental to educational testing than validity, but for a host of reasons validation is an area which historically has received short shrift in state assessment programs. This chapter acknowledges external factors that affect the validation in state testing such as time, money, resources, urgency, and federal law. The focus of the chapter, however, is on three internal factors that have and likely will continue to affect thinking about validity and the validation of state tests. The first is the longstanding confusion within the measurement and assessment communities regarding what validity is and the role that consequences and uses play in validation. The second is the increasing lack of clarity regarding what constitutes a test. The third is the tendency to define what is being assessed by state tests (i.e., student proficiency) solely by student performance on that test. That tautology raises the question, “What actually is student proficiency?”; a question that leads back to the starting point in this discussion of state testing, achievement standards and standard setting, a fitting ending given the cyclical nature of state testing and the never-ending process of validation.

Pages: 3

When I entered the field of large-scale educational assessment in the fall of 1989, state testing was on the cusp of great change.