Home/Resources/Innovative Ideas/Data Management
Innovative Ideas
in Educator Compensation Reform
Data Management and Analysis System
A data management and analysis system consists of databases to organize, store, and secure an organization's data; reporting software to retrieve and display data consistent with an organization's business rules and staff roles; and software (including staff properly trained to use the software) to enable more open-ended query and analysis of an organization's information. The implementation of a robust and thoughtful data management and analysis system can support the implementation of alternative compensation programs. For example, such a system tracks the metrics used to measure performance (student and staff data), includes the rules used to calculate the amount of individual awards, and potentially can track the long-term consequences of those decisions on student learning or other measures of organizational effectiveness. Overall, a data management and analysis system can thoroughly document changes and can link those changes to efforts that lead to more effective educational decisions.
What Is This?
Why Is This Important?
What Are the Benefits?
What Are Some Tips for Implementation?
What Are the Selection Criteria?
Innovative Idea: Create Communities of Practice to Share and Manage Knowledge.
What Is This?
The data management system provides a logical structure for the various databases that contain the operational data—working together, these should help provide answers to important questions. The data analysis system explores those databases to formalize policy questions and test answers. This exploration and confirmation is especially crucial in alternative compensation programs, particularly those that seek to link teacher performance and student learning outcomes and those that inform ongoing teaching practices and student learning beyond annual standardized test scores.
There is very limited formal, empirical research on the differential effectiveness of school, programs, or teacher effectiveness on student performance. Key questions that guide the administration of alternative compensation programs—such as which teachers' students showed the greatest learning gains, which teachers performed well on personnel evaluations, and which teachers are eligible for additional pay—can be answered more thoroughly by integrating data from multiple data sources across the educational organization. Even with excellent data integration, however, any analysis of the complex sets of relationships that link teachers' capacities, skills, and practices to student performance will require careful modeling and testing. The analytic component of the data management and analysis system holds the tools to make this possible.
A data management and analysis system must be carefully selected, methodically implemented, and rigorously applied to ensure that the system is well supported by the organization's data structures. The system also must provide the types of information the organization needs to allow the decision to take action and, if necessary, make changes.
In an educational organization, refining or developing a data management and analysis system is more than a simple technical process led and performed by data administrators and data analysts. It consists of building trust among the different program offices with databases; involving the right people at the right time; understanding the data needs of the different users; coping with different definitions of common terms to meet differing external requirements; establishing up front the relevant policy questions to answer with the data; devising a large-scale plan that is implemented in small, doable increments; and aligning the information and data systems plan with the organizational goals and plans.
Why Is This Important?
Traditionally, districts and schools collect and manage data to comply with state and federal reporting requirements. It is not atypical for districts to collect data from the same audiences (e.g., teachers, students, administrators) in multiple, inconsistent ways. One challenge is that the timing of data collection and reporting activities is not always well aligned across federal or state program areas, which can lead to duplication of efforts and inconsistent analyses of data that look similar but that were collected at different times during the year. Moreover, the data needed to support alternative compensation processes and decisions simply may not be available.
Because education data systems were originally designed to generate reports on discreet activities, most districts and schools will encounter difficulties linking data that result in reliable indicators connecting student learning to teacher performance. Basic data management systems used in education, particularly the various payroll and student information systems, are essentially business systems, intended to ensure that teachers are paid and students are tracked. For alternative compensation, other kinds of information is needed; attempting to award differential pay using these basic business systems can easily result in misuse or misinterpretation of data. A few examples follow:
- Educators may be misidentified for additional pay because the complex activity of teaching is not captured in the data system. For example, team teaching or informal ability or subject-area grouping in elementary schools often is not tracked centrally. This lack of a clear link between teachers and students by subject taught can lead to a misidentification of recipients for additional pay, with the following consequences:
- Loss of trust between and among members of the school community.
- Public scrutiny and criticism.
- Loss of confidence in the legitimacy of the program, which may eventually dismantle the program.
- This lack of a consistent linkage can be exacerbated by increasing demands to demonstrate what programs or instructional practices work best. Many districts are not able to track the professional development that is delivered to a particular school or teacher. This inability to link changes in teacher resources (professional development) to individual teachers and students by subject tested is a serious breakdown in the chain of evidence needed to establish a valid system. Systems that have difficulty providing teachers and administrators with the necessary data to make decisions about which instructional practices to implement undermine trust in choosing strategies that work.
- Many data collection systems are only point-in-time snapshots. They do not contain information about changes in the educational system. Without detailed information about teacher practices and content delivered, it is not possible to track changes that occur within course units, marking periods, semesters, and years. Without such data points, the impact of changes at this level of the organization cannot clearly model the detailed relationships that structure teaching and learning.
What Are the Benefits?
The benefits of investing in a high-quality, robust data management and analysis system to more effectively implement educator compensation reform include the following:
- Integration of data from multiple sources.
- Integration of relevant data to monitor and evaluate programs.
- Improved data quality through consistent data collection that systemically collects the right data.
- Improved information quality and timeliness because data are acquired on a known schedule, stored meaningfully, and equipped with appropriate reporting and analytics.
- Support of cross-organizational collaboration and knowledge sharing.
- Broad access to data and continual reporting, which builds a common knowledge base, enhances collaboration, and ensures aligned efforts.
- Enhanced trust of data and decision making based on data.
- Continual refinement of data to be more useful to all.
- Quicker turnaround of relevant information.
- Thicker data nexus.
- More equitable decisions based on the use of high-quality data.
What Are Some Tips for Implementation?
The successful implementation of a data management and analysis system to support educator compensation reform is no small task. Fortunately, there are existing resources to assist state and district leaders with assessing their readiness to invest in a data management and analysis system (Learning Point Associates & Educational Service Agency [ESA] Alliance of the Midwest, 2006), and expert reviews of the commercial and not-for-profit products are available (Wayman, Stringfield, & Yakimowski, 2004). Additional resources are included in this online resource, but a good starting place is the following list of implementation tips:
- Think ahead and implement in small increments, following a broad outline for developing and refining data management and analysis systems. Build a system based not on where you are but on where you want to be in the future (Learning Point Associates & ESA Alliance of the Midwest, 2006).
- Build trust among program areas (i.e., distinct district offices) and those individuals and groups that will develop, manage, provide, and use data. System designers often have weak knowledge of the ways in which staff members use data daily and weekly for decision making. Therefore, involve information technology staff and data managers in organizational mission and vision discussions as well as other discussions where data will be needed or used. However, be cautious in involving them too deeply too early; they may lose interest. But absolutely avoid bringing information technology staff and data managers in too late in the discussion. Once implementation begins, make sure technical specialists continue to talk frequently with substantive experts, leaders, and practitioners.
- Resist making a list of the data you already have; instead, determine what data you will need to persuade others to act (Learning Point Associates & and ESA Alliance of the Midwest, 2006). Let the program and its needs drive your data collection and structure. Be sure you are clear about why you need each data element.
- Develop a general understanding of any existing data system's strengths and weaknesses, but do not let that understanding govern where you want to end up. Establish a checklist that assesses the capacity of the existing data systems, given what you know you will need in the future.
- Establish a data dictionary that specifies and defines data elements within the system.
- Design data systems so that the data can be used for the intended purposes. Explore scenarios in which data will be used by different users to identify how decision making will be reshaped.
- Continually assess and improve access to data and data reports. Work to ensure that all relevant staff receive and use relevant reports. Work to get feedback about relevance of reports. This approach will tell you both about how to improve the data system and what the staff still needs to know and learn.
What Are the Selection Criteria?
Because innovative ideas, by their very nature, are new approaches to education reform, the research base has not kept pace. New approaches to education reform cannot be studied unless they are tried. The following criteria were applied by CECR to determine whether aspects of various approaches to developing, implementing, and maintaining an effective data management and analysis system to support educator compensation reform are truly innovative:
- Did the approach view the data management and analysis system as more than the just software? In other words, are people and processes part of the system?
- Did the approach include use of standardized organizationwide data definitions?
- Did the approach contribute positively to the overall implementation of a data management and analysis system?
- Did the approach help identify ways to monitor and provide feedback on the data management and analysis system? Stated another way, is the idea helping to determine if the data management and analysis system is working?
- Did the approach support alignment of the organizational goals with the data management analysis system's requirements?
This page last updated on: April 28, 2008
