Evaluation Baron

After Governor Charlie Crist of Florida vetoed SB 6, which was designed to overhaul merit pay for teachers, he convened a group to prepare for the second round of the Race to the Top grant. This group included teachers, superintendents, school board members, members of the PTA, and other leaders. However, a vital member of the team was noticeably absent: a professional evaluator.

Professional evaluators help organizations plan and conduct evaluations of programs, products, processes, policies, etc. In the education realm, evaluators help to determine (among other aspects) whether a program is increasing student achievement, decreasing achievement gaps, increasing high school graduation rates, and increasing college enrollment - all of which are components in the Race to the Top grant program.

When you have a professional evaluator on your team, that person (1) helps you plan and implement techniques to gain visibility as to whether your program is meeting the set objectives, (2) advises you on how to allocate resources where they will bring the most benefit, and (3) provides results, recommendations, and information for you to make informed decisions on how to proceed in making that program a success.

Based on a listing by the American Evaluation Association (AEA), "an international professional association of evaluators devoted to the application and exploration of [evaluation]", there are at least ten different evaluation organizations in the state of Florida that specialize in educational evaluation. I'm sure that one or more of them would be happy to assist Florida in the grant application process.

This should be a wake-up call for all states applying for grants of any kind: if there is an evaluation component in the grant (and there definitely IS in the Race to the Top grants), then you need a professional evaluator to assist you in planning and conducting that evaluation.

How you plan and conduct that evaluation can mean the difference between winning and losing the grant - which equates to vital federal funding that states cannot afford to pass up.

Topics of teacher effectiveness and merit pay have developed from such programs as the NCLB reform and the Race to the Top grant programs. Individual states such as Florida, Colorado, and California, to name a few, have begun to step up to the plate in designing their own evaluation plans for these programs.

If nothing else, the Race to the Top grant program has become a catalyst for change to a more efficient and effective paradigm: that being a more holistic evaluation plan that takes a host of factors into account and puts power into the hands of school leadership to work together with teachers and parents for improved student achievement, a school culture geared toward learning, and a cadre of young people prepared for the post-high school plans of their choice.

An effective evaluation of teacher effectiveness comes not from one test, nor from testing (or any other one factor) alone, but from a comprehensive array of factors. While the only true measure of teacher effectiveness may come from the results of a student's life ten years hence (my 8th grade math teacher comes to mind, who helped students feel valuable and confident in their abilities), schools can and should design a holistic evaluation for teacher effectiveness and merit pay scenarios. These factors may include, but are not limited to, the following (schools may want to focus on different aspects depending on their school's purpose):

• Student test scores
• Teacher observations and interviews
• Teacher self-reports
• Teacher portfolios of their work
• Parent involvement and interaction

Some argue that there are many factors beyond a teachers' control (such as socio-economic status, broken homes, etc.), and they fear that teachers will be held accountable for those factors. While I am not advocating that teachers be called on the carpet for these situations, there is more to a teacher's effectiveness than simply test scores.

As states and schools wrestle with defining and evaluating teacher effectiveness, they may want to ask themselves the following questions:

1. What difference are teachers making in their students' lives?
2. How are teachers preparing students to succeed in and out of the classroom?
3. Do students develop a love for learning because of their teachers?
4. Is the school culture geared toward learning, or is it run like a business or an assembly line?
5. Are teachers rewarded for their efforts in their students' learning, thus giving teachers the motivation to think beyond the textbook?
6. Are teachers given access to professional development opportunities and other resources to build their skills and increase their ability to help make a difference?
7. Do school administrators make the time to meet with teachers often to encourage efforts and provide guidance?

If a teacher is being paid to simply exist in the classroom, then we as a society are doing our children a disservice.

Designing and executing a comprehensive evaluation of teacher effectiveness and merit pay will not only provide visibility and evidence of effectiveness, but will pinpoint areas for improvement so that schools may allocate resources where they are most needed. An effective evaluation spans the whole realm of a given program, and directly stems from the missions and goals of an organization. Schools and states would do well to step back and formulate a theory of change (also known as a logic model, impact map, etc.), and then use that information to create long-term evaluation plans to help capitalize on strengths, identify areas of improvement, and develop their capacity to evaluate their programs.

The Department of Education's draft of the National Education Technology Plan for 2010 (http://www.ed.gov/technology/netp-2010) has evaluation at its core, with overall goals describing outcomes-based education, the effectiveness and efficiency of educational infrastructure, the need for monitoring and evaluation of progress, and accountability for performance.

What can the Department of Education do to ensure that these evaluation goals become a reality?

1. Facilitate the process of recruiting and using evaluation experts at the school, district, state, and national levels to design, conduct, and report on the continuous progress of these goals and trends.
2. Provide generous funding through direct contributions and grant opportunities for the development of core standards and specific learning goals, the creation of realistic activities and assessments directly aligned with those goals, and the monitoring and evaluation of progress and trends associated with these goals.
3. Ensure that information needs are well defined at each level in the educational hierarchy to ensure data-driven decision-making can take place.
4. Allow for the collection and analysis of relevant performance data via refined educational evaluation policies.
5. Facilitate the diffusion of their educational philosophy so that all stakeholders are willing and able to use evaluation results on a continuous basis.

Educators cannot afford to try to handle these processes on their own. Evaluators internal and external to organizations must step in to provide evaluation expertise and to build evaluation capacity within schools and administrations in order to increase program visibility and self-reliance of educational developments.

A common response as I discuss evaluation with people and organizations is that they don't see a real need, or that they cite economic woes as a reason to ignore evaluation. However, here's some of the translation of what they're really saying:

"I don't care how my business is doing. I just want to keep making widgets."

"I don't know a thing about evaluation, so maybe my problems will go away by themselves."

"Evaluation is too much work. I'm busy enough as it is."

The fact is that our current economic state, organizations cannot afford to ignore the need for evaluation. Federal and state governments are focusing on evaluation to root out inefficiency. Non-profit organizations must continually be accountable to funders. For-profit organizations must increase the efficiency of their business processes in order to be ahead of the competition.

All of the above is accomplished by evaluating programs and services to align what should be happening (via criteria and standards) with what is happening. The results may then be used for decision making to make those programs and services more refined, more productive, and more serving of beneficiaries.

Take the time, and make the time to evaluate. The status and growth of your organization depends on it.

The Evaluation Baron, LLC, can introduce you to evaluation concepts and how they will benefit your organization. Contact us for a free initial consultation!

There have been many reports of the growing need for educational reform and its effect on the global economy. Many leaders in education and government are also voicing their concern and stressing the need for accountability, visibility, and evaluation of federal programs (see Obama Administration Launches Program Evaluation Effort at http://www.govexec.com/dailyfed/1009/100709e1.htm). Science, Technology, Engineering, and Math (STEM) education is no exception.

Recent news reports of the reform of STEM education (see The State of Stem at http://www.techlearning.com/article/26400) recommend integrated teaching of the STEM components in order to enhance learning effectiveness. "An intriguing finding was that engaging young people in hands-on projects in engineering and design provides effective ways for them to learn core math and science concepts."

These are some of the benefits of evaluation: to assess the impact and effectiveness of teaching STEM components separate or together; to understand the impact of STEM pedagogy on the global economy, and to maximize the role of teacher effectiveness in this process. Evaluation provides visibility of these vital decision-making tools!

These issues are not without political unrest, though. Nevertheless, teacher unions, administrators, parents, and the students themselves must come together on this and other educational issues. Being able to evaluate and use the resulting information helps individuals, schools, and communities. A first step is to facilitate evaluation of STEM programs in the classroom. Together, we can foster a competitive nation and world (see Universities Pledge to Train Thousands More Math and Science Teachers by 2015 at
http://chronicle.com/article/Universities-Pledge-to-Train/63439/?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+chronicle%2Fnews+%28The+Chronicle%3A+Top+Stories%29&utm_content=Google+Reader)
- one teacher, one student, and one classroom at a time. The key is to evaluate!

The Evaluation Baron, LLC can help you in this process. Contact us for a free initial consultation!

1. Accrediting schools
2. Judging the quality of education or corporate training curriculum
3. Judging the effectiveness of a training program
4. Improving the recruitment process of volunteers or employees
5. Improving a commercial product
6. Improving employee retention
7. Determining public perception of a new policy or program
8. Understanding the cost-benefit, cost-effectiveness, or cost-utility of an employee benefit package
9. Deciding whether to implement a community development program
10. Developing greater accountability to project funders

The Evaluation Baron, LLC, can help you understand the benefits of these types of evaluations, and what it means to your organization specifically. Contact us for a free initial consultation!

A recent news article from the Christian Science Monitor, "Eyeing Stimulus Money for Education, States Adopt Reforms (posted Thursday, December 3, 2009 at http://www.eschoolnews.com/funding/funding-news/?i=62081), highlights the need for evaluation in making progress with education reforms.

In particular, the article points out four areas where states could receive grant money: "Establishing data systems that track students' progress from preschool through college, developing and using rigorous standards and assessments, improving teacher effectiveness and the distribution of high-quality teachers, and turning around the lowest-performing schools." The article states that the last two issues have not been addressed much because of the obscure nature of defining teacher effectiveness and developing a system to improve schools.

That's what evaluation is all about. Organizations work with professional evaluators to define topics such as teacher effectiveness, establish criteria and standards for what should be happening, and then take steps to evaluate whether or not those things are happening. Organizations may then choose to implement recommendations generated from the study to enhance and improve the program.

While creating standardized definitions, criteria, and standards for schools across the country will certainly be a more involved process with all levels of education, schools can and should start somewhere by evaluating what is working or not working in school reform (to at least establish a baseline), and then proceeding with strategies for improvement.

National, state, and local organizations must work hand in hand in order for true reform to succeed. Schools would do well to look at this grant offering as an opportunity to evaluate their current situation and then take steps toward improvement.

The Evaluation Baron, LLC can help you evaluate how your school, district, or state is doing in any of these topics. Contact us for a free initial consultation!

Recently I attended a one-day conference, during which time I literally heard 14 different terms to denote evaluation. Let me briefly share them with you, along with a simple explanation of each one.

1. Evaluation: Judging the merit or worth of something, such as a program, process, product, etc. Yes, it was only a few times, but I did hear this word...like a sweet melody ringing in my ears.
2. Analysis: An investigation of data pertaining to a situation, or the situation itself.
3. Assessment: This term is usually used within the context of educational testing, but is generally considered a subset of evaluation.
4. Find Out: Although simple, what logically follows from this phrase is the question of how and why are you going to find out...for what purpose and by what means.
5. Observation Checklist: This is one of many tools used with data collection to make sense of what is being observed, such as a class, a performance, a group interacting somehow, etc.
6. Tracking: This term denotes the monitoring side of evaluation, which is an ongoing collection of data on a given topic, usually to determine compliance.
7. Review: To make an appraisal of data or of a situation, such as a review of an existing program, etc.
8. Key Indicators: This phrase is synonymous with criteria and standards (i.e., what should the program be doing, and to what standard should it occur).
9. Measuring Success/Measurement: This term is also considered a subset of evaluation, and often pertains to quantitative data, although measurement itself can be in quantitative or qualitative terms.
10. Performance Improvement: This term is often used in the human resource or healthcare fields to denote such things as employee productivity and wellness.
11. Feedback: Reflective of customer service, feedback can be opinions or other information about your programs. Feedback can come from both internal and external sources.
12. Rate: This term denotes a judgment of quality, and is often seen in surveys (e.g., "On a scale of 1-5, rate the effectiveness of the following...").
13. Accountability: Evaluations help to establish accountability for organizational performance. Evaluations link performance criteria to results to show funders or other stakeholders the efficiency, effectiveness, and return on investment of a program, product, or process.
14. Harvesting: This term denotes the data collection and analysis phase of an evaluation, and can be by various means (e.g., web analytics, documentation review, surveys, interviews, data mining, ethnography).

Let us know how you feel! Tell us what term you affiliate with, and how you use it (or want to use it) in your organization!

No matter what term you use, The Evaluation Baron, LLC can help you increase the effectiveness and efficiency of your programs, products, or business processes by aligning results with your organizational goals.

Contact us for a free initial consultation!

Over the last decade, and more recently with the economic roller coaster we've been on, organizations are becoming more self-sufficient. We're seeing this evidence via corporate downsizing, professional development, and a structure of learning taking place in companies worldwide.

Along with self-sufficiency in business strategy planning, marketing, manufacturing processes, and retail production and distribution should bring the evaluation of these aforementioned tasks. The development of an organization's capacity to plan, conduct, and use evaluation is crucial to not only the success of individual programs, processes, and tasks, but also the overall success of the organization as a whole. This applies to all disciplines (e.g., non-profits, NGOs, for-profits, education, government, military).

Although many definitions of evaluation capacity building (ECB) now circulate the globe, I believe the simplest working definition can be found in the writing of Stockdill, Baizerman, and Compton (2002), where they define ECB as, "the intentional work to continuously create and sustain overall organizational processes that make quality evaluation and its uses routine" (p. 14). Here evaluation becomes the norm rather than the exception. Evaluations themselves and the corresponding structure become transparent within the organization. All hierarchical levels in the organization - from the clerical staff to the team leaders to the CEO - understand and capitalize on the importance of evaluation and how it affects business processes and the accompanying bottom line. Creating evaluation capacity with any type of organization involves not only evaluators, but also the organization itself. Both parties need to take a proactive mindset to create the organizational structure needed to support evaluation capacity.

As an organizational leader, think of evaluation capacity building as creating evaluation self-sufficiency within your organization. You not only need to see the importance of
evaluation in the short and long term, but also develop the skills to conduct evaluations and use the results of such to further your business vision.

A good place to start is to think about the topics listed in A Checklist for Building Organizational Capacity (Volkov and King, 2007) made available through The Evaluation Center at Western Michigan University. Discuss these principles with your colleagues and how you can implement them in your daily practice. There are many resources available to assist you, such as evaluation consultants, professional development opportunities, etc.

The first step is to understand how becoming self-sufficient in evaluation can improve your organizational mission, vision statements, goals, and ultimately the bottom line. Then take additional steps to see that goal become a reality.

• You launch a new website for your organization. How do you know if anyone has seen it, is using the information on it, or if it's effective in gaining new customers?
• You develop a new employee-training program to be used at all your area offices. How do you know if employee performance has improved because of that program, or what employees think of the program?
• You received state funding for your community literacy program. How will you know if the program achieved its goals? How will you account for the program effectiveness to the funders?

The answer to all three of these scenarios is that you don't know unless you evaluate the progress or impact of those programs.

Evaluation judges the merit or worth of something, whether that something is a program, product, process, performance, or policy. Evaluation helps you determine its impact on your organization, the community, and the people involved. Evaluation can guide your strategic planning. Evaluation can save you time and money.

As an organizational leader, consider the following questions:

1. In these tough economic times, how can I determine which programs work and which I should consider down-scaling or canceling?



2. How do my organization's activities coincide with short, medium, and long-term goals?



3. How can I get greater return on investment for the quantity and quality of work that goes into this organization?



4. How can I ensure my programs are cost effective?

These questions are just the first step in helping your organization become more effective. To answer these questions, you can seek out information on how to plan, conduct, and analyze evaluations. You can work with an external evaluator, or learn how to develop your organization's capacity for conducting internal evaluations.

Evaluation equals understanding and decision-making power.

So what type of areas or topics can web analytics assist you with? Here are some examples from the online learning field:

• Ensuring class members complete online assignments
• Discovering access or informational problems with a website
• Finding out if others are benefiting from the information on your website (with reference to open learning as applicable)
• Obtaining feedback from users regarding content, usability, and applicability - and being able to make adjustments accordingly
• Assessing the need for certain courses or topic areas
• Learning who the most frequent users are, why they participate, and how they can propel your courses to success

Web analytics, an up and coming technology that evaluates website activity, can be a huge benefit to all organizations, whether you have a small website or a large e-business. Here's how web analytics can improve your organization:

• Find out (among other vital data) who is on your website, for how long, and what they look at most
• Based on the above information, you can cater your website you specific groups of users and you can provide them the vital information that may mean the difference between having them do business with you and having them go elsewhere
• External evaluators (i.e., those who help you make sense out of the web information), can learn more about how customers and potential customers feel about your organization so they can make accurate and appropriate recommendations for improving your organization.

During the past few years, I've had an amazing experience conducting evaluations with an academic support group on the Brigham Young University (BYU) campus. The Center for Teaching and Learning (CTL) strives to “improve student learning in the context of the Aims of a BYU Education. The center accomplishes this goal by assisting BYU faculty, instructors, and administrators to -

• refine effective program- and course-level learning outcomes,
• design learning experiences to achieve desired outcomes,
• enhance the quality of learning experiences through technology integration,
• evaluate the effectiveness of learning experiences, and
• advance knowledge of effective learning and teaching.”

The evaluation field in general loves to debate and discuss the important issues surrounding evaluation. One longstanding topic (if there is anything more longstanding than the quantitative/qualitative debate :) is the difference between evaluation and research.

There are many opinions, and there probably is no right or wrong answer, but since this is my blog, I get to state my opinions and experiences to my heart's content. :)

As I understand it, both research and evaluation ask what is. Then research asks why, which indicates a theoretical basis, whereas evaluation asks what should be, which makes reference to the standards and criteria of evaluations.