Evaluation Baron

A common response as I discuss evaluation with people and organizations is that they don't see a real need, or that they cite economic woes as a reason to ignore evaluation. However, here's some of the translation of what they're really saying:

"I don't care how my business is doing. I just want to keep making widgets."

"I don't know a thing about evaluation, so maybe my problems will go away by themselves."

"Evaluation is too much work. I'm busy enough as it is."

The fact is that our current economic state, organizations cannot afford to ignore the need for evaluation. Federal and state governments are focusing on evaluation to root out inefficiency. Non-profit organizations must continually be accountable to funders. For-profit organizations must increase the efficiency of their business processes in order to be ahead of the competition.

All of the above is accomplished by evaluating programs and services to align what should be happening (via criteria and standards) with what is happening. The results may then be used for decision making to make those programs and services more refined, more productive, and more serving of beneficiaries.

Take the time, and make the time to evaluate. The status and growth of your organization depends on it.

The Evaluation Baron, LLC, can introduce you to evaluation concepts and how they will benefit your organization. Contact us for a free initial consultation!

There have been many reports of the growing need for educational reform and its effect on the global economy. Many leaders in education and government are also voicing their concern and stressing the need for accountability, visibility, and evaluation of federal programs (see Obama Administration Launches Program Evaluation Effort at http://www.govexec.com/dailyfed/1009/100709e1.htm). Science, Technology, Engineering, and Math (STEM) education is no exception.

Recent news reports of the reform of STEM education (see The State of Stem at http://www.techlearning.com/article/26400) recommend integrated teaching of the STEM components in order to enhance learning effectiveness. "An intriguing finding was that engaging young people in hands-on projects in engineering and design provides effective ways for them to learn core math and science concepts."

These are some of the benefits of evaluation: to assess the impact and effectiveness of teaching STEM components separate or together; to understand the impact of STEM pedagogy on the global economy, and to maximize the role of teacher effectiveness in this process. Evaluation provides visibility of these vital decision-making tools!

These issues are not without political unrest, though. Nevertheless, teacher unions, administrators, parents, and the students themselves must come together on this and other educational issues. Being able to evaluate and use the resulting information helps individuals, schools, and communities. A first step is to facilitate evaluation of STEM programs in the classroom. Together, we can foster a competitive nation and world (see Universities Pledge to Train Thousands More Math and Science Teachers by 2015 at
http://chronicle.com/article/Universities-Pledge-to-Train/63439/?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+chronicle%2Fnews+%28The+Chronicle%3A+Top+Stories%29&utm_content=Google+Reader)
- one teacher, one student, and one classroom at a time. The key is to evaluate!

The Evaluation Baron, LLC can help you in this process. Contact us for a free initial consultation!

1. Accrediting schools
2. Judging the quality of education or corporate training curriculum
3. Judging the effectiveness of a training program
4. Improving the recruitment process of volunteers or employees
5. Improving a commercial product
6. Improving employee retention
7. Determining public perception of a new policy or program
8. Understanding the cost-benefit, cost-effectiveness, or cost-utility of an employee benefit package
9. Deciding whether to implement a community development program
10. Developing greater accountability to project funders

The Evaluation Baron, LLC, can help you understand the benefits of these types of evaluations, and what it means to your organization specifically. Contact us for a free initial consultation!

A recent news article from the Christian Science Monitor, "Eyeing Stimulus Money for Education, States Adopt Reforms (posted Thursday, December 3, 2009 at http://www.eschoolnews.com/funding/funding-news/?i=62081), highlights the need for evaluation in making progress with education reforms.

In particular, the article points out four areas where states could receive grant money: "Establishing data systems that track students' progress from preschool through college, developing and using rigorous standards and assessments, improving teacher effectiveness and the distribution of high-quality teachers, and turning around the lowest-performing schools." The article states that the last two issues have not been addressed much because of the obscure nature of defining teacher effectiveness and developing a system to improve schools.

That's what evaluation is all about. Organizations work with professional evaluators to define topics such as teacher effectiveness, establish criteria and standards for what should be happening, and then take steps to evaluate whether or not those things are happening. Organizations may then choose to implement recommendations generated from the study to enhance and improve the program.

While creating standardized definitions, criteria, and standards for schools across the country will certainly be a more involved process with all levels of education, schools can and should start somewhere by evaluating what is working or not working in school reform (to at least establish a baseline), and then proceeding with strategies for improvement.

National, state, and local organizations must work hand in hand in order for true reform to succeed. Schools would do well to look at this grant offering as an opportunity to evaluate their current situation and then take steps toward improvement.

The Evaluation Baron, LLC can help you evaluate how your school, district, or state is doing in any of these topics. Contact us for a free initial consultation!

Recently I attended a one-day conference, during which time I literally heard 14 different terms to denote evaluation. Let me briefly share them with you, along with a simple explanation of each one.

1. Evaluation: Judging the merit or worth of something, such as a program, process, product, etc. Yes, it was only a few times, but I did hear this word...like a sweet melody ringing in my ears.
2. Analysis: An investigation of data pertaining to a situation, or the situation itself.
3. Assessment: This term is usually used within the context of educational testing, but is generally considered a subset of evaluation.
4. Find Out: Although simple, what logically follows from this phrase is the question of how and why are you going to find out...for what purpose and by what means.
5. Observation Checklist: This is one of many tools used with data collection to make sense of what is being observed, such as a class, a performance, a group interacting somehow, etc.
6. Tracking: This term denotes the monitoring side of evaluation, which is an ongoing collection of data on a given topic, usually to determine compliance.
7. Review: To make an appraisal of data or of a situation, such as a review of an existing program, etc.
8. Key Indicators: This phrase is synonymous with criteria and standards (i.e., what should the program be doing, and to what standard should it occur).
9. Measuring Success/Measurement: This term is also considered a subset of evaluation, and often pertains to quantitative data, although measurement itself can be in quantitative or qualitative terms.
10. Performance Improvement: This term is often used in the human resource or healthcare fields to denote such things as employee productivity and wellness.
11. Feedback: Reflective of customer service, feedback can be opinions or other information about your programs. Feedback can come from both internal and external sources.
12. Rate: This term denotes a judgment of quality, and is often seen in surveys (e.g., "On a scale of 1-5, rate the effectiveness of the following...").
13. Accountability: Evaluations help to establish accountability for organizational performance. Evaluations link performance criteria to results to show funders or other stakeholders the efficiency, effectiveness, and return on investment of a program, product, or process.
14. Harvesting: This term denotes the data collection and analysis phase of an evaluation, and can be by various means (e.g., web analytics, documentation review, surveys, interviews, data mining, ethnography).

Let us know how you feel! Tell us what term you affiliate with, and how you use it (or want to use it) in your organization!

No matter what term you use, The Evaluation Baron, LLC can help you increase the effectiveness and efficiency of your programs, products, or business processes by aligning results with your organizational goals.

Contact us for a free initial consultation!

Over the last decade, and more recently with the economic roller coaster we've been on, organizations are becoming more self-sufficient. We're seeing this evidence via corporate downsizing, professional development, and a structure of learning taking place in companies worldwide.

Along with self-sufficiency in business strategy planning, marketing, manufacturing processes, and retail production and distribution should bring the evaluation of these aforementioned tasks. The development of an organization's capacity to plan, conduct, and use evaluation is crucial to not only the success of individual programs, processes, and tasks, but also the overall success of the organization as a whole. This applies to all disciplines (e.g., non-profits, NGOs, for-profits, education, government, military).

Although many definitions of evaluation capacity building (ECB) now circulate the globe, I believe the simplest working definition can be found in the writing of Stockdill, Baizerman, and Compton (2002), where they define ECB as, "the intentional work to continuously create and sustain overall organizational processes that make quality evaluation and its uses routine" (p. 14). Here evaluation becomes the norm rather than the exception. Evaluations themselves and the corresponding structure become transparent within the organization. All hierarchical levels in the organization - from the clerical staff to the team leaders to the CEO - understand and capitalize on the importance of evaluation and how it affects business processes and the accompanying bottom line. Creating evaluation capacity with any type of organization involves not only evaluators, but also the organization itself. Both parties need to take a proactive mindset to create the organizational structure needed to support evaluation capacity.

As an organizational leader, think of evaluation capacity building as creating evaluation self-sufficiency within your organization. You not only need to see the importance of
evaluation in the short and long term, but also develop the skills to conduct evaluations and use the results of such to further your business vision.

A good place to start is to think about the topics listed in A Checklist for Building Organizational Capacity (Volkov and King, 2007) made available through The Evaluation Center at Western Michigan University. Discuss these principles with your colleagues and how you can implement them in your daily practice. There are many resources available to assist you, such as evaluation consultants, professional development opportunities, etc.

The first step is to understand how becoming self-sufficient in evaluation can improve your organizational mission, vision statements, goals, and ultimately the bottom line. Then take additional steps to see that goal become a reality.

• You launch a new website for your organization. How do you know if anyone has seen it, is using the information on it, or if it's effective in gaining new customers?
• You develop a new employee-training program to be used at all your area offices. How do you know if employee performance has improved because of that program, or what employees think of the program?
• You received state funding for your community literacy program. How will you know if the program achieved its goals? How will you account for the program effectiveness to the funders?

The answer to all three of these scenarios is that you don't know unless you evaluate the progress or impact of those programs.

Evaluation judges the merit or worth of something, whether that something is a program, product, process, performance, or policy. Evaluation helps you determine its impact on your organization, the community, and the people involved. Evaluation can guide your strategic planning. Evaluation can save you time and money.

As an organizational leader, consider the following questions:

1. In these tough economic times, how can I determine which programs work and which I should consider down-scaling or canceling?



2. How do my organization's activities coincide with short, medium, and long-term goals?



3. How can I get greater return on investment for the quantity and quality of work that goes into this organization?



4. How can I ensure my programs are cost effective?

These questions are just the first step in helping your organization become more effective. To answer these questions, you can seek out information on how to plan, conduct, and analyze evaluations. You can work with an external evaluator, or learn how to develop your organization's capacity for conducting internal evaluations.

Evaluation equals understanding and decision-making power.

So what type of areas or topics can web analytics assist you with? Here are some examples from the online learning field:

• Ensuring class members complete online assignments
• Discovering access or informational problems with a website
• Finding out if others are benefiting from the information on your website (with reference to open learning as applicable)
• Obtaining feedback from users regarding content, usability, and applicability - and being able to make adjustments accordingly
• Assessing the need for certain courses or topic areas
• Learning who the most frequent users are, why they participate, and how they can propel your courses to success

Web analytics, an up and coming technology that evaluates website activity, can be a huge benefit to all organizations, whether you have a small website or a large e-business. Here's how web analytics can improve your organization:

• Find out (among other vital data) who is on your website, for how long, and what they look at most
• Based on the above information, you can cater your website you specific groups of users and you can provide them the vital information that may mean the difference between having them do business with you and having them go elsewhere
• External evaluators (i.e., those who help you make sense out of the web information), can learn more about how customers and potential customers feel about your organization so they can make accurate and appropriate recommendations for improving your organization.

During the past few years, I've had an amazing experience conducting evaluations with an academic support group on the Brigham Young University (BYU) campus. The Center for Teaching and Learning (CTL) strives to “improve student learning in the context of the Aims of a BYU Education. The center accomplishes this goal by assisting BYU faculty, instructors, and administrators to -

• refine effective program- and course-level learning outcomes,
• design learning experiences to achieve desired outcomes,
• enhance the quality of learning experiences through technology integration,
• evaluate the effectiveness of learning experiences, and
• advance knowledge of effective learning and teaching.”

The evaluation field in general loves to debate and discuss the important issues surrounding evaluation. One longstanding topic (if there is anything more longstanding than the quantitative/qualitative debate :) is the difference between evaluation and research.

There are many opinions, and there probably is no right or wrong answer, but since this is my blog, I get to state my opinions and experiences to my heart's content. :)

As I understand it, both research and evaluation ask what is. Then research asks why, which indicates a theoretical basis, whereas evaluation asks what should be, which makes reference to the standards and criteria of evaluations.