Certificate of Excellence - Foundations of Impact Evaluation
In cooperation with the United States Department of Energy, AESP Institute is proud to launch an in-depth training program on DSM program impact evaluation. Upon completion of all necessary coursework and meeting other requirements, participants will receive AESP’s “Certificate of Excellence – Foundations of Impact Evaluation.”
The learning objectives of the certificate program are:
· Understand the demand side management industry, including its basic purpose, components, and function.
· Identify the purpose of evaluation, measurement and verification.
· Understand and apply foundational program impact evaluation knowledge that grounds and guides professional practice.
· Identify and define program impact evaluation methodology terms.
· Demonstrate the ability to frame impact evaluation questions.
· Determine appropriate impact evaluation methods, including quantitative, qualitative and mixed methods.
This course covers the basics of the energy industry and demand-side management industry, before diving into the principles of program impact evaluation.
In this cohort-based program, participants enroll at the same time as their peers and experience the learning journey together (communicating via an online discussion board and eventually meeting in person), culminating in a final project which they complete as a team.
Individually, participants will be required to complete two courses and a Project. The first course will be online and self-paced. The second course will be a two day in-person course. The Project will be introduced during the in-person course.
Waltham, MA - April 8-9, 2019
1. EE Programs
- EE programs, their basic purpose, components, and functioning
- History of EE programs and why EE programs were originally created
- Basics of the utility business including load profiles, obligation to serve, load duration curves, load factors, rudimentary rate making (which includes differentiation between fixed and variable costs)
- Different types of EE and DER programs and purposes of each, savings taxonomy (e.g. portfolios, programs, projects, measures)
- Compare demand-side and supply-side resources, IRPs, and common cost-effectiveness assessments (e.g. California Standard Practice Manual and National Standard Practice Manual)
- Types of customers and participants (i.e., various utility sectors and their customer characteristics)
- Players within the EE program sphere and the political realities of each
- Types of regulatory bodies and how they interact with EE programs
2. Program Evaluation in General
- Foundations of program evaluation e.g. standards, guidelines, principles, competencies, approaches, and theories.
- Key program evaluation concepts e.g. evaluability, impacts, processes, worth, etc.
- How evaluation and EE programs interact; how evaluation timing may affect a program
- The different goals of evaluation
- Types of evaluation associated with EE/Demand Response/Distributed Energy Resource programs e.g. impact, process
- Evaluation standards/guidelines e.g. the JCSEE standards and the AEA guidelines
- Roles and relationships of the evaluator in the program design/implementation/evaluation cycle, in regard to regulators and program administrators
- What is included in evaluation frameworks
3. EE Impact Evaluation
- Methodology that grounds inquiry in program evaluation practice
- Key EE impact evaluation terms such as effective useful life, net and gross energy, and non-energy impacts
- Various impact designs, when to apply them, and tools of each e.g. quasi-experimental, experimental designs
- Counterfactual, and how baselines are related to the counterfactual
- EE evaluation resources e.g. SEE Action publications, Technical Reference Manuals (TRMs), etc.
- Relationships and differences between impact evaluations and cost effectiveness evaluations, process evaluations, or market evaluations
4. Key Impact Evaluation Methods and Tasks
- Appropriate methods, including quantitative, qualitative, and mixed methods (to the extent practical, absent significant professional experience)
- How to plan an impact evaluation and the tasks that must occur prior to choosing or implementing any impact method e.g. research goals, questions, etc.
- How a sampling frame can affect an impact evaluation
- Different types of impact evaluation approaches/methods along with their uses, strengths, limitations, risks, and relative costs
- Important tasks within four key impact methods
- Complications that may arise during each task of estimating energy savings
- Savings persistence and savings interactions
- Relationship between evaluation costs, accuracy, and timeliness
- Relative precision, absolute precision, and accuracy, as well as the differences in each
5. Framing the Evaluation
- Appropriate evaluation purposes
- Evaluation questions
- How evaluation goals and research questions support choice of impact method(s)
- What a report should include and how to structure a report depending on the audience
- How to write about sampling and non-sampling errors within a report
The program is designed with new evaluation staff in mind, those with less than 3 years’ experience in the energy industry. Its alternatives are a formal in-house training program, or more commonly “learning-on-the-job” where a supervisor or colleague guides new staff over several months or years.
Ideal candidates for this program include new staff entering a research and evaluation function in DSM consultancies, consultancies that specialize in research and evaluation, government, state and regional energy organizations and large utilities with in-house evaluation needs (National Grid, etc.).
A new cohort will meet November 4-5 at the National Grid offices located in Waltham, MA.
As mentioned before, the course will be delivered in two parts:
- Course 1: Delivered online. Self-paced.
- Course 2 and capstone project: Course participants will meet in-person for 2 days in a city and date that will have been pre-determined before the start of the course. Here they will attend live lectures to complete Course 2, then work collaboratively with a team to complete the project.
For more information, contact firstname.lastname@example.org