Systems in Evaluation TIG
February 2018 Newsletter
SETIG Principles Project Update
Heather Britt and Marah Moore, SETIG Co-Chairs
In 2017, the Systems in Evaluation TIG (SETIG) launched the Principles Project to develop guiding principles for the use of systems concepts in evaluation. (See October newsletter for important background information).
TIG Members Create Draft Principles in time for Annual Conference
Ten TIG members collaborated with the Leadership Team and the Principles Project Coordinating Team over a 4-month period to draft principles for applying four key systems concepts in evaluation: interrelationships, perspectives, boundaries, dynamics. Michael Quinn Patton launched the group effort with a presentation on crafting useful, inspiring, and evaluable principles. Volunteers joined one of four small teams to delve deeply into one of the four core systems concepts. Each group shared their progress in a second webinar, and Bob Williams provided input on the interrelationships between the core concepts. Volunteers reviewed the drafts of each team. The group effort concluded with a webinar discussion on progress, the process, and next steps.
Although each principle is described individually, they only make sense in relationship to one another. Volunteers outlined several ways the document can best represent the relationships between all the principles. The group also outlined the key points that should be included in the document’s preamble.
During the annual AEA conference, the Principles Project benefited from TIG member input and a session piloting their use in proposal review.
TIG Members Provide Encouragement and Feedback at Business Meeting
The SETIG Business Meeting at AEA 2017 was an important event for the Principles Project. We were fortunate to have a good turnout. A good number of TIG members (longstanding and new) brought their energy and insight to our late evening meeting. We want to thank them all.
Attendees provided feedback in four areas: 1) how people felt about the principles and the process; 2) what people thought about the content of the principles; 3) what people thought the next steps were for the project; and 4) how people were willing to help. A summary of the main comments is presented below around the four areas that were addressed:
TIG members shared very encouraging comments about how people feel about the principles. Attendees were enthusiastic about both the principles and member engagement in their development, using words like “inspired”, “grateful”, “breakthrough” and “cool.” Attendees reported that the principles may enhance clarity in communicating key systems concepts and increase evaluators’ confidence in using concepts and approaches in their practice.
Attendees provided insightful feedback on the content of the principles. Comments confirmed that the document must describe the ways that systems concepts relate to one another and work together. A number of comments provided helpful suggestions for communicating the key concepts more accurately and clearly.
Attendees provided good suggestions about next steps. Members agreed that drafting the document’s preamble is a top priority. TIG members described different ways that they would like to use the principles, and made suggestions on how to disseminate them. Other great ideas emerged, including convening an online event around the principles, engaging people outside the systems field for testing the principles, applying them to case studies, developing checklists, rubrics and tools, or even writing a paper. The Principles Project Coordinating Committee will consider all feedback as we move forward.
Many thanks to the people who raised their hands in the “what can I do to help this move forward” section. .If you would like to get involved just drop a email to any of the members of the Leadership Team or write to firstname.lastname@example.org.
Strong Turnout to Pilot the Principles
Between 70 and 80 people showed up on Saturday morning to pilot the principles; more than 60 stayed until the end. Wow! The strong turnout reinforced the interest our Principles Project. While the enthusiasm was high, the 45-minute session didn’t provide enough time to complete the task properly. Even with the time constraints, participants provided helpful feedback on the experience of using the principles.
Six groups of participants reviewed three anonymized AEA conference proposals and scored them against the draft principles. Each group read and scored a proposal similar to those submitted to the TIG for inclusion in the annual conference program. The three proposals, represented a range of alignment with the principles (good alignment, moderate alignment, and weak alignment). Observations from reviewing the input from all six groups included:
- Scoring proposals was not consistent. For example, at least two groups ranked the
same proposal quite differently. Furthermore, the proposal we felt represented good alignment with the principles was scored low.
- Several groups commented that it considering each principle one-by-one leads to siloing and not the “holistic” approach they associate with “systems thinking.” We’ll need to find ways to address the way the principles work together as we move forward.
- One stated purpose of the principles is to improve the quality and fit of proposals submitted to the TIG. There were comments during the session that this was difficult to do, given the limited information that is available in a proposal. Also, somebody expressed the concern that some types of proposals (for example, those addressing capacity building around using systems thinking in evaluation) might not clearly fit the way the principles are worded. We will need to consider how to incorporate the principles into the entire proposal process, beginning with communicating them with proposal submission guidance, and continuing through the review and selection process.
- There were also questions about how the principles will be shared and what other purposed they will serve. After the preliminary draft is ready to be shared with the wider TIG membership, we’ll need to give careful thought to dissemination and use.
Principles Ready for Release in Spring 2018
Since the close of the conference, the Leadership Team, Coordinating Team, and volunteers have drafted the preamble and explanation of how each principle meets key criteria (using the GUIDE framework). Using the ideas developed before the annual conference, members of the working group are drafting content to explain the way the principles work together. Once that task has been completed, the document will be reviewed for clarity before sharing with the wider TIG membership in spring 2018.
State of the TIG 2017
The Systems in Evaluation TIG membership reached 400 in 2017. Membership has been steady in the previous three years at 374, 363, 379 in 2014, 2015, and 2016 respectively.
The TIG received 29 proposals in 2017, a number similar to the previous two years. Twenty-five volunteer reviewers participated in the blind peer review process. These volunteers help ensure the quality of the TIG program and allows our TIG to provide each submitter tailored feedback. The SETIG 2017 program included 19 sessions and 7 posters.
Thank you to reviewers, presenters, and attendees!
AEA management set the 2017 acceptance rate at 75% for all TIGs. This was down slightly from 80% in 2016 because 7:00 AM sessions were eliminated in response to conference attendee feedback. Once the review process was concluded and the ranked proposals were submitted to AEA, the management was forced to reduce the acceptance rate by another 10% to accommodate available rooms in the conference venue. The final SETIG acceptance rate was 66% (not including Ignite, Birds of a Feather and posters).
AEA is growing! What does it mean for our TIG?
Every year, the AEA management team hosts the TIG Leadership Breakfast on the Thursday morning of the conference. The growth of the association was a key topic of conversation at this year’s breakfast. AEA membership has increased steadily over the years to about 7,500 members total. Conference attendance is increasing at a rate that exceeds predictions. This is exciting, but it has also brought challenges for conference management. We are at the limit in terms of conference venues, and the number of hours in the day folks are willing to sit in presentations. Consequently, conference attendees may have noticed overflowing rooms, insufficient refreshments at receptions, and other visible signs that we are nearing the limits of what our pre-signed venues can accommodate. In addition, the AEA management team reports scrambling to accommodate larger than predicted numbers of last-minute attendees.
Inclusion and Quality. As both membership and attendance increase, the association faces important decisions about its values and character. These issues are more important than inconveniences of having to stand through a session or go without a snack at the reception. One of the most important debates is whether to emphasize quality or inclusion in the conference. Many members are unable to get leave or funding from their employers to attend if they are not presenting. Also, presenting is an important professional development opportunity for graduate students and new practitioners alike. If the proposal selection is more restrictive, the association may limit the important role it plays in this area. On the other hand, keeping acceptance rates high has budget and scheduling implications that few wish to live with.
Looking for solutions. The AEA management team is considering various ways of constructing the conference program to address these challenges. One option would be to allot each TIG a percentage of rooms and timeslots and the TIG Program Chairs will create their own program. One of the benefits of this approach would be that at the conference each TIG’s sessions would take place in a suite of room. Even as the conference grows, TIG members would have opportunities for informal hallway conversations. This is still an open discussion.
Time to focus on SETIG recruitment? TIGs are the heart of the association and their importance is likely to increase as the association grows. AEA data shows that association members that join TIGs are more likely to attend conferences and remain AEA members. Unfortunately, some AEA member don’t know how to sign up for TIGs. At least some people that attend our SETIG sessions, reception and business meeting have not joined the TIG.
It’s Proposal Writing Time!
Proposals to present at Evaluation 2018 (October 28, 2018 to November 3, 2018 in Cleveland, Ohio) are due March 15, 2018 by 11:59 p.m. US Eastern Time. This year’s theme is Speaking Truth to Power. In the invitation to submit proposals, the AEA leadership posed these questions:
- What responsibilities do we have as evaluators for Speaking Truth to Power?
- In what contexts or situations?
- With what consequences?
- At what risk or cost?
- To whom, with what expectations?
- What is power? Who has it, and how can they best be influenced? What is the power held by evaluators and evaluation?
- What is truth? Whose truth? How can we best discover these truths?
- And, what is speaking? Whispering? Public pronouncements? Influence? Activism? And by whom on behalf of whom?
This topic, and these questions, clearly call for a systems lens as we think about our role and responsibilities as evaluators. Please consider submitting a proposal to the Systems in Evaluation TIG—we look forward to learning with you!
Visit the AEA website to learn more about Speaking Truth to Power and join Leslie Goodyear for a live Coffee Break presentation to hear more about this year’s theme.
Details on the Evaluation 2018 Theme
February 16, 2018, 2:00 – 2:30 p.m. ET
How to Submit
Visit the AEA website to start the submission process. You must be logged into your AEA account to submit a proposal. If you do not have an AEA account, you can easily create a guest account. Creating an account 1) reduces the potential for duplicate records in the system, 2) helps AEA staff avoid double-booking speakers in various sessions, and 3) allows attendees to correctly identify you and connect with you before, during, and after the conference. Accurate information in your profile also reduces the chances for typographical errors in the final program and conference app.
Need More Assistance — Join AEA’s Webinar
How to Submit to Evaluation 2018 – Tips and Guidelines
Thursday, February 15 | 12:00 p.m. – 12:30 p.m.
or Tuesday, February 20 | 2:00 p.m. – 2:30 p.m.
You have two opportunities to join this webinar and learn more about the submission process for Evaluation 2018. AEA will be covering tips and sharing advice to help make the submission process easy. Register Today!
New AEA Executive Director Announced
The American Evaluation Association has selected Anisha Lewis as Executive Director of the association, starting February 19. To former Executive Director Denise Roosendaal, thank you for all the ways you contributed to AEA and we wish you success in your new position! We welcome Anisha Lewis to her new professional home and look forward to working with her! To read more about the selection process and access the full press release, visit AEA’s website: http://www.eval.org/page/new-executive-director-announced.