Dettenmaier, M., Forestry Extension Educator, Utah State University Cooperative Extension
Dupéy, L.N., Regional Natural Resources Educator, Forestry, UW-Madison Division of Extension
Kuhns, M., Extension Forestry Specialist, Utah State University Cooperative Extension
McAvoy, D., Extension Assistant Professor, Utah State University Cooperative Extension

ABSTRACT

Demonstrating the value of Extension programming is vital, especially when justifying long-running programs and conferences. We sent an Internet survey to previous conference attendees (n = 1,277) to examine if and how attending an annual conference impacted their professional lives. Most respondents (76%) agreed their professional actions were positively influenced, and 58% agreed the natural resource(s) they work with have been positively impacted because of their attendance. Results from this study demonstrate how precise evaluative tools can demonstrate the value of a long-running Extension conference, improve the capacity and reach of an existing Extension program, and justify the existence of a reoccurring Extension program over time.

Introduction

The idea for a regional restoration-based conference was first implemented in fall 2004 when Utah State University (USU) hosted a conference titled “Managing Aspen in Western Landscapes” at Southern Utah University. This conference was planned by a partnership that included USU researchers and Extension specialists, several county Extension agents, a U.S. Forest Service researcher, and others who felt the need to have a forum for sharing scientific information and experiences on management and restoration of aspen forests in the West. That first conference included one day of indoor presentations and 2 days of field trips in aspen forests. Based on the success of this conference and interest from state and federal land managers for a land restoration conference focused on challenges facing western ecosystems, we developed the Restoring the West (RTW) Conference.

Utah State University Extension Forestry and several partners held the first RTW Conference in September of 2006 and every year following (14 years total, 12 years of data is included in this paper). Themes have varied widely, from aspen ecology and management, to restoring sagebrush ecosystems, to climate change, to fire (Appendix A). The audience has included state and federal land managers and resource professionals, landowners, researchers, and students. Since 2006, we have averaged 181 attendees per year. The lowest attendance was 115 in 2008, while the highest attendance was 283 in 2015. Currently, the conference consists of two days of presentations with keynote speakers, a poster session, and an off-campus social reception. We chose the speakers each year based on the specific conference theme, their expertise, recommendations by peers, recent research, and management involvement.

Evaluating Extension programs is essential to documenting their benefits. This is important to Extension’s long-term sustainability as there is increasing pressure to produce and report on the substantive, tangible benefits of publicly funded programs (McClure et al., 2012; Lamm et al., 2013). Evaluation becomes even more important when Extension programs persist for many years, as is the case with RTW (Hatchfield et al., 2013). Conducting follow-up surveys one to five years following the occurrence of an Extension program can help educators measure program quality, quantify benefits (Larese-Casanova, 2018), and measure outcomes related to perceived changes in behavior over time (Koundinya et al., 2016). Follow-up surveys can provide beneficial feedback for Extension practitioners, however, evaluations that measure long-term benefits are rarely documented (Workman & Sheer, 2012).

The RTW planners (and authors of this paper) administer an online survey immediately following every RTW conference to measure short-term impacts and capture perceived changes in knowledge, skills, and abilities as a result of attending the conference. These results provide useful feedback that guides future conference content, themes, and logistics. While this information is useful, we were interested in understanding the longer-term impacts resulting from attending RTW, so in 2018 we surveyed RTW attendees spanning the 12-year period from 2006 to 2017. We did this with an online survey that attempted to measure broad, long-term impacts of RTW attendance for all previous attendees. Our findings demonstrate the value and importance of conducting long-term evaluations on continuing Extension programs.

 

Purpose and Objectives

The RTW Conference mission is to bring together resource professionals, landowners, academics, and land managers to facilitate collaboration on pertinent natural resource management and restoration issues. The 2018 follow-up survey measured perceived impacts from attending this conference. The purpose of this study was to determine if and how RTW conferences have impacted our attendees. Specifically, we aimed to:

• assess if the goals of RTW were reflected in attendee experiences,
• assess the specific areas of attendees’ professional lives that were impacted by RTW attendance,
• estimate the amount of land that attendees managed or impacted,
• determine if attendees perceived a positive change in the condition(s) of the natural resources they managed or studied due to their attendance, and
• determine the value of conducting follow-up surveys to evaluate long-term Extension program impacts and compare that to single-event surveys administered shortly after a program.

 

Methods

The 12 RTW Conferences that were included in this study had a total of 1,996 registrants, or 1,277 registrants with duplicates removed. In our outreach emails we indicated our desire to evaluate the conference and to determine if it was meeting the needs of attendees. We included a link to an online Qualtrics survey and asked them to take the survey (Appendix B). We sent the first email on May 21, 2018; 20 of those emails bounced and 296 surveys were started. On May 28, 2018 we sent the email to 1,172 registrants; 13 of those emails bounced. We sent a final email to the remaining 1,029 registrants on June 14, 2018; 23 of those emails bounced. We sent the survey in three batches to remind people to take the survey and to give everyone a chance to complete the survey if they had started it but had not completed it (Qualtrics has unique links for each person that takes the survey which allows users to completed partially finished surveys instead of starting over). Of the 1,221 valid surveys sent, we received 175 completed surveys, representing a 14.3% response rate.

 

Results

Did Attendee Experience Reflect RTW Goals and Objectives?

We asked respondents to rate how strongly they agreed or disagreed that the objectives of RTW were met in the conference(s) they attended. While some (19%) were neutral or disagreed, most (81%) agreed or strongly agreed that the conference successfully “brought together resource professionals, landowners, academics and land managers to facilitate collaboration on pertinent natural resource management and restoration issues” (Figure 1).

 

Figure 1. Proportion of respondent’s indicating different levels of agreement with the statement “The mission of Restoring the West was met in the conference(s) I attended”.

 

 

We asked the question: “In what way has your attendance at past RTW Conferences influenced your professional actions? (e.g., management decisions, research, teaching, monitoring protocols, treatment prescriptions)” (Figure 2). The majority (76%) of respondents reported that attending RTW had a positive influence on these professional actions.

 

Figure 2. Responses to the question “In what way has your attendance at past RTW Conferences influenced your professional actions? (e.g., management decisions, research, teaching, monitoring protocols, treatment prescriptions).

 

 

To investigate this topic further we asked about specific ways RTW impacted respondents’ professional actions. Most (81%) agreed that RTW generated topical discussion and nearly two thirds (64%) agreed that RTW facilitated collaborations (Figure 3).

 

 

Figure 3. Levels of agreement with the statement: "Please rank how strongly you disagree or agree with the following statements as they relate to your professional life. My attendance at RTW: Facilitated collaborations, Generated topical discussion, Influenced my management or research decisions, Established inter/intra-agency relationships, Improved inter/intra-agency partnerships”.

 

 

Did Attendance at RTW Positively Impact Natural Resources?

We asked respondents if they agreed that their attendance at RTW positively impacted the condition of the natural resources they work with (Figure 4). Approximately 58% percent indicated they somewhat or strongly agreed with this statement, 34% were neutral, and 8% either strongly or somewhat disagreed with this statement.

 

 

Figure 4. Respondents’ levels of agreement with statement that “My attendance at Restoring the West positively impacted the condition of the natural resource(s) I work with”.

 

 

How Many Acres Were Managed by Attendees Decisions?

We asked two questions to determine the number of acres impacted by respondents’ attendance at RTW. First, we asked respondents to estimate the amount of land they work with to gauge approximately how many acres may be impacted due to the existence of RTW (Table 1). Next, we examined the data from the 101 individuals that indicated their attendance at RTW positively impacted the natural resources they worked with and combined the acres they manage. The most conservative estimate of the number of acres managed by this group of respondents is over 229 million acres.

 

 

Table 1. Amount of land that survey respondents work with

Number of acres	Number of respondents	Proportion of respondents
None	35	20%
Less than 1,000	10	6%
1,000 to 100,000	29	17%
100,000 to 1 million	36	21%
1 to 20 million	49	28%
More than 20 million	14	8%

 

 

Limitations

Our study has some limitations that future studies could address to enrich and improve the ability to make inference and draw connections between participating in an Extension program and public benefits. We had a lower than desired response rate (14%). The low response rate in our study could have been improved by reducing the number of survey questions. Inserting an assurance that a survey only has 4 questions (or will only take 4 minutes) in the subject line of an email containing a survey is one tactic that has helped us increase survey response rates in the past. The low response rate prevented us from being able to analyze survey data by conference year; instead we had to analyze responses over the life of the conference. If this survey was replicated and had a higher response rate, Extension professionals could:  

• investigate if differences exist between people that attended for multiple years and one-time attendees,
• evaluate if and how perceptions of benefits from attending RTW changed over time, and
• assess how respondents’ professions and/or the amounts of land they manage impacted their answers.

We asked questions in a way that emphasized perceived behavior changes resulting from RTW (e.g., my attendance at RTW had the following impact on my areas of work). Therefore we cannot say with certainty that an actual impact or change in condition was observed, instead, we rely on the perceptions of the attendees. Our study demonstrates the value of conducting long-term surveys for reoccurring programs as such surveys provide rich data for program administrators and professionals.

 

Conclusions

Our findings have several implications for Extension programming and program evaluation.

1. Long-running Extension programs can have long-term impacts; however, the only way to quantify these impacts is to administer multiyear follow-up surveys. While somewhat coarse, the estimated impacts from RTW are substantial. Extension professionals may find value in conducting similar multiyear surveys to measure outcomes and perceived benefits, especially for annual programs.
2. Most respondents agreed that the RTW objectives are being met. However, there is a need to improve the experience of the 19% that were neutral or disagreed. To reach this group, we plan to provide multiple open-ended questions on all future surveys to understand how to better meet these objectives. Understanding how to improve experiences for people who engage with Extension is important information. Positive experiences can lead to continued engagement from the public. If improving the experiences of those engaging with Extension is a goal, we suggest incorporating similar questions to program evaluations.  
3. We conducted this survey 12 years after the first RTW. Combining brief annual surveys with follow-up surveys can further clarify if an Extension program is meeting its established objectives. To encourage long-term attendance and increase survey response rates, it may be useful to conduct follow-up surveys that include in-depth questions at shorter intervals (i.e., every 3 years instead of every 10). This approach would allow us to combine the annual feedback with broader three-year data when planning future programs. Also, conducting long-term surveys more frequently increases the likelihood that attendees will recall specifics from the conferences in question and may increase survey response rates. We suggest adopting this schedule for evaluating annual programs or conferences. Long-term surveys can provide important information to Extension practitioners and can justify the existence of specific programs.   

Our study demonstrates the professional benefits realized by past RTW attendees. RTW brings together professionals from across the region to discuss and collaborate on pertinent natural resource issues. Regional conferences allow for conversation and collaboration on landscape-scale issues – the scale at which these ecosystems and issues operate.

 

References

Hatchfield, G. A., Bau, D. B., Holcomb, C. R., & Craig, J. W. (2013). Multiple year Extension program outcomes & impacts through evaluation. Journal of Extension, 51(1), Article 1FEA2. Available at: https://www.joe.org/joe/2013february/a2.php

Koundinya, V., Klink, J., Deming, P., Meyers, A., & Erb, K. (2016). How do mode and timing of follow-up surveys affect evaluation success? Journal of Extension, 54(1), Article 1RIB1. Available at: https://www.joe.org/joe/2016february/rb1.php

Lamm, A.J., Israel, G. D., & Diehl, D. (2013). A national perspective on the current evaluation activities in Extension. Journal of Extension, 51(1), Article 1FEA1. Available at: https://www.joe.org/joe/2013february/a1.php

Larese-Casanova, M. (2018). The importance of evaluating long-term impacts: Utah Master Naturalist Program as a case study. Journal of Extension, 56(6), Article 6RIB3. Available at: https://joe.org/joe/2018october/rb3.php

McClure, M. M., Furhman, N. E., & Morgan, A. C. (2012). Program evaluation competencies of Extension professionals: Implications for continuing professional development. Journal of Agricultural, Extension 53(4), 85-97.

Workman, J. D., & Scheer, S. D. (2012). Evidence of impact: Examination of evaluation studies published in the Journal of Extension, 50(2), Article 2FEA1. Available at: https://joe.org/joe/2012april/a1.php

 

 

Appendix A.

Restoring the West Themes and Attendance by Year

Year	Theme	Attendance
2017	Forest Restoration: What’s Working, What’s Not?	160
2016	Climate, Disturbance, and Restoration in the Intermountain West	156
2015	Restoration and Fire in the Interior West	283
2014	Down by the River: Managing Resilient Riparian Corridors	234
2013	Change Agents and Managing for Forest Resilience	154
2012	Balancing Energy Development and Biodiversity	151
2011	Sustaining Forests, Woodlands, and Communities through Biomass Use	152
2010	Managing Plant and Animal Conflicts	168
2009	Peaks to Valleys: Innovative Land Management for the Great Basin	205
2008	Frontiers in Aspen Restoration	115
2007	Sagebrush Steppe Restoration	218
2006	Aspen Restoration	N/A

 

 

Appendix B.

Survey Questions, Response Categories, and Answer Choices in Survey Instrument Administered in May 2018, to All Past Restoring the West Conference Attendees

Survey question	Response categories	Answer options
To the best of your ability, please indicate which years you have attended the Restoring the West (RTW) Conferences.	2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017	Yes No Unsure
In what way has your attendance at past RTW Conferences influenced your professional actions? (e.g., management decisions, research, teaching, monitoring protocols, treatment prescriptions)		Strong negative influence Some negative influence Neutral Some positive influence Strong positive influence
Please rank how strongly you disagree or agree with the following statements as they relate to your professional life, interactions with other attendees, interactions with other people, or interactions with groups back home. My attendance at RTW:	Facilitated collaborations Generated topical discussion Influenced my management and/or research decisions Established inter- or intra-agency partnerships Improved inter- or intra-agency partnerships	Strongly disagree Somewhat disagree Neutral Somewhat agree Strongly agree
Please indicate how strongly you disagree or agree with the following statement: My attendance at RTW positively impacted the condition of the natural resource(s) I work with.		Strongly disagree Somewhat disagree Neutral Somewhat agree Strongly agree
My attendance at RTW had the following impact on my areas of work or interest; select “Don’t work with” if the area does not apply:	Riparian resources Rangeland resources Forest resources Fire Human impacts on resources Biomass resources Restoration Climate change Wildlife resources	Don’t work with Conference didn’t address this topic No impact Slight improvement Moderate improvement Significant improvement
Approximately how much land do you manage, work with, or impact (e.g., manage, conduct research on, etc.)? If this number has changed over time, select the amount you currently manage.		None Less than 1,000 ac 1,000 to 100,000 ac 100,000 to 1 million ac 1 million to 20 million ac More than 20 million ac
The mission of Restoring the West is to bring together resource professionals, landowners, academics, and land managers to facilitate collaboration in natural resource management and restoration. Please rate how strongly you agree or disagree that the above objective was met in the RTW conference(s) you attended?		Strongly disagree Somewhat disagree Neutral Somewhat agree Strongly agree

 

---