While the Systems Evaluation Protocol can be implemented as a manual process that does not depend on any specific technology platform, it is designed so that it can be enhanced throughout by using a system developed in our current research called the Netway.
The Netway is a evaluation system that will revolutionize how managers to track all the evaluation efforts across their organizations- overcoming the challeges of geographic dispersal, inconsistent reporting methods, isolated evaluation efforts, and varying evaluation capacity. Incorporting Netway tools and resources into your work results in smarter organizations - by increasing team collaboration and program planning, simplifying documentation and reporting, and developing well thought out evaluation plans that streamline evaluation with program management priorities. The built-in evaluation tool is built from years of research and experience to provide you with proven methods and resources for evaluation tailored to your program priorities.
- One location, one format - quickly track all the program plans within your organization, in one place, in one consistent format that is designed to support you in project management and funding requests.
- Quick and simple reports for any or all your programs in a single report.
- Increase team collaboration and communications
- Integrate program priorities across multiple programs
- Facilitate program evaluation
- Increase collaboration between organizations
- See what others are doing to evaluate similar programs
The Netway provides:
- A tool: A platform for program modeling, evaluation planning, and collaborating. Programs are "groups of inputs and activities that are intended to result in outputs and outcomes" - they lead to change in participants.
- Program Modeling
- software for logic modeling
- software for pathway modeling
- methods for linking measures to your outcomes
- reporting capability
- Collaboration tools
- collaborate on modeling and evaluation planning
- link to similar programs to stimulate your thinking
- connect with peers, join discussions and post faqs related to this work.
- A Set of Resources:
- General pages can be accessed through the "Help" link at the top of the page, or through the "Resources" tab.
- Item specific help may be accessed through the book icon found to the right of many of the available headings.
- Protocol (aka Systems Evaluation Protocol or SEP) - step-by-step guide for evaluation.
- Guidance - worksheets, videos, exercises to help you think about your own program model and evaluation plan.
- Neway Help - provides the "how to" of doing things on the Netway.
What's in the Name?
The term "Netway" is derived from the phrase "networked pathway". The system assumes that any program or project can be described with its own pathway model (a causal logic model), but these models are likely to be relatable - they are likely to share common activities, outputs or outcomes. The Netway is constructed so that when project educators and managers enter project information about activities, outputs and outcomes, the system can immediately identify and suggest other existing projects that have similar or common elements and enable the users to adopt or adapt these elements for their own projects while automatically creating networked linkages of their models with others. Each new project model adds to the online network of such models and can be accessed in turn by others. This also helps ensure that different parts of the system can learn from each other and that even projects with no direct contact with one another, can use the cyberinfrastructure to benefit from each other's experiences.
The Netway was initiated in 2005 as an online tool in our work with 4-H at Cornell Cooperative Extension (CCE) in New York City. The task was to developing evaluation plans and implement evaluation for their outreach programs - the challenge was to make it easier. The Netway was created as a tool to help educators build program models and develop evaluation plans, and was initially intended to be a Participation and Activity Tracking System of evaluation data for the CCE system. However, out of that work we found a greater need for evaluation capacity building across the system, rather than a need for a system to evaluate program system-wide. Although system-wide evaluation was a goal, planning on how to use evaluation data prior to having the capacity to plan and conduct evaluations was putting the proverbial horse in front of the cart.
In October 2006 we received NSF funding (NSF Award #0535492) , "Building Evaluation System Capacity for STEM Programs". Out of this work came the beginnings of the Systems Evaluation Protocol (SEP, or Protocol), and the Netway as a modeling tool that was used for online collaboration, developing program logic and pathway models, and drafting evaluation plans. Partnering with nine organizations, and 2-4 programs within each organization, each program worked toward developing a well-articulated evaluation plan that would address the program's and stakeholders' goals and objectives. We worked with one person from each organization (who we called the EPM - Evaluation Project Manager), who took the SEP and Netway back to their organizations for evaluation planning. Notable conclusions from this phase of project development included: a systems perspective has the potential to significantly enhance evaluation, practitioners began to view evaluation as an on-going process throughout the life of a program, and could provide key feedback useful for program development (not just end-of project evalution), and the different evaluation methods are suitable at different stages in a program's lifecycle. We found that use of a formalized protocol was beneficial to our participants, and that the partnership with them - we as evaluation experts and them as the program experts - was effective in building good evaluation. And the Netway was a key tool that was both efficient and useful in their work.
The Systems Evaluation Protocol (referred to as either the SEP, or the Protocol) is a standardized ordered list of tasks that nevertheless enables any program to develop an evaluation uniquely tailored to that program. Putting evaluation concepts into this simple set of steps requires that we present it in a linear format. In fact, an important objective for us in this work has been to instill the idea that effective modern evaluation requires evaluators to move beyond a linear mindset. Good evaluation requires feedback, and is embedded within a dynamic changing system. Although any written document is by definition linear, systems evaluation is a non-linear and iterative process. We expect that in various contexts it will be appropriate to perform steps out of the presented sequence or in tandem, as well as to revisit steps repeatedly throughout the process.
In August 2008 we received additional NSF funding (NSF Award #0814364) to take this work and disseminate it. The SEP expanded from a set ot steps to include resources and documents used for building evaluation capacity. Documents and videos from our partnership presentations were developed into tools that were built into Netway - so that Netway, in essence, offered a self-directed option for evaluation. Netway tutorials and FAQs were added, along with the cyberinfrastructure for stakeholder mapping, associating evaluation measures with outcomes, improved interfaces for working with the models and evaluation plans, and areas to track implementation and utilization of evaluations. The SEP - including the Protocol step, the supporting theories, and the ECB resources - became better articulated and more fully developed into the Facilitator's Guide to the Systems Evaluation Protocol. The SEP partnerships took on various formats and methods - in addition to training EPMS to take training back to their organizations we worked directly with the practitioners and staff, themselves, as well as launching our first "Virtual SEP" - where partners completed their evaluation planning on their own with nothing more than an orientation training from us. The SEP also expanded from simply Evaluation Planning, to include a protocol and support resources for evaluation implementation and utilization.
In July 2015 the Netway had expanded greatly beyond our original work with CCE 4-H programs - and is now used for various purposes. Although, in our mind, the resources are targetted towards supporting an evaluation system built upon a systems approach to evaluation, preliminary research into how Netway is used includes evaluation planning, program planning, team building, capacity building, project management. A restructuring of the underlying hierarchical structure of Netway will necessitate seasoned users to learn some new approaches to managing their programs in the Netway, it will simplify collaborations between organizations, facilitate management of program reporting, and finally enable self-registration of new users. The Netway is available at http://www.evaluationnetway.com.