This section summarises the research and evaluation methodology and how the evaluation protocol was implemented in each participating country. The methodology was influenced by the complexity of the project aspirations and by the variation in the contexts for the project in practice. In summary the research and evaluation activities of the VISKA project had a number of objectives. It was intended to support the field trials or interventions by guiding the development of an evaluative framework, through both formative measures and a summative assessment of impacts. Another objective was to extract evidence and analysis that would contribute to future policy and practice progression in validation processes.
Evaluation design and methods
In developing the methodological framework for VISKA the realities within which the project would be undertaken were influential.
- The project context in each of the project partner countries including variations in:
- Target cohort for the project intervention – influenced by the policy aim and environment
- Target cohort size
- Economic and social imperatives for the project
- Legislative and regulatory framework for VPL
- The project partners in each country had varying roles and responsibilities in relation to VPL
- Partners selected the interventions in which they participated and which were relevant for their organisations and their context
- The project resources were limited and were rightly focused on the implementation of the interventions and the identification of transferable policy implications rather than the research and evaluation.
In addition, the make-up of the project partnership and therefore the practical ability to undertake the implementation of the field trials changed during the course of the project which added to the complexity of the task.
As indicated in the proposal document it was not anticipated that an experimental or quasi-experimental evaluation could be achieved, nor indeed might it be desirable in the circumstances. It was clear at the proposal stage that the identified target cohorts for the interventions would not be a representative sample of the available population subgroup within the country but would be chosen with reference to economic imperatives, skills gaps, accessibility, willingness and coherence. Indeed, it was also clear that these imperatives might change during the course of the project.
A (modified) Realistic Evaluation approach was adopted which allowed the contexts for the interventions enacted under the project to form a real background to the interpretation of the findings and the project focused on the collection of rich information around the practice setting which contributed to the later extraction of value in the policy domain by the project partners themselves as the experts within their contexts. This interplay between the contexts and the actions came to the fore throughout the project.
In adopting the Realistic Evaluation approach, the VISKA project team considered the relevant aspects of the context such as the legislative and regulatory setting, the views of the stakeholders of the VISKA project as well as conducting a pre-trial SWOT analysis across the five interventions. The interventions had been agreed by the project team at the proposal stage with a view to the desired outcomes and the research methods included collection of both qualitative and quantitative data sets.
The project partnership adopted an open and collaborative approach which recognised the project partners as experts within their own context and practice setting.