ARCHIVED – Summative Evaluation of the Metropolis Project Phase II: Knowledge Transfer Activities and Impacts
5. Conclusions
This section of the report is organized to answer the questions that were addressed in this evaluation. The evaluation report focuses on the knowledge creation and transfer process and addresses three issues: how the Metropolis Project Centres solicit research priorities from federal funding partners; federal policy makers’ perceptions and evaluations of the research products; and the extent and kinds of uses of the research products that users and potential users report.
Key Evaluation Questions
Have the Metropolis Centres successfully integrated key policy issues identified by federal funding partners into their research plans?
Overall, stakeholders indicate that Metropolis Centres have had limited success in integrating federal partners’ policy issues into research plans. One quarter of potential user survey respondents indicated that their needs and priorities were ‘often’ or ‘always’ integrated into Centres’ research plans, while 24% indicated ‘sometimes’, 7% ‘never or rarely’, and 44% did not know. Focus group participants noted that the absence of feedback from research retreats and meetings made it impossible to tell whether their input was incorporated into the research conducted.
Some Metropolis stakeholders pointed out that the research conducted depended on the interests of individual researchers and on the quality of the proposals received by Centres in response to their calls for proposals.
Although there was a collective exercise made by funding partners to identify eleven priorities and the Centres identified processes used to solicit policy needs from departments, there was little ongoing, formal soliciting of policy issues. Focus group participants noted that input into the research agenda depended upon informal conversations, or the relationships they had developed with their contact at the Secretariat or the Centres.
Have the Centres, the Secretariat and the Federal Consortium operated as effective knowledge “brokers”?
There wasn’t a formal knowledge brokering function built into Phase 2, however, overall there was some success in knowledge brokering and the knowledge transfer process. The interviews and case studies indicated that this success depended upon informal conversations, or on the relationships that had been developed. Research staff were generally more positive. Engagement by federal funding departments was also a factor that varied over time.
Two evaluation questions focused on how policy-makers access research products: Do federal policy-makers access/use Metropolis research and do departments support accessing and using the research in policy-making?
Overall, users of the research products find them useful. There is limited support/time in departments for accessing and using Metropolis research products. Results of the survey of potential users of Metropolis research indicated that the majority rated all research outputs as either ‘very useful’ or ‘quite useful’. The most useful outputs were reports and publications and the most supportive user groups were federal department researchers and policy analysts.
In general, survey respondents did not spend a great deal of time reading or reviewing research materials or attending research dissemination activities. About one third of survey respondents felt that the overall amount of time they spent reading research materials or attending research dissemination events was not adequate for their job. The principal barrier is lack of time. About one quarter of survey respondents indicated that the use of research and participation in research activities was not a priority for their manager or department.
Is the Metropolis Project producing research products that are relevant to government policy makers?
Overall, Metropolis research products are considered to be relevant although concerns were expressed about their link to policy. Seventy percent of survey respondents reported that Metropolis research was relevant to government policy makers. All but one federal funding partner agreed that the research is relevant. However, just over 1/3 of interviewees and case study users expressed concern about the policy link of the Metropolis research.
A somewhat different view of relevance emerges when Metropolis research products are compared with research produced by other sources. Funding partners’ and case study interviewees views were mixed when asked to compare the relevance of Metropolis products to that of research produced by other sources. Although about one third of funding partner interviewees indicated that Metropolis research is as relevant as research from other sources, another third said it was less relevant. A weak link to policy was the most commonly cited reason for concerns about relevance.
Macro (including pan-Canadian), comparative and longitudinal studies were identified as a priority for Phase II of the Metropolis Project. Have they been conducted?
The Metropolis Centres conducted some macro, comparative and longitudinal studies. The majority of survey respondents indicated satisfaction with the extent to which the Centres conducted these types of studies. For potential users (particularly those in research and policy development roles and those in the NCR) of Metropolis research products, these types of studies are important for their work.
Case study interviewees and focus group participants noted the lack of funding for large studies. Also noted was the one year time frame for funding, as large-scale studies typically take longer than one year. Metropolis Centres tended to prefer to fund their “own” researchers, as opposed to researchers across the country, which could be necessary for pan-Canadian studies. Finally, during Phase II there were limited incentives in place to encourage researchers to propose and undertake larger, more complex, studies.
Has Metropolis research knowledge informed or influenced the development of government policy?
There is some evidence that Metropolis research has informed government policy-making, but limited evidence that it has influenced policy-making. About 40 percent of the potential users who were surveyed indicated that Metropolis research had been used to inform policy discussions. Among policy analysts, that percentage increases to 79 percent.
Other stakeholders (federal funding partners and case study interviewees) were less positive, with about half saying that it had informed policy-making and the other half saying that it had not.
Informing policy-making is a less strict criterion for impacts than is influencing policy-making. The degree to which Metropolis research has directly influenced policy development is uncertain. Interviewees and case study participants had difficulty in identifying specific policy documents that had been influenced by Metropolis research. Research was considered to be one input into the policy-making process among a broad range of influences.
Page details
- Date modified: