ARCHIVED – Summative Evaluation of the Metropolis Project Phase II: Knowledge Transfer Activities and Impacts

Summary

Background

The Metropolis Project (Metropolis) was established in 1995/96 as a joint initiative of the Social Sciences and Humanities Research Council of Canada (SSHRC) and Citizenship and Immigration Canada (CIC). Its purpose was to support research and public policy development on population migration, cultural diversity and the challenges of immigrant integration in Canadian cities. Funded by SSHRC and a consortium of federal departments and agencies, the Metropolis project consists of five regional Centres of Excellence, each of which is a partnership between the three levels of government, academic researchers and community organizations. The total budget for Metropolis for the five-year period between 2002-03 and 2006-07 was $7,911,062.

The current evaluation is jointly sponsored by CIC and SSHRC. The objective of the evaluation of Metropolis is to assess the extent to which the Project has been successful in transferring research knowledge to government policy-makers. The evaluation focuses on knowledge transfer at the federal level (encompassing the participating federal departments). The evaluation relies on multiple lines of evidence: a literature review, document review, stakeholder interviews, case studies, administrative data, and a survey. The findings are organized into three sections that correspond to the essential dimensions of knowledge transfer – research processes, products and application (impact).

Excluded from the scope of the evaluation are the international Metropolis project, as it does not receive core funding from the federal partners, and the Atlantic Centre of Excellence, as it was established during the course of Phase II.

Key findings

Have the Metropolis Centres successfully integrated key policy issues identified by federal funding partners into their research plans?

Overall, stakeholders indicate that Metropolis Centres have had limited success in integrating federal partners’ policy issues into research plans. One quarter of potential user survey respondents indicated that their needs and priorities were ‘often’ or ‘always’ integrated into Centres’ research plans, 24% reported ‘sometimes’ and 7% reported ‘never’ or ‘rarely’. The remainder reported that they ‘do not know’. Focus group participants noted that the absence of feedback from research retreats and meetings made it impossible to tell whether their input was incorporated into the research conducted.

Some Metropolis stakeholders pointed out that the research conducted depended on the interests of individual researchers and on the quality of the proposals received by the Centres in response to their calls for proposals.

Although there was a collective exercise by federal funding partners to identify eleven priorities and the Centres identified processes to solicit policy needs from departments, there was little ongoing, formal soliciting of policy issues. Focus group participants noted that input into the research agenda depended upon informal conversations, or the relationships they had developed with their contact at the Secretariat or the Centres.

Have the Centres, the Secretariat and the Federal Consortium operated as effective knowledge “brokers”?

There was some success in knowledge brokering [ Note 1 ], however, there wasn’t a formal knowledge brokering function built into Phase 2 of the Project. The interviews and case studies indicated some success in knowledge brokering and the knowledge transfer process was due to informal conversations, or relationships that had been developed. Research staff was generally more positive. Engagement by federal funding departments was also a factor that varied over time.

Do federal funding partners access/use Metropolis research and do departments support accessing and using the research in policy-making?

Overall, users of the research products find them useful. However, there is limited support/time in departments for accessing and using Metropolis research products. Results of the survey of potential users of Metropolis research indicated that the majority rated all research outputs as either ‘very useful’ or ‘quite useful’. The most useful outputs were reports and publications and the most supportive user groups were federal department researchers and policy analysts.

In general, survey respondents did not spend a great deal of time reading or reviewing research materials or attending research dissemination activities. About one third of survey respondents felt that the overall amount of time they spent reading research materials or attending research dissemination events was not adequate for their job. The principal barrier is lack of time. About one quarter of survey respondents indicated that the use of research, and participation in research activities, was not a priority for their manager or department.

Is the Metropolis Project producing research products that are relevant to government policy makers?

Generally, Metropolis research products are considered to be relevant although concerns were expressed about their link to policy. Seventy percent of survey respondents reported that Metropolis research was relevant to government policy makers. All but one federal funding partner agreed that the research is relevant. However, just over 1/3 of interviewees and case study users expressed concern about the policy link of Metropolis research.

Funding partners’ and case study interviewees views were mixed when asked to compare the relevance of Metropolis products to that of research produced by other sources. Although about one third of funding partner interviewees indicated the Metropolis research is as relevant as research from other sources, another third said it was less relevant. Some federal partner interviewees (5/13) and case study users (3/8) expressed concerns about the policy link of Metropolis research.

Macro (including pan-Canadian), comparative and longitudinal studies were identified as a priority for Phase 2 of the Metropolis Project. Have they been conducted?

The Metropolis Centres conducted some macro, comparative and longitudinal studies. The majority of survey respondents indicated satisfaction with the extent to which the Centres conducted these types of studies. For potential users of Metropolis research products (particularly those in research and policy development roles and those in the NCR), these types of studies are important for their work.

Case study interviewees and focus group participants noted the lack of funding for large studies. Also noted was the one year time frame for funding, as large-scale studies typically take longer than one year. Metropolis Centres tended to prefer to fund their “own” researchers, as opposed to researchers across the country, which could be necessary for pan-Canadian studies. Finally, Metropolis Centres can only fund research for which they have received proposals, and during Phase II, there were limited incentives in place to encourage researchers to propose and undertake larger, more complex, studies.

Has Metropolis research knowledge informed or influenced the development of government policy?

There is some evidence that Metropolis research has informed government policy-making, but limited evidence that it has influenced policy-making. About 40 percent of the potential users who were surveyed indicated that Metropolis research had been used to inform policy discussions. Among policy analysts, that percentage increases to 79 percent.

Other stakeholders (federal funding partners and case study interviewees) were mixed in their views of whether the research products informed policy-making, with about half reporting that it had, and the other half reporting that it had not.

Informing policy-making is a less strict criterion for impacts than is influencing policy-making. The degree to which Metropolis research has directly influenced policy development is uncertain. Interviewees and case study participants had difficulty identifying specific policy documents that had been influenced by Metropolis research. Research was considered to be one input into the policy-making process, among a broad range of influences.

___________

1. Knowledge brokering can be defined as the building of relationships and networks between researchers and research users to facilitate the transfer and use of existing research knowledge, and to support the production of new research knowledge in support of evidence-informed decision-making.

Page details

Date modified: