Evaluation of Military Family Support Programs and Services

January 2013

1258-195(CRS)

Reviewed by CRS in accordance with the Access to Information Act (AIA). Information UNCLASSIFIED.

Acronyms and Abbreviations

Results in Brief

This report presents the results of an evaluation of support to military families through 12 programs and services. The evaluation was undertaken between October 2011 and July 2012. Its purpose was to examine program relevance and performance for the period 2007 to 2012 and to inform future management decisions related to program/service delivery and resource allocation. The clients for this evaluation are the Deputy Minister (DM) and the Chief of Defence Staff (CDS) as well as Level 1 senior managers within the Department of National Defence (DND) and the Canadian Forces (CF).

Background

The overarching objective of programs/services within the Chief Military Personnel (CMP) Morale and Welfare line of operations, which includes MFS programs/services, is to contribute to the enhancement of morale, welfare and well-being of CF members and their families, thus contributing to the operational readiness and effectiveness of the CF. Military families, recognized as the “strength behind the uniform,” are important contributors to organizational effectiveness and operational readiness. However, the frequency of relocations and separations imposed by the military lifestyle can create disadvantages for military families as compared to non-military families. Therefore there is a need to address the disadvantages imposed upon them by operational requirements.

The core of MFS programs/services is the Military Family Services Program (MFSP), which provides military personnel and their families with a range of services to promote health and well-being, provides information, and/or gives referrals to other local services and agencies. The program serves full-time CF personnel (Regular and Reserve Forces in full-time engagement), their spouses, dependents, and parents, as well as civilian employees (and their families) deployed with the CF on missions outside Canada, and families of the fallen. In addition to MFSP, there are eleven other MFS programs/ services¹ targeted to meet specific needs for subsets of military families. Annual spending directly attributed to MFS programs/services averaged $51.2 million over the period od fiscal year (FY) 2008/09 to FY 2010/11. MFSP represented half (49.74 percent) of all direct spending. In addition to the direct spending, approximately $6 million per year, spent primarily on infrastructure and employee benefits was not directly attributed to MFS programs/services.

Methodology

The evaluation used multiple lines of evidence to ensure the reliability of reported results. The evaluation incorporated both qualitative and quantitative methods, including the following:

• document review;
• key informant interviews;
• online survey;
• site visits;
• comparative analysis;
• other consultations; and
• administrative and financial data review.

Findings – Relevance

Findings #1 to 3. Relevance

As a result of the hardships and disruption resulting from the demands of military life, continuity of and access to childcare and health care (including mental health care and services for families with special needs), spousal career support, access to education support (for children and spouses) and deployment support (during separation and reunion) are the key on-going support requirements of military families. Support to military families is aligned with the Government of Canada’s priorities and is strongly aligned with DND’s Care and Support strategic outcome. Support to military families is also indirectly aligned with DND’s Readiness and Operations objectives. The DND/CF’s role in supporting military families is appropriate because military families have unique needs associated with the military lifestyle.

Findings – Performance (Effectiveness)

Finding #4. Overall Effectiveness

Despite some success in achieving intended outcomes, MFS programs/services are not sufficiently focused on the key military family support requirements. In particular, there are unmet needs with respect to continuity of and/or access to childcare and health care, as well as spousal employment/career support. While it will be difficult for the DND/CF to make significant progress on access to health care and spousal employment support due to systemic issues,² the Department's leadership and sustained effort is required to address these systemic disadvantages for military families. Improvements to continuity of access to child care are more directly achievable by the DND/CF and options to improve continuity for military families should be identified and assessed for implementation.

Recommendations

1. The DND/CF should establish and lead a working group of provincial/territorial/ regional stakeholders to develop an action plan to address systemic disadvantages (access to health care and spousal employment) to military families. Performance measures to track progress should also be developed.

OPI: CMP

2. The DND/CF should identify and assess feasible options to improve continuity of access to childcare and implement solutions to meet military family needs.

OPI: CMP

Finding #5. Immediate Outcome – Improved Continuity of Family Lifestyle

Stakeholders³ perceived improvement in continuity of lifestyle and attributed it primarily to MFSP, Community Recreation, and Youth Model. However, inconsistencies in service provision across locations continue to pose challenges for achieving the continuity of lifestyle outcome.

Recommendations

3. Community Needs Assessments (CNA) should be completed or updated in all locations using the Directorate Military Family Services (DMFS) toolkit in order to identify service gaps and inconsistencies. The frequency of mandatory CNA updates should be determined and identified in an updated Parameters of Practice document. CNAs should include questions to help determine why some military families do not use MFS programs/services.

OPI: CMP

4. A consistent core of mandated programs/services should be accessible to all CF families at all CF locations, except where CNAs show there is no need for specific programs/services.

OPI: CMP

Finding #6. Immediate Outcome – Improved Awareness of Available Resources

Stakeholders perceived an increase in military family awareness of available resources and attributed it largely to MFSP and Community Recreation. However, greater efforts are required to raise awareness of MFS programs/services among military families. There is also a need to take greater advantage of information technology and social media tools.

Recommendation

5. To increase the profile and awareness across CF locations of available MFS resources, develop a national marketing/promotion plan. This plan should be representative of all MFS programs/services and inclusive of all military families, with particular effort to target those with a spouse but no children, single military members, and non-traditional families. The plan should include increased use of information technology and social media tools, specific actions to raise the profile of the C/MFRC; and FamilyForce.ca brands among their intended audiences, and coordination of promotional efforts across the various program/service providers.

OPI: CMP

Finding #7. Immediate Outcome – Greater Community Engagement through Volunteering

The evaluation found that there is a significant degree of volunteering but the extent to which volunteering contributes to community engagement⁴ is unclear. There was little consensus among stakeholders as to whether there has been an improvement in community engagement through volunteering in recent years. Among those who perceived an improvement, progress was most frequently attributed to MFSP, Youth Model, and Community Recreation.

Finding #8. Immediate Outcome – Better Access to Emergency Assistance

Stakeholders perceived some improvement in access to and, to a lesser degree, in quality of emergency assistance in recent years. Survey data attribute improvement to MFSP, IPSCs, MFFs, DSC (access and quality), and OSISS (quality). A potential unmet need for emergency assistance, particularly emergency childcare and higher quality emergency assistance, reinforces the need to update CNAs to ensure planned capacity is adequate. However, military families may not be sufficiently aware of programs/services available, which could explain the apparent unmet need.

Finding #9. Immediate Outcome – Improved Capacity to Manage General Health, Crises, and Stressors

Evidence is inconclusive as to whether military families’ capacity to manage their general health, crises, and stressors has improved in recent years; however, their current capacity appears to be adequate. Survey respondents who perceived an improvement in the capacity to manage general health most frequently attributed it to MFSP, OSISS, Community Recreation, IPSC, and R2MR. They most frequently attributed improvements in capacity to manage crises and stressors to MFSP, IPSC, OSISS, and R2MR.

Findings #10 to 12. Intermediate Outcomes – Better Integration into New Communities, Increased Confidence in Ability to Manage the Stresses of CF Lifestyle, and Decreased Impact of Stressors on Families

Stakeholders believe military families are integrating well into new communities, are confident in their ability to manage the stresses of CF lifestyle, and are in fact able to manage these stressors. Integration of families into new communities is not necessarily due to MFS programs/services, but rather due to the natural resilience of families.

Findings #13 to 15. Ultimate Outcomes – Increased Family Support for Members’ Careers, Greater Confidence that Family is Supported, and Enhanced Social Functioning and Emotional Well-being

Spousal support of military members’ careers appeared to be high. The most likely contributors were better societal understanding of the military lifestyle and better informed spousal expectations. There was a lack of consensus among stakeholders as to whether family support increased in the past five years. Military members are more confident that their families are supported than they were five years ago. However, support to families would be enhanced by better meeting key needs for medical care, childcare and spousal employment support. CF families seem to function well and have good emotional well-being. Many stakeholders believe social functioning and emotional well-being have improved in the past five years. However, the effects of a high operational tempo (and recent combat missions) are not yet fully understood and may have an impact on social functioning and emotional well-being in the future.

Findings #16 to 17. Operational Outcomes – Increased Willingness to Deploy and Increased Retention of Military Personnel

The majority of CF members are willing to deploy. Consistent with the literature, most key informants were of the opinion that MFS programs/services contribute to a willingness to deploy. Retention has improved in recent years; however, it is unlikely that this change is attributable to MFS programs/services. Nevertheless, family issues are important in making career decisions and it is likely that if MFS programs/services did not exist, military members would be less willing to remain in the military.

Finding #18. Unintended Outcomes

The support structure currently in place for military families may be creating dependencies on external support and feeding unsustainable expectations of some military families for support programs/services.

Findings – Performance (Efficiency and Economy)

Finding #19. Current Spending Insufficient to Address Key Gaps in Military Family Support

Current spending does not adequately address key gaps in support to military families. Addressing significant aspects of these gaps need not require significant spending increases.

Recommendation

6. Reallocate existing MFS resources to ensure funding is available to implement recommendation #1 (stakeholders working group to develop an action plan to address systemic disadvantages to military families) and recommendation #2 (identification and assessment of options to improve continuity of access to childcare for CF families).

OPI: CMP

Finding #20. Overspending on Management and Administration

The amount of funds spent by MFS programs/services on management and administration is higher than expected and may indicate inefficiencies and duplication of effort.

Recommendation

7. MFS overhead costs should be reviewed and accounting issues resolved to ensure overhead costs are minimized and the greatest amount of funding is spent on service delivery.

OPI: CMP

Finding #21. Dispersed Governance

Lack of coordination among MFS programs/service may contribute to overhead costs and reduce the level of integration of programs/services.

Recommendation

8. Assign DMFS responsibility for coordination of MFS programs/services in order to reduce overhead costs. In particular, DMFS should do the following:

• implement the DMFS system to track program/service user data (numbers of users and their demographics) in all MFS programs/services;
• implement the DMFS system to track spending relative to activity areas identified in the logic model in all MFS programs/services; and
• develop a plan to coordinate the collection of data on military families (e.g., site visits, needs assessments, town halls, surveys, and focus group research) to meet both internal and external information requirements.

OPI: CMP

Finding #22. Attention Required for Specific Programs/Services and Logic Model

The greatest spending is on MFSP, which, based on survey respondents’ perceptions, has been the most successful in making progress towards intended outcomes. Spending on DEM and MFFC is the second and third highest, respectively. Both programs address components of key support requirements for military families. Attention is required for the MFF in order to improve efficiency and return on investment.

Note: Please refer to Annex A—Management Action Plan for the management response to the Chief Review Services (CRS) recommendations.

1.0 Introduction

This report presents the results of an evaluation of support to military families through twelve programs/services under activity 4.1.1.4 —Military Family and Casualty Support of the Program Alignment Architecture (PAA). Approximately half of the total $200 million allocated to this PAA element in FY 2010/11 was estimated to be expended on MFS (i.e., roughly $100 million).⁵ The evaluation was undertaken by CRS between October 2011 and July 2012 on behalf of the CDS and the DM of DND. It was conducted in accordance with the DND/CF Evaluation Work Plan and examined program relevance and performance for the period 2007 to 2012.

The clients for this evaluation are the DM and the CDS as well as the Level 1⁶ senior managers within the Department. Recommendations resulting from the evaluation will be used to inform future management decisions related to program/service delivery and resource allocation.

The program was last evaluated in FY 1994/95. While this program does not represent a significant proportion of DND’s annual spending, it has high visibility within the CF and in communities with large CF establishments. Military families are recognized in the DND/CF as the “strength behind the uniform.” Military families are important to individual and organizational readiness to deploy as well as to military operational effectiveness. ⁷ However, the demands of military life (i.e., frequent relocations, separations particularly during high operational tempo, etc.) can disadvantage CF families in comparison to others within Canadian society, ⁸ resulting in requirements for support mitigate these disadvantages. Due to the increased operational tempo of the last 10 years, as well as significant additions to the spectrum of MFS programs and services, an evaluation of the program was considered timely.⁹

1.1 Profile of the Military Family Support Programs/Services

1.1.1 Background

Programs and services for military family support in the DND/CF included in the evaluation are primarily situated within Morale and Welfare operations under theCMP. As shown in Annex B, the following groups offer programs to support military families:

• DGPFSS
• Director – Dependent Education Management (DDEM)
• Director – Casualty Support Management (DCSM)
• Director General Health (DGHS)
• Personnel Support Program (PSP)
• DMFS

The core of MFS programs/services is the MFSP. The MFSP was launched in 1991 under the direction of the DMFS. The policy authority for the program is provided by Defence Administrative Orders and Directives (DAOD) item 5044-1¹⁰ and the Canadian Forces Family Covenant.¹¹ The Family Covenant “reflects a commitment to military families and serves as the cornerstone, or foundation, from which the Department of National Defence can continue to enhance military family services.”

The MFSP provides services to promote health and well-being in personal development and community integration, child and youth development and parenting support, family separation and reunion, and prevention, support, and intervention. It also provides information and referrals to other local services/agencies as necessary. As per the MFSP Parameters for Practice, the program serves full-time CF personnel (Regular and Reserve Forces in full-time engagement),¹² their spouses,¹³ dependents,¹⁴ and parents. Civilian employees (and their families) deployed with the CF on missions outside of Canada also have access to MFSP programming, as do families of the fallen.¹⁵

In addition to MFSP, there are targeted MFS programs/services, including the following:

• OSISS Family Peer Support Coordinators provide social support to CF members and families affected by operational stress injuries;
• HOPE volunteers/management offers peer bereavement support;
• IPSC, FLOs, provides support for ill and injured personnel;
• the MFF is used to provide short-term emergency assistance;
• the DEM is an education compensation and benefits program for dependent children;
• Community Recreation offers recreation programs to enhance morale and community welfare;
• R2MR is a training program that addresses each stage of the deployment cycle;
• Family Violence is an awareness program that promotes responsibility and prevention;
• MFFC is a training program to educate families about care and support to the member;
• DSCs provide support for families during CF members' extended absences on duty; and
• the Youth Model offers support for youth and their families.

See Annex C for a detailed description of these 12 MFS programs/services.

1.1.2 Objectives of the Program

The overarching objective of programs/services within the CMP’s Morale and Welfare line of operation, which include MFS programs/services, is to contribute to the enhancement of morale, welfare, and well-being of CF members and their families,¹⁶ thus contributing to the operational readiness and effectiveness of the CF.¹⁷ In DAOD 5044-1, the CF committed to the following:

• to provide services for the well-being of CF members and their families;
• to assist in reducing the impact created by frequent postings on the social and educational integration and stability of CF members and their families; and
• to take measures to reduce the impact of long and frequent periods of family separation.

As a group, MFS program/services are primarily involved in the areas of child and youth development and parenting support, family development and community integration, family separation and reunion support, and prevention, support, and intervention. Programs and services in these areas help military families better integrate into their communities and have the capacity to manage the stresses of a CF lifestyle. In turn, these outcomes are expected to contribute to greater family support of a military member’s career, improved CF members’ confidence that families are being supported, and enhanced well-being of military families. By extension, these outcomes are expected to contribute to CF operational objectives related to deployment and retention, as well as to the DND strategic outcomes related to providing care and support to providing the CF and to an overall contribution to Canadian society.

The specific activities, outputs, and outcomes of MFS programs/services are illustrated in the program logic model, as shown in Annex D.

1.1.3 Delivery Approach

MFS programs/services are predominantly delivered at 32 Military Family Resource Centres (MFRC) at CF locations across Canada and at 11 Canadian Military Family Resource Centres (CMFRC) outside the country where CF members and their families are posted in significant numbers (five in the United States and six overseas).

In Canada, MFRCs are third-party, incorporated, not-for-profit organizations, governed by elected volunteer boards of directors, that work in partnership with the local commanding officers under a memorandum of understanding. Outside Canada, CMFRCs are not third-party organizations but rather operate within the local CF command structure. The C/MFRCs operate under a service level agreement and are supported by appointed volunteer advisory committees.

Mandated services are offered by all C/MFRCs where demonstrated need exists and in accordance with policies published by DMFS. Site-specific services may also be offered in response to identified and documented need, if acceptable to the commanding officer and, in the case of C/MFRCs in Canada, the board of directors. Site-specific services are funded by the commanding officer or through fundraising, donations, grants, fees for services, or other revenue-generating methods available to not-for-profit organizations.

C/MFRC staff, which may include social workers, psychologists, and educators, deliver the MFSP, Youth Model, and family-related components of OSISS, IPSC and R2MR. They work with agencies and service providers in their local communities to co-ordinate services and minimize duplication. Some programs/services are delivered on base by military personnel, including Community Recreation, Family Violence, MFFC and DSC. Other programs/services such as HOPE, MFF and DEM are delivered through CF headquarters.

1.1.4 Program Resources

Annual spending directly attributed to the MFS programs/services averaged $51.8 million over the period FY 2008/09 to FY 2010/11. MFSP represented half (49.74 percent) of all direct spending and DEM represented almost one-third (32.8 percent). In addition to the direct spending, approximately $6 million per year (i.e., spent primarily on infrastructure and employee benefits) was not directly attributed to MFS programs/services.

2.0 Methodology

The evaluation scope and methodology was set out in an evaluation work plan developed during the evaluation planning phase (October 2011 to March 2012). The evaluation work plan was designed to align with the federal government’s Policy on Evaluation (April 2009). The following sections describe the evaluation objective, scope, issues and questions, data collection methods, and the limitations of the evaluation.

2.1 Evaluation Objective and Scope

The objective of the evaluation was to examine the relevance and performance of DND/CF MFS programs/services. The evaluation focused on programs and services that use public funds to provide direct support to military families and for which military families are the primary intended recipients of delivery. Standard employee assistance program-type benefits provided to families of other public servants (e.g., health care benefits) were excluded from the evaluation, as were DND/CF programs that do not directly benefit military families. Site-specific programs/services delivered by C/MFRCs that are funded by donations, grants, fees for services, or other revenue-generating methods were also excluded.

2.2 Evaluation Issues and Questions

The evaluation of MFS programs/services examined issues related to relevance and performance. The issues and questions addressed in the evaluation are as follows (see Annex E for the complete evaluation matrix, which also includes specific indicators and methodologies for each evaluation question).

2.3 Data Collection Methods

The evaluation of MFS programs/services included the use of multiple lines of evidence and complementary research methods as a means to help ensure the reliability of information and data collected. The following data collection methods were used to gather qualitative and quantitative data for the evaluation:

• document review and comparative analysis;
• key informant interviews and site visits;
• administrative and financial data review;
• questionnaire (online survey); and
• advisory group and stakeholder consultations.

Where appropriate, the data obtained from the lines of evidence were analyzed using a customized template organized according to evaluation questions and corresponding indicators. Each of these methods is described in more detail in the following sections.

2.3.1 Document Review and Comparative Analysis

A comprehensive document review was conducted to assess the relevance of the MFS programs/services. Documents reviewed included foundational program documents and federal/departmental accountability documents. In addition, deployment and retention data obtained from the Military Administrative Support System, CF exit surveys, and CF retention surveys were examined to assess continued need for the program.

Program documentation such as memoranda of agreement, service level agreements (i.e., for MFSP), CNAs, and annual reports was reviewed¹⁸ to gain a general understanding of MFS programs/services and to assess performance. Information on the performance of MFS programs/services (i.e., outcomes) was also drawn from surveys and previous studies of MFS programs/services conducted over the years by the DND/CF including a DMFS/MFSP participant survey and previous CRS evaluations.¹⁹ The previous surveys and studies, as well as the CNAs, include direct input from military families.

Military family support programs/services provided by allied nations were compared for relevance to the DND/CF MFS programs/services. Promising/best practices and lessons learned pertaining to military family support programs/services, policies, and/or initiatives were informed by a review of initiatives undertaken by the Garrison Community Council (GCC) and the Royal Canadian Air Force’s Helping Professionals Committees and Family Support initiative.

2.3.2 Key Informant Interviews and Site Visits

Key information interviews served as an important source of information by providing qualitative input from those directly or indirectly involved in MFS programs/services, including program/service staff, C/MFRC board of directors members (which include military family members),²⁰ base/wing commanders and the Directors General Personnel/Director Land Personnel Management (DG Pers/G1)²¹ from the three environments. A sampling strategy was used to select bases for interviews based on their characteristics. These included environment, number of families, location, size, isolation, and personnel tempo. From stakeholder indications, military families were extensively consulted and a broad set of direct data already existed from previous studies/surveys/consultations. Therefore, military families were not expressly interviewed for the purpose of this evaluation (see 2.4 Limitations for further details). At selected bases, interviews were sought with C/MFRC board members (some of which are military family members), as well as both the top of the chain of command (i.e., the base commander or administrative officer) and the most senior resource manager (i.e., the base commander or unit commanding officer).

A total of 49 interviews were conducted with 63 interviewees. The distribution of interviews is shown in Table 1.

Table 1. Distribution of Key Informant Interviews by Interview Group. This table displays the number of interviews and interviewees undertaken for this evaluation, categorized by interview group.

The interviews were conducted in person and/or via telephone. Interview guides were developed for each of the interview groups and the interview questions were designed to align with the evaluation questions identified in the evaluation matrix. The interviews lasted approximately one hour and were conducted in the preferred official language of the interviewee. Two site visits were undertaken: one at the National Capital Region (Uplands) MFRC and the other at the Petawawa MFRC. The purpose of the site visits was to gather more in-depth information on particular issues faced by military families. The National Capital Region (Uplands) site was selected to represent a C/MFRC in an urban environment (Ottawa), with military families disbursed geographically across the region (few live in private married quarters), and with a low personnel tempo. The Petawawa site was selected to represent a main operational base that is quite isolated, with a high operational tempo and where there are known family support issues, such as a lack of support programs for mental health issues.²²

During each site visit, CRS evaluators conducted on-site interviews with C/MFRC executive directors and base/wing commanders and gathered relevant data and documentation, such as annual reports and community needs assessments.

2.3.3 Administrative and Financial Data Review

An analysis was conducted to assess resource utilization (per output and per outcome) of MFS programs/services that appear under PAA element 4.1.1.4 along with Casualty Support. In order to separate the costs of MFS programs/services from the costs of Casualty Support, consultations were held with financial comptrollers.

In addition to the costs captured under PAA element 4.1.1.4, other public funds and non–public funds were expended on MFS programs/services. Consultations were held with MFS program/service managers to obtain information on the following:

• public funds expended on MFS programs/services not included in PAA element4.1.1.4;
• non-public funds expended on MFS programs/services;²³
• distribution of funding by MFS program/service area;
• number of families served; and
• spending relative to budget (MFSP only).

The data obtained were compiled in an excel spreadsheet and analyzed by year (over three or five years, as applicable).

2.3.4 Questionnaire

An online survey questionnaire was administered to three groups: 1) C/MFRC executive directors/Canadian Community Program Coordinators (CCPC); 2) Community Recreation Directors/Fitness Sports and Recreation Directors; and 3) PSP managers.

The objective of the survey was to gather information that could not be obtained in reports produced by C/MFRCs, such as feedback on efficiency in achieving outcomes (i.e.,alternative delivery options that are more cost effective), and to confirm findings obtained from other lines of evidence (i.e., key informant interviews). The survey was designed to take an average of 45 minutes to complete and was pre-tested with a random sample (n=4) of survey respondents. Some modifications were made to the survey following the pre-test.

A list of potential survey respondents representing 39 bases²⁴ was provided by MFSP. A list of potential respondents representing outside Canada remote (satellite) centres outside Canada was provided by the MFSP executive directors/CCPCs. The survey was administered electronically by sending email invitations with a link to the online survey (in both French and English) to 112²⁵ potential respondents. The survey was posted for a total of five weeks – from 14 May, 2012 to 18 June, 2012, and two reminder emails were sent during that period. A total of 54 surveys were completed, which represents an overall response rate of 48 percent.

2.3.5 Advisory Group Consultations

Consultations were held with representatives of other organizations in order to obtain more detailed information on particular issues. These consultations included the following:

• a group consultation with the National Military Family Council (NMFC);
• four separate telephone consultations with NMFC representatives;
• a consultation with a non-member who is extensively involved in the NMFC and other family engagements;
• a consultation with a representative of the DND/CF Ombudsman’s office to clarify information obtained from their website pertaining to complaints filed by military families; and
• a consultation with a representative of the GCC (a local, grass-roots, citizen-based, volunteer organization) to obtain information on the Council’s family outreach activities.

2.4 Limitations

The evaluation methodology was designed to provide multiple lines of evidence in support of evaluation findings. The data and information were collected to respond to the evaluation questions and issues. As in all evaluations, there are limitations and considerations that should be noted.

MFS programs/services are not delivered by a single service provider but rather by numerous service delivery partners. Documentation describing the alignment and contributions of service delivery partners is limited and key performance data, such as numbers of program/service users and their characteristics, are not collected on a systematic basis.

Feedback is frequently and extensively sought from CF members and their families.²⁶ Direct input from military families was available through an array of data from previous surveys of military families. However, response rates for many of these surveys were low (e.g., approximately 14 percent for one MFSP survey). In a consultation with key stakeholders for the evaluation, the majority agreed that CF families are “research-fatigued” and it was therefore decided not to survey or interview military families directly for the express purpose of this evaluation.

Although the evaluation attempted to collect information on all sources of funding for MFS programs/services, information was not available on contributions from all sources including social workers, padres, etc. Significant Regular Force pay costs were allocated to PAA element 4.1.1.4 ($8 million in FY 2008/09, $61 million in FY 2009/10, and $49 million in FY 2010/11). However, it was not possible to identify the portion of these costs related to MFS programs/services versus the Casualty Support program. Only Regular Force pay costs explicitly identified by program staff or through interviews were included in the analysis. Due to attribution rules for the PAA, only three years worth of financial information was available for MFS programs and services, thus limiting trend analysis.

3.0 Evaluation Findings

3.1 Relevance

The following section examines the extent to which MFS programs/services address a demonstrable need, are aligned with Government of Canada priorities and DND/CF strategic outcomes, and are appropriate to the federal government’s role.

3.1.1 Continued Need for MFS Programs/Services

Finding #1. As a result of the hardships and disruption resulting from the demands of military life, continuity of and access to childcare and health care (including mental health care and services for families with special needs), spousal career support, access to education support (for children and spouses), and deployment support (during separation and reunion) are the key on-going support requirements of military families.

The findings in this section are based on documents reviewed for the evaluation, key informant interviews, site visits, and other consultations.

In a survey of CF spouses carried out in FY 2008/09, respondents identified frequent relocation, temporary housing, spousal unemployment/underemployment, separation, and deployment of military members to potentially dangerous situations as key challenges. They also felt that isolation or semi-isolation compounded these challenges. The frequent transitions required of military families (i.e., relocating three times more frequently than civilian Canadian families)²⁷ have a direct impact on continuity of health care, child care, spousal employment, education, and general family stability.²⁸

The CF Ombudsman²⁹ and key informants interviewed or consulted for the evaluation identified access to childcare, access to health care (including mental health care and services for families with special needs), spousal career support, access to education (for children and spouses), and deployment support (during separation and reunion) as being the key support requirements of military families. These requirements were identified as early as 1998³⁰ and have been reiterated consistently.³¹

Key informants interviewed for the evaluation estimated overall that at least 50 percent of military families currently use MFS program/services. Similarly, half of respondents to the evaluation’s online survey estimated overall that 30 to70 percent of military families residing at their centre use MFS programs/services.³² Usage at isolated or semi-isolated locations is likely even higher. It is estimated that more than 90 percent of military families at Petawawa use MFS programs/services.³³

The need for support to military families is also demonstrated by those who make donations to MFS programs/service and volunteer their time. For example, the MFF has received $5,099,000 in donations since the fund was established in 2007 and C/MFRCs raised $1,014,274 in FY 2010/11 through their fundraising efforts. HOPE has trained 42 volunteers who spend 10 to 20 hours a month supporting peers and all Community Recreation departments make extensive use of volunteers.

While there is an ongoing need for support to military families, the level of required support will vary with changes in operational tempo and the nature of the required support will change as military families change. The requirement for services that respond to the diversity of military families (i.e., Reserve Forces families, single-parent members, dual-service families, single members, etc.) has gained prominence.³⁴ However, a lack of program/service user data prevents further inquiry in this area.

3.1.2 Alignment with Government Priorities and the DND Strategic Outcomes

Finding #2. Support to military families is aligned with Government of Canada priorities and is strongly aligned with DND’s Care and Support strategic outcome. Support to military families is also indirectly aligned with DND’s Readiness and Operations objectives.

The findings in this section are based on documents reviewed for the evaluation.

Support for families was mentioned in the 2010 Throne Speech and was identified as a strategic area in the 2011 Government of Canada Budget. MFS programs/services contribute to this Government of Canada priority by providing support to military families.

Although a distinction needs to be made between support to military families and casualty support, “caring for military members and their families” is identified as a “core responsibility” in DND’s 2012 Report on Plans and Priorities.³⁵ The Department also placed emphasis on enhanced support to the ill and the injured and to the families of CF members in its 2010 Report on Plans and Priorities.³⁶ A Departmental Directives document states that “people, both civilian and military, are our most important asset and taking care of them and their families remains a top Defence priority.”³⁷

MFS programs/services are directly aligned with the strategic outcome (4.0) — Care and Support in the departmental PAA and are indirectly aligned with two operational objectives: Readiness³⁸ and Operations.³⁹ As explained in the 2012 Report on Plans and Priorities, “Personnel Readiness and Deployability is enhanced through comprehensive support to military personnel and their families.”⁴⁰ Furthermore, departmental studies draw theoretical conclusions on links between spousal support/relationship satisfaction and organizational outcomes, including intention to leave the CF, organizational commitment, morale, and mission readiness.⁴¹

3.1.3 Alignment with Federal Roles and Responsibilities

Finding #3. The DND/CF’s role in supporting military families is appropriate because military families have unique needs associated with the military lifestyle.

The findings in this section are based on documents reviewed for the evaluation, key informant interviews, the online survey, other consultations, and the comparative review.

Some MFS programs/services are mandated by legislation. For example, support for spouses, common-law partners, and dependants is included in the New Veteran’s Charter (Canadian Forces Members and Veterans Re-establishment and Compensation Act) and education programs at DND Dependant Schools Overseas are authorized under the National Defence Act, Section 12(1). Other programs/services support the intentions of the CF Family Covenant.⁴² Many of Canada’s allies, such as Australia, New Zealand, the United Kingdom, and the United States also recognize that they have a responsibility to support military families and have responsibility statements similar to the CF Family Covenant. Most provide support to military families in the critical areas of schooling, child care, spousal employment support, healthcare, family separation support, and housing.⁴³

Most⁴⁴ survey respondents and key informants interviewed for the evaluation felt that the role of the DND/CF in supporting military families is appropriate because of the conditions of service and the unique needs of military families. Less than half of survey respondents and key informants interviewed believed there was duplication with programs and services available in the community. The remainder believed that MFS programs/services complement, rather than duplicate programs and services available in the community and thet help to improve consistency across the country in light of provincial/territorial disparities.

Some⁴⁵ key informants stressed the importance of support to families in meeting operational objectives, (i.e., military members are more willing to deploy and more willing to remain in the military if their families are supported).

3.2 Performance (Effectiveness)

The following section provides the findings with respect to whether MFS programs/services have achieved their intended results over the course of the evaluation period. The section is separated into discussions of findings with respect to the achievement of immediate outcomes, intermediate outcomes, ultimate outcomes, and operational objectives.

In the immediate and intermediate outcomes section, attribution of improvements in achievement of outcomes is linked to specific MFS programs/services based on survey data. Only those survey respondents who indicated they were somewhat familiar with each of the 12 MFS programs/services were included in the attribution analysis. Overall, participants were familiar with all but two of the 12 programs/services. Less than 50 percent of participants reported familiarity with HOPE and MFFC.

3.2.1 Overall Finding Effectiveness

Finding #4. Despite some success in achieving intended outcomes, MFS programs/services are not sufficiently focused on the key military family support requirements. In particular, there are unmet needs with respect to continuity of and/or access to childcare and health care, as well as spousal employment/career support. While it will be difficult for the DND/CF to make significant progress on access to health care and spousal employment support due to systemic issues,⁴⁶ DND leadership and sustained effort are required to address these systemic disadvantages for military families. Improvements to continuity of access to child care are more directly achievable by the DND/CF and options to improve continuity for military families should be identified and assessed for implementation.

MFS programs/services have produced improvements towards the intended outcomes identified in the logic model. However, despite this apparent success, MFS programs/ services are not sufficiently focused on key military family support requirements. As identified in the Continued Needs section of this reports (section 2.1.1), the primary ongoing needs of military families include continuity and/or access to childcare and health care (including mental health care and services for families with special needs), spousal career support, access to education support (for children and spouses), and deployment support (during separation and reunion). The programs and services offered by the DND/CF do not explicitly address each of these high-need areas and/or do not address them to a depth that will effect a significant reduction in support gaps in these key areas. Instead, many of the programs/services address issues peripheral to the key family support requirements.

Systemic issues affect access to health care and transferability of professional certifications and designations related to spousal career support. Resolution of these issues is not within the control of the DND/CF. Nevertheless, the DND/CF may be able to exert influence through inter-governmental discussion (i.e., advocacy) and collaboration (federal, provincial/territorial, regional). The Director Quality of Life (DQOL) has recently re-established Quality of Life initiatives that include awareness-raising and engagement with provincial/territorial/regional policy makers and service providers in these domains.⁴⁷ DND leadership and sustained effort will be required to make progress in addressing these systemic disadvantages to military families.

As childcare is largely provided in the private sector and funded privately (e.g., user-pay), it is less affected by systemic issues. With current shortages in childcare across Canada, military families are not unlike many other Canadian families who face challenges in accessing childcare. However, frequent relocations throughout military life make accessing childcare more challenging for military families as they move from place to place. Therefore, improved support for continuity in access to child care for military families needs to be addressed.

3.2.2 Immediate Outcome – Improved Continuity of Family Lifestyle

Finding #5. Stakeholders perceived improvement in continuity of lifestyle and attributed it primarily to MFSP, Community Recreation, and Youth Model. However, inconsistencies in service provision across locations continue to pose challenges for achieving the continuity of lifestyle outcome.

The findings in this section are based on key informant interviews, advisory group/stakeholder consultations, the online survey questionnaire, document review, and site visits.

Based on stakeholder perception, progress has been made in continuity of family lifestyle. However, a few key informants among the program staff (27 percent; 4 of 15) felt unsure of progress made in this area. According to the logic model, the MFS programs/services that are expected to contribute to this outcome through the provision of programs/services are as follows:

• MFSP
• OSISS (Family Peer Support Coordinators)
• HOPE (volunteers/management)
• DEM
• Community Recreation
• Youth Model

Of the six programs/services expected to contribute to this outcome, MFSP, Community Recreation, and Youth Model were mentioned most frequently by survey respondents who perceived improvements and were very or somewhat familiar with the programs. OSISS, DEM, and HOPE were not mentioned as frequently as expected, despite survey respondents’ reported familiarity with these programs (i.e., 70 percent, 61 percent, and 50 percent were familiar, respectively).

In general, inconsistency of service delivery (availability) and accessibility (geography, navigation/bureaucracy, language) across C/MFRCs were cited by stakeholders as a continued challenge to achieving the continuity of lifestyle outcome. As per the DMFS Parameters for Practice (2004), C/MFRCs offer services in four mandated service categories using public funds “where demonstrated need exists.” Given that CNAs have not been undertaken in all locations and that in some locations they have not been recently updated, community needs may not have been adequately documented to trigger program/service offerings at C/MFRCs. A lack of up-to-date CNAs likely contributed to inconsistency of service delivery across locations. Evaluators noted that DMFS had recently developed a tool kit and guidelines for C/MRFCs to use in conducting CNAs.⁴⁸

3.2.3 Immediate Outcome – Improved Awareness of Available Resources

Finding #6. Stakeholders perceived an increase in military family awareness of available resources and attributed it largely to MFSP and Community Recreation. Greater efforts are required to raise awareness of MFS programs/services among military families, with consideration for the diversity of military families, and to take greater advantage of information technology and social media tools.

The findings in this section are based on key informant interviews, advisory group/stakeholder consultations, online survey questionnaire, analysis of program data, document review, and site visits.

Previous studies⁴⁹ indicated high levels of awareness of available resources among military families. However, CNAs reviewed for the evaluation revealed that many families were of the opinion that services provided by the local C/MFRC are poorly advertised in the military community. Therefore, lack of awareness on the part of the families was one of the main reasons identified across multiple lines of evidence⁵⁰ as to why they do not use MFS programs/services. Both interviewees and information gathered from other consultations indicated that continued challenges to awareness of available programs/services stem from more military families living off-base, making it more difficult to engage them, and the changing demographics of CF families (i.e., military members who are single, members with a spouse but no children, non-traditional families), who sometimes have the impression that MFS programs/services would not be of interest to them. Key informant interviewees also noted that, at least in some locations, web sites and social media are not used effectively in communicating.

Interviewees and survey respondents perceived that military families’ awareness of available resources has improved in the past five years. From the program logic model, the following MFS programs/services are expected to contribute to awareness of available resources:

• MFSP
• OSISS
• HOPE
• Community Recreation

Survey respondents, who perceived an increase in awareness of resources and were very or somewhat familiar with the programs, mentioned MFSP, IPSC, and Community Recreation most frequently as contributors to improved awareness. HOPE was mentioned less frequently than expected, as well as less frequently than many of the other MFS programs/services, despite respondents’ moderate program familiarity (50 percent). In contrast, awareness of IPSC was attributed to improvements in this outcome the second most frequently and awareness of DSCs the fourth most frequently, despite both programs not being expected to contribute directly to this outcome according to the logic model.

3.2.4 Immediate Outcome – Greater Community Engagement through Volunteering

Finding #7. The evaluation found that there is currently a significant degree of volunteering, but the extent to which volunteering contributes to community engagement is unclear. There was little consensus among stakeholders as to whether there has been an improvement in community engagement through volunteering in recent years. Among those who perceived an improvement, progress was most frequently attributed to MFSP, Youth Model, and Community Recreation.

The findings in this section are based on key informant interviews, the online survey questionnaire, analysis of program data, and site visits.

Almost all respondents surveyed (96 percent, 52 of 54) agreed with the theory that volunteer involvement helps with community engagement. Previous surveys also indicated a significant degree of family involvement in local communities, including through volunteering.⁵¹ Program data reviewed indicated that MFS programs/services are providing volunteer opportunities⁵² and many MFS programs have registered/active volunteers. For example, HOPE has trained 42 volunteers, each of whom spend about 10 to 20 hours a month supporting peers. In addition, all Community Recreation departments are led by a team of volunteers (in accordance with policy outlined in Canadian Forces Administrative Order 50-2) and use volunteers in all recreation clubs (there are several at each base/wing/unit).

Despite reported levels of volunteering, there was little consensus on whether there has been an improvement in community engagement through volunteering in the past five years. Survey respondents’ perception was split: 44 percent (24 of 54) perceived improvement, while 35 percent (19 of 54) perceived no improvement. Most C/MFRC board members (82 percent, 9 of 11) perceived that progress had been made. Other key informant interviewees were not asked.

From the logic model, the following programs are expected to contribute to community engagement through volunteering:

• MFSP
• OSISS
• HOPE
• Community Recreation

MFSP, Community Recreation, and HOPE were mentioned by at least 39 percent of survey respondents who perceived improvements and were very or somewhat aware of the programs. Youth Model was mentioned with the second highest frequency, even though it is not one of the programs expected to contribute to this outcome. Similarly, DSCs were not expected to contribute to this outcome but it was among the top five and ranked higher than OSISS.

3.2.5 Immediate Outcome – Better Access to Emergency Assistance

Finding #8. Stakeholders agreed that families are currently able to access emergency assistance programs/services and perceived some improvement in access to and, to a lesser degree, in quality of emergency assistance in recent years. Survey data attribute improvement to MFSP, IPSC, MFF, and DSCs (access and quality) and OSISS (quality). A potential unmet need for emergency assistance, particularly emergency childcare and higher quality emergency assistance, reinforces the need to update CNAs to ensure planned capacity is adequate. However, military families may not be sufficiently aware of programs/services available, which could explain the apparent unmet need.

The findings in this section are based on key informant interviews, advisory group/ stakeholder consultations, the online survey, analysis of program data, document review, and site visits.

The online survey conducted for this evaluation indicated a perception that families have some success in accessing emergency assistance of all kinds and from all sources (in the community and through MFS programs/services). Three-quarters of survey respondents (74 percent, 40 of 54) agreed that families are currently able to access emergency assistance programs/services. However, only 61 percent (33 of 54) rated the overall quality as “very good” or “good.”

Available user data on emergency assistance (both level of use and demographics of users) was sparse. A selection of MFSP Service Delivery Agreements reviewed indicated that C/MFRCs were anticipating less than three percent of military families to access emergency assistance in FY 2012/13. This rate seems low given the apparent need for emergency assistance, particularly emergency childcare.

The potential unmet need for emergency assistance, particularly emergency childcare, and higher quality services, reinforces the need to update CNAs to ensure planned capacity is adequate. As discussed in section 2.2.1.2, military families may not be sufficiently aware of the breadth of programs/services available and therefore may not use available services. As recommended in section 2.2.1.1, in updating CNAs, C/MFRCs should undertake research to determine why some military families do not use MFS programs/services.

Most interviewees, predominantly represented by senior managers or administrators, perceived some improvement to both access and quality (to a lesser extent) of emergency assistance. However, fewer survey respondents, who may be more directly involved with service delivery, perceived improvement (access: 63 percent, 34 of 54; quality: 50 percent, 27 of 54). A few stakeholders from both lines of evidence reported being unsure whether progress was made in this area in the past five years.

All but two (DEM and Youth Model) of the 12 MFS programs/service are expected to contribute to this outcome. MFSP, IPSC, MFF, and DSCs were mentioned as contributing to improved access by at least 50 percent of respondents who perceived improvements and were very or somewhat familiar with the programs. These four programs were also perceived to have contributed to improved quality by over 54 percent of those respondents. Community Recreation was mentioned the least frequently for both access and quality (program familiarity: 90 percent). Respondents also suggested that the Community Gateway website and an organizational focus on mental health may have contributed to improvements in this outcome.

3.2.6 Immediate Outcome – Improved Capacity to Manage General Health, Crises, and Stressors

Finding #9. Respondents indicated that military families’ capacity to manage their general health, crises, and stressors is currently adequate. However, evidence is inconclusive as to whether improvements have been achieved in recent years. Survey respondents who perceived an improvement in capacity to manage general health most frequently attributed it to MFSP, OSISS, Community Recreation, IPSC, and R2MR. They most frequently attributed improvements in capacity to manage crises and stressors to MFSP, IPSC, OSISS, and R2MR.

The findings in this section are based on key informant interviews, consultations with other stakeholders, the online survey, analysis of program data, document review, and site visits.

Almost all (89 percent, 48 of 54) survey respondents indicated that military families are currently able to manage their general health, crises, and stressors. This aligns with results of previous MFS surveys in which approximately three-quarters of respondent families consistently reported feeling capable of managing health and stressors (i.e., CF-related separation and reunion) with little change over time.

As discussed in Section 2.1.1 and consistent with previous studies,⁵³ those consulted for the evaluation suggested that challenges faced by military families are similar to those faced by many Canadian families. These are access to medical care (especially specialized care), childcare, and inter-provincial/territorial recognition of professional credentials (i.e., career continuity). However, these issues are more pronounced for military families due to CF conditions of service. Particularly, they experience more frequent relocations than other Canadian families.

There were conflicting perceptions of improvements to military families’ capacity to manage their general health, crises, and stressors. Over two-thirds of base/wing commanders (68 percent, 13 of 19) interviewed perceived some progress but only half of C/MFRC board members (50 percent, 5 of 10) perceived progress. Close to one-third (28 percent, 8 of 29) of both groups perceived that no progress had been made in the past five years. Only half (50 percent, 27 of 54) of survey respondents perceived improvements in capacity to manage crises and stressors and 63 percent (34 of 54) perceived improvements in capacity to manage general health. At least one-quarter of survey respondents indicated that they did not know or did not provide a response.

According to the program logic model, the following MFS programs/services are expected to contribute to improving the capacity of military families to manage general health, crises, and stressors:

• MFSP
• OSISS
• HOPE
• IPSC
• MFF
• Community Recreation
• R2MR
• Family Violence
• MFFC

Online survey respondents, who perceived improvements and were very or somewhat aware of the programs, mentioned that MFSP, OSISS, Community Recreation, IPSC, and R2MR most frequently related to improved capacity to manage general health. They mentioned that MFSP, IPSC, OSISS, and R2MR most frequently related to improved capacity to manage crises and stressors.

DSCs, which were not expected to contribute to this outcome, were mentioned more frequently than a number of the programs/services that were expected to contribute. Since DSCs provide support to families and CF personnel in specific circumstances (i.e., during extended absences), it is not surprising that survey respondents attributed improvements in this outcome area to this program.

Attribution to Youth Model was unexpected, but may indicate perceived support offered by this program to the general health of youth. Of note, respondents also attributed improvements in capacity to manage general health to Health Promotion programs.

3.2.7 Intermediate Outcome – Better Integration into New Communities

Finding #10. Stakeholders believed military families are integrating well into new communities, which may not necessarily be due to MFS programs/services, but rather to their natural resilience. Many stakeholders perceived an improvement in integration in the past five years.

The findings in this section are based on key informant interviews, advisory group/ stakeholder consultations, the online survey questionnaire, and document review.

It appears that military families are generally integrating well into their new communities upon relocation. Previous studies reported that military families generally felt welcomed by their new communities⁵⁴ and almost all (98 percent; 53 of 54) online survey respondents felt that families are (currently) very or somewhat successful with respect to integrating into new communities.

However, successful integration is not necessarily attributable to MFS programs/services, however. Close to half (43 percent, 23 of 54) of those surveyed felt that military families would be very or somewhat successful in integrating into new communities in the absence of MFS programs/services. Some key informant interviewees expressed the same view, reinforcing the notion of the inherent resilience of military families.

Improvements in integration of families into new communities in the past five years were perceived by many survey respondents (65 percent, 35 of 54) and many key informant interviewees (67 percent, 18 of 27). The same groups attributed these improvements to some extent to MFS programs/services, but also to local community resources and the Community Gateway website.

3.2.8 Intermediate Outcome – Increased Confidence in Ability to Manage the Stresses of CF Lifestyle

Finding #11. Military families are confident in their ability to manage the stresses of a CF lifestyle. Stakeholders perceived improvement in the achievement of this outcome over the past five years.

The findings in this section are based on key informant interviews, advisory group/stakeholder consultations, the online survey questionnaire, and document review.

The document review conducted for the evaluation indicated military families may be better able to manage the stresses of a CF lifestyle as they are more resourceful than in the past, but this is highly dependent on the type of base/wing/unit (i.e., environment and location), the type or source of stress and the general level of family resilience. An MFRC evaluation (2008) showed that 73 percent of C/MFRC participants felt their family was able to effectively manage deployment-related separation and reunion, though responses differed by base type (i.e., Navy, Army, Air Force). Another study⁵⁵ showed personnel tempo (i.e., deployment frequency, length of separation and garrison workload) is a key factor affecting the ability to handle these stresses. Lack of downtime was suggested as a primary cause of mental health problems.

Online survey respondents indicated that military families are confident in their ability to manage the stresses of a CF lifestyle.⁵⁶ Both survey respondents and interviewees perceived that improvements in this outcome have been made in the past five years.⁵⁷ Improvements were perceived more frequently by C/MFRC board member interviewees and survey respondents (78 percent or 7 of 9 and 65 percent or 35 of 54, respectively) than by base/wing commanders (55 percent or 6 of 11). Improvements were attributed to some extent to MFS programs/services and to health promotion programs via PSPs. Interviewees linked their perception of increased resources available to support families with improvements in this outcome.

Among interviewees who did not perceive an improvement in military families’ confidence to manage the stresses of a CF lifestyle over the past five years, an overreliance on external support to the detriment of a military family’s own resilience was suggested as a possible factor. Others suggested that older recruits, of which there is a growing number, and their families who are new to the CF lifestyle may not have the experience or confidence to manage the stresses associated with this new lifestyle.

3.2.9 Intermediate Outcome – Decreased Impact of Stressors on Families

Finding #12. Military families are currently able to manage family stressors. Stakeholders believe progress has been made in decreasing the impact of these stressors.

The findings in this section are based on key informant interviews, the online survey questionnaire, and document review.

The documents reviewed for the evaluation suggest that social support networks and social work support may be important factors in families’ successful adaptation to stressors.⁵⁸ All interviewees agreed that MFS has helped decrease the impact of stressors on families by reducing isolation, bringing people together, providing resources, and providing assurance that services are in place if needed. Most survey respondents (87 percent, 47 of 54) indicated that they perceived that military families are very or somewhat able to manage the impact of family stressors. Almost all interviewees who provided a response (94 percent or 16 of 17) and 65 percent of survey respondents (35 of 54) perceived that there have been improvements in this outcome in the past five years, which they attributed to MFS programs/services as well as health promotion via PSPs and family support in local communities. Interviewees also suggested that having more programs and resources in place than in the past and more support from the chain of command have contributed to the improvements. Survey respondents indicated that further improvements could be gained from increased awareness of available resources and greater information sharing between stakeholders.

3.2.10 Ultimate Outcome – Increased Family Support for Members’ Careers

Finding #13. Spousal support of military members’ careers appeared to be high. The most likely contributors were better societal understanding of the military lifestyle and better informed spousal expectations. There was a lack of consensus among stakeholders as to whether family support increased in the last five years.

The findings in this section are based on document review, key informant interviews, consultations with other stakeholders, the online survey, and site visits.

Results of the 2009 Perstempo survey indicated that spousal support for CF careers was high.⁵⁹ Previous studies⁶⁰ have suggested that improved societal understanding and more informed spousal expectations of the military life may contribute to greater support of military members’ careers. Among those key informants who agreed there has been more spousal support for members’ careers, some mentioned these factors as contributors. Of note, positive associations were found in previous studies⁶¹ between the support provided by spouses and organizational commitment/retention, as well as the psychological well-being of both members and spouses.

Stakeholder perceptions are varied as to whether there have been improvements in spousal support of members’ careers in the past five years. C/MFRC board members interviewed and survey respondents perceived improvements in this respect to a much greater extent than either base/wing commanders or program staff.⁶²

Among those who perceived improvements (72 percent, 39 of 54), most survey respondents (97 percent, 38 of 39) and 69 percent (9 of 13) of key informants gave credit to MFS programs/services.

3.2.11 Ultimate Outcome – Greater Confidence that Family is Supported

Finding #14. Military members have greater confidence in the level of support available to their families. However, support to families would be enhanced by better meeting key needs for medical care, childcare and spousal employment support.

The findings in this section are based on key informant interviews, advisory group/stakeholder consultations, the online survey questionnaire, document review, and site visits.

Evidence found in both the online survey and key informant interviews supports the conclusion that military members are more confident⁶³ that their family is supported. Although a few interviewees were unsure (program staff interviewees particularly⁶⁴), those that ventured an opinion (48 percent, 12 of 25) based this on both formal and informal feedback obtained from military members themselves.

Among survey respondents who perceived improvements (89 percent, 48 of 54), 94 percent (45 of 48) gave credit to MFS programs/services. In addition, 87 percent (14 of 16) of key informants credited MFS programs/services with an improvement in the achievement of this outcome.

Previous studies⁶⁵ reported that military members’ pronouncements on whether their family members received needed services were also generally positive, with the exception of specific areas, including the following:

• availability and quality of medical services;
• access to childcare (particularly emergency respite childcare, emergency childcare arrangements, and affordable casual childcare services); and
• employment transition support (civilian family members).⁶⁶

3.2.12 Ultimate Outcome – Enhanced Social Functioning and Emotional Well-being

Finding #15. Overall, CF families seem to function well and have good emotional well-being. Many stakeholders believe social functioning and emotional well-being have improved in the past five years. However, the effects of a high operational tempo (and recent combat missions) are not yet fully understood and may have an impact on social functioning and emotional well-being in the future.

The findings in this section are based on document review, key informant interviews, consultations with other stakeholders, the online survey, and site visits.

Previous studies show that military families have a good level of well-being, and that this has been the case for many years. Many C/MFRC board members and base/wing commanders interviewed for the evaluation think that social functioning and emotional well-being have been enhanced in the past five years (73 percent, 8 of 11 and 68 percent, 13 of 19, respectively), as do 76 percent (41 of 54) of survey respondents.

However, many stakeholders remained uncertain of progress made in this area.⁶⁷ In fact, evidence from other consultations suggested that the high operational tempo experienced by military members (and their families) in recent years may be a significant and confounding factor. These same stakeholders indicated that the effects of recent combat missions (Bosnia and more recently Afghanistan) have yet to be well understood and will likely have a sustained impact on the social functioning and emotional well-being of military families.

Among the survey respondents who perceived improvement (76 percent, 41 of 54), credit was given to MFS programs/services by 98 percent (40 of 41) as well as by 100 percent (9 of 9) of key informants who provided a response.

3.2.13 Operational Outcome – Increased Willingness to Deploy

Finding #16. The majority of CF members are willing to deploy. Consistent with the literature, most key informants were of the opinion that MFS programs/services contribute to a willingness to deploy. However, the willingness to deploy has not increased in the past five years.

The findings in this section are based on document review, key informant interviews, advisory group/stakeholder consultations, and site visits.

Previous surveys indicate that the majority of CF members (approximately 75 percent) are willing to deploy on international operations or missions, with rates remaining relatively consistent between 2006 and 2009 (a range of 73.4 percent to 79.8 percent).⁶⁸ Trends from this data show a very minimal increase over this period, with a slight drop in fall 2008 followed by an overall rise again in 2009. The literature⁶⁹ indicates a link between family well-being, adaptation, and willingness to deploy. Key informant interviewees were generally of the opinion that MFS programs/services influence willingness to deploy to a great extent. More than three quarters (78 percent, 35 of 45) of all key informants felt that if MFS programs/services did not exist, military members would be less willing to deploy. However, a few base/wing commanders (21 percent, 4 of 19) stated that the lack of MFS programs/services would have no impact because deployment is an accepted condition of service.

3.2.14 Operational Outcome – Increased Retention of Military Personnel

Finding #17. Retention has improved in recent years; however, it is unlikely that this change is attributable to MFS programs/services. Nevertheless, family issues are important in making career decisions and it is likely that if MFS programs/services did not exist, military members would be less willing to stay in the military.

The findings in this section are based on document review, key informant interviews, an analysis of program data, and site visits.

CF attrition rates were on the rise after 2003, reaching a high of nine percent of the Regular Force per year before dropping significantly to 6.6 percent of the Regular Force in March 2009. Attrition rates fell even further to 4.7 percent in March 2010 before rising to 6.7 percent in March 2011. Comparing results from the 2008 and the 2010 Retention in the CF survey,⁷⁰ it appears that intentions to stay in the CF have shifted considerably. In 2008, 27 percent of CF members reported that they intended to leave as soon as another job became available, whereas only 15 percent reported this intent in 2010.⁷¹

Previous studies have found work/family balance and family issues to be among the factors considered in remaining in the CF,⁷² and frequency and length of deployment and separation from family are factors reported by members leaving the military.⁷³ Members place significant importance on their families when making decisions about their careers,⁷⁴ and spousal support has been shown to be a positive predictor of retention, organizational commitment, and willingness to deploy.⁷⁵ CF exit surveys⁷⁶ have concluded that the likelihood of members leaving the CF would be reduced by increasing members’ satisfaction with family-related aspects of military service. However, the observed change in attrition and intention to leave the CF is likely the result of changes in operational tempo, changes in the Canadian economy subsequent to the 2008 recession and a Military Personnel Retention campaign begun in 2009, not MFS programs/services.

Almost half (46 percent) of key informant interviewees felt that if MFS programs/services did not exist, military members would be less willing to stay in the military. However, a few base/wing commanders (21 percent) felt that military family services do not affect retention to a great degree. A minority of base/wing commanders (21 percent), program staff (20 percent), and C/MFRC board members (18 percent) were unsure of the impact of MFS programs/services on retention.

3.2.15 Unintended Outcomes

Finding #18. The significant support structure currently in place for military families may be creating dependencies on external support and feeding unsustainable expectations on the part of some military families for support programs/services.

Findings in this section are based on key informant interviews and advisory group/stakeholder consultations.

Key informants and other individuals consulted warned that the burgeoning support programs/services have created dependencies on these external support systems among some military families. They suggested that expectations that “someone else” would solve families’ problems were historically high and unsustainable. These lines of evidence suggested that the DND/CF as a whole should focus on promoting programs/services that help families to help themselves. Interviewees also emphasized the need to temper the image of “brokenness” in military families and focus instead on promoting families’ inherent resilience. The creation of many support programs/services for military families, especially via charitable fundraising, may have perpetuated an image of military families as “needy.” Many interviewees underscored that military families are resilient (i.e., “the strength behind the uniform”), but may require additional support in times of heightened stress (e.g., deployments). All MFS programs/services should strive to support the enhancement of resilience within the family unit and focus promotional communications and support mechanisms on the premise of inherent resilience.

3.3 Performance (Efficiency and Economy)

Under the 2009 Treasury Board Policy on Evaluation, efficiency is defined as maximizing the outputs produced with a fixed level of inputs or minimizing the inputs used to produce a fixed level of outputs (paraphrase). Economy is defined as “minimizing the use of resources … to achieve expected outcomes.”⁷⁷ For the purposes of the Policy on Evaluation, the following elements of performance are demonstrated when:

• outputs are produced at minimum cost (efficiency); and
• outcomes are produced at minimum cost (economy).

The following is a discussion of the cost of producing the outputs and outcomes of MFS programs/services. Findings in this section are based on administrative and financial data review. Prior to this evaluation, spending on MFS had not been evaluated distinctly from Casualty Support. As such, the discreet proportion of funds expended on MFS was unknown. Spending directly attributed to MFS programs/services averaged $51.8 million per year over the period FY 2008/09 to FY 2010/11 and remained stable (i.e., it did not increase much more than the rate of inflation). In addition, there was an average of $6.3 million per year spent primarily on infrastructure and employee benefits not directly attributed to MFS programs/services. Overall, public spending translates to a cost of approximately $950 per year⁷⁸ per family (Regular and Reserve Force) for MFS programs/services. MFS spending represents 0.3 percent of an overall DND budget of $19.8 billion.⁷⁹ Considering federal spending on the current National Child Tax Benefit of $1,367 annually per child per Canadian family,⁸⁰ overall public spending on MFS is relatively modest.

In addition to public funding, some MFS programs/services generate revenue through fundraising, donations, grants, and fees for services. Several programs/services make extensive use of volunteers. Leveraged resources increase the overall cost of MFS programs/services from a societal perspective, but increase the efficiency of public funds by enhancing the outputs that can be produced by the programs/services.

Demonstrated improvements in outcomes without corresponding funding increases indicate that MFS programs/services are improving their efficiency in terms of use of resources. The proportionality of public spending within the envelope has shifted over the past five years. Spending proportions have increased on prevention/support/intervention and separation/reunion activity areas (i.e., IPSC FLOs, OSISS Family Peer Support Coordination, DSCs, and HOPE) and, to a lesser degree, on family development/community integration and child/youth development/parenting support activity areas. Public spending on child and youth development and parenting support actually declined by 2.5 percent on average over the three years (i.e., Youth Model). Given that there has been no sizeable increase in funding beyond inflation, the noted shift in resource allocation towards prevention/support/intervention and separation/reunion reflects an economical responsiveness to the recent operational context and high personnel tempo.

Finding #19. Current spending does not adequately address key gaps in support to military families. Addressing significant aspects of these gaps need not require significant spending increases.

As discussed in the overall finding "Effectiveness", current spending within MFS does not sufficiently address some of the core gaps in support to military families. There is a focus on education support for dependent children of CF members through DEM and on deployment support through DSCs and MFSP efforts. However, CNAs and Community Site Visit reports reviewed for the evaluation, as well as key informant interviewees, consistently identified gaps in continuity of and/or access to childcare and health care, spousal education, and career support. Addressing significant elements of these gaps will not require significant increases to funding (e.g., advocacy, user-fees) but may require a reallocation of current spending to ensure sufficient resources are available to carry out these activities.

Finding #20. The amount of funds spent by MFS programs/services on management and administration is higher than expected and may indicate inefficiencies and duplication of effort.

MFS programs/services are predominantly delivered through MFRCs in Canada and through CMFRCs outside the country. Expenditure reports show that, from FY 2006/07 to FY 2010/11, actual expenditures were on average 95 percent of budgeted expenditures. However, on average from FY 2009/10 to FY 2010/11,⁸¹ 25 percent of C/MFRC public funding was spent on management and administration at the C/MFRCs and another 16 percent was spent on management and administration at the DMFS level, for a total of 41 percent. Overhead at this level is too high. Management and administration costs should typically be much lower (i.e., in the 15 to 20 percent range). Even charities that have to raise money to fund all of their activities are expected to keep overhead costs under 35 percent.⁸² Given the challenges encountered during the evaluation in identifying spending on MFS programs/services, accounting issues may explain the high overhead cost percentage. However, concerns about overhead and duplicate layers of bureaucracy were also raised during the consultations with other organizations and stakeholders and were observed in the document review. These lines of evidence indicate a lack of clarity in roles and responsibilities.

Finding #21. Lack of coordination among MFS programs/service may contribute to overhead costs and reduce the level of coordination/integration of programs/services.

The evaluation noted that governance of MFS programs/services is dispersed among a number of organizations (see Figure 1). Dispersed governance may result in duplication of effort, contribute to the high overhead costs noted and reduce the level of coordination/integration of MFS programs/services. While CNAs are undertaken by DMFS via C/MFRCs, other MFS programs/services also undertake community needs research (e.g., Community Recreation surveys, DQOL town halls and site visits, and DGMPRA research) that may benefit all MFS programs/services. A coordinated approach to collecting community needs information (i.e., assessing the availability of local programs/services and identifying the reasons military families do not use available services) and program/service user data (numbers of users and their demographics) should reduce research fatigue and improve efficiency. It would reduce duplication of effort, improve collaboration among MFS programs/services, and pinpoint efficiencies to address the identified needs of military families (i.e., task and resource-sharing). More streamlined data collection would also help to better track expenditures relative to MFS activity areas across all programs/services in order to inform decision-making and provide important information for the regular updating of CNAs. Furthermore, coordination of the marketing/promotional efforts of various program/service providers through a national plan would improve both awareness of MFS programs/services among military families and communication.

Almost half (46 percent) of key informant interviewees were of the opinion that there are more efficient ways to provide MFS programs/services. Suggestions included a one-stop shop (41 percent), improved communication (24 percent), better coordination/integration with services in the community (15 percent), and outsourcing some services to the private sector (15 percent). While there was no consensus on this question among survey respondents, 37 percent believed improvements could be made, 24 percent believed no improvements were required, and 30 percent did not know.

In Section 3.2.2, it was noted that key informant interviewees had observed that, at least in some locations, web sites and social media were not being used effectively in communicating. As recommended, the development of a national marketing/promotion plan that includes increased use of information technology and social media tools, as well as coordinated efforts among the various programs/service providers would improve communication and respond to this requirement.

It is expected that better coordination with services in the community can be achieved once CNAs are completed or updated. CNAs are intended to identify services available in the community but, as noted in Section 3.2.1, CNAs have not been completed at all MFRCs and are not updated on a regular basis. Other MFS programs/services might also benefit from information on the availability of programs/services in the community to reduce overlap and duplication and from information on local military family needs to ensure these are being met.

Finding #22. The greatest spending is on MFSP, which, based on survey respondents’ perceptions, has been the most successful in making progress towards intended outcomes. Spending on DEM and MFFC is the second and third highest, respectively. Both programs address components of key support requirements for military families. Attention is required for MFF in order to improve efficiency and return on investment.

On average from FY 2008/09 to FY 2010/11, spending on MFSP represented 49.7 percent of spending and 57.6 percent of total spending (including leveraging). Program familiarity was high and perceived improvements in immediate outcomes were attributed, at least in part, to MFSP.

Spending on DEM represents 32.8 percent of public spending and 28.7 percent of total spending (second highest). In addition to transitional education support to dependents in Canada, the DND/CF operate two schools overseas (SHAPE International-Canadian Section in Casteau, Belgium and AFNORTH-Canadian Section in Brunssum, the Netherlands) to provide education to eligible dependent children. This service delivery model results in an average annual spending of $16.2 million per year for 274 eligible dependents (averaged annually), or $29,548 per eligible dependent.⁸³ DEM is expected to contribute to the “continuity of family lifestyle” outcome. However, perceived improvements in this outcome were not frequently attributed to DEM by respondents. Program familiarity is relatively low, possibly because DEM is applicable to a subset of military families.⁸⁴ As education is one of the five key support requirements of military families and alternative international schools are not available in these locations, DEM provides a necessary support to military families.

Spending on MFFC represents 5.9 percent of public spending and 5.1 percent of total spending (third highest). MFFC facilitates the provision of mental health services to families of military members who are undergoing medical treatment,⁸⁵ in direct support of the CF member’s care or treatment, and primarily in the way of marital and psychological counselling/education.⁸⁶ MFFC is expected to contribute to the “access to emergency assistance” and “capacity to manage general health, crises and stressors” outcomes but attribution of perceived improvements to MFFC in these outcomes was low. However, it is worth noting that program familiarity was also low among stakeholders. Given that MFFC facilitates access to mental health support for military families and counselling staff⁸⁷ are fully resourced with eligible clients, it appears that this mental health support is being fully utilized by eligible military families.

Awareness of MFS programs and services among survey respondents and key informant interviewees ranged from 71 percent to 90 percent for most programs. Overall awareness of HOPE and MFFC was low; however, both programs are relatively new.⁸⁸ The attribution analysis revealed a number of potential flaws in the program logic model, which was reflected in specific cases of unexpected outcome attribution. Considering the logic model, HOPE and MFFC were attributed with improvements towards intended outcomes consistently less often than expected. MFF was attributed less often than expected for improvements in “capacity to manage general health” but more often than expected for improvements in “continuity of family lifestyle.” Conversely, Youth Model, IPSC, DSCs and R2MR were attributed with improvements towards intended outcomes more frequently than expected and more frequently than expected programs/initiatives. In addition, the logic model also included flaws where program/service mandates did not align with expected attribution of improvements towards outcomes (e.g., Community Recreation and R2MR were expected to contribute towards improvements in “access to emergency assistance”).

Annex A—Management Action Plan

Performance (Effectiveness)

Management Action

As part of the Defence Renewal initiative, the DND/ CF is currently assessing the feasibility of reducing the number of relocations required as part of military service. In addition to cost savings, fewer relocations should reduce disadvantages to military families in continuity of health care and spousal employment. The number of interprovincial and access issues to be addressed creates a complex stakeholder environment requiring a specialized approach for each issue area to address systemic disadvantages experienced by CF families because of provincial relocation. These include the following:

• Access to health services and a family physician. Work is currently under way with the College of Family Physicians of Canada, the Canadian Association of Interns and Residents, the Canadian Association of Surgeon and Physician Recruiters and the Canadian Conference of Medical Educators to address access to health services and access to a family physician. Target Date: Ongoing
• Access to health insurance in province of residence. Work is currently under way with provincial/territorial/regional ministries of health to address access to health insurance on posting to a new province or territory. To date, eight provinces/territories have waived the 90-day access to health care waiting period for CF families moving to a new province. The remaining provinces/territories, with one exception, are in the process of considering/making the legislative changes to waive the waiting period. Target Date: December 2013
• Spousal employment continuity. Some work is under way with the CF Liaison Council to develop a common approach with employers of Reserve Force members and military spouses. Further effort is still required to identify spousal employment sectors where sufficient numbers are qualified/employed to warrant such dialogue with organizations such as the Canadian Nurses Association, the Canadian Bankers Association, the Canadian Education Association, and the Canadian Teachers’ Federation (interim target for completion FY 2013/14). Once the spousal labour and qualification profiles are better understood, these associations will be engaged, along with other appropriate employers, to remove barriers to the mobility of military spouses between provinces and locations. Specifically, DGPFSS/DMFS will work with employment stakeholders to address the portability of professional credentials on posting to a new province or territory. Target Date: FY 2014/15
• Dependent education continuity. DGPFSS is working to integrate DEM with Quality of Life (QOL) and Military Family Services during FY 2012/13. Ministries of education and school boards will be engaged regarding portability of acquired studies between provinces and from schools in Europe back to Canada. The initial target to establish contacts and open this dialogue is end of 2013. The identification of specific areas of study that are problematic (e.g., French immersion, mathematics, etc.) and the development of plans to address these challenges are anticipated in 2014. Target Date: December 2014
• Lack of consistency in rent controls. DGPFSS will address, in conjunction with the Canadian Forces Housing Agency and the Director General Compensation and Benefits, the regional inconsistencies in provincial rent control methods. This work will include a review and update of the appropriate Queen’s Regulations and Orders in 2013.Target Date: January 2014
• Access to childcare. DGMPRA and DGPFSS have conducted or sponsored considerable research in this area over a number of years. A review and analysis of this research should be completed in 2013. A DGPFSS staff-led working group will explore with child and youth agencies and with provincial ministries of education the ways and means to improve child care capacities in communities that have significant concentrations of military families. Target Date: September 2013

Performance measures are to be developed and will include simple threshold reporting, trend reporting, and external benchmarking/best practice comparison where available.

The management of these inter-governmental issues remains a very sensitive and complicated venture.

OPI: CMP

Target Date: April 2015

Management Action

As part of the Defence Renewal initiative, the DND/ CF are currently assessing the feasibility of reducing the number of relocations required as part of military service. In addition to cost savings, fewer relocations should reduce disadvantages to military families in continuity of access to childcare. A report is forthcoming in the next six months which will inform methods and options for moving forward appropriately on this issue. Subsequent to the report, however, options to evolve existing childcare policy and programming will be submitted for consideration by the CF leadership.

OPI: CMP

Target Date: August 2013

Management Action

The Military Family Services CNA Toolkit, launched in September 2012, is now in use on a three-year renewal cycle by each MFRC in accordance with Parameters for Practice and the memoranda of understanding signed between CMP and MFRCs annually.

Options will be developed for further coordination and potential consolidation of different CNAs into a single CNA for the military family community. For example, the provision of Community Recreation Program services must also be identified through an annual CNA. Currently, PSP Community Recreation staff correlate and analyze the results of the CNA. However, coordination of effort with MFS to develop a single CNA for military families will be considered in 2013/14.

The provision of a community recreation component is one of the key elements in the non-public property funding distribution model. It recognizes the essential outcomes that non-public property beneficiaries should receive regardless of location. Bases/wings must provide Community Recreation Program services as identified through their annual CNA. Non-public property funds are then allocated to provide the identified programs and services.

OPI: CMP

Target Date: FY 2013/14

Management Action

Military family perceptions of nationally-mandated services do not necessarily reflect the reality because there is a lack of awareness of the difference between national programs and services and local site-specific offerings. This can lead to the perception of inequality of nationally mandated services. A national communications plan is required to change awareness and understanding of the national program elements and the local program offerings.Interim Target Date: April 2014

In accordance with Parameters for Practice and the memoranda of understanding signed between CMP and MFRCs annually, nationally-mandated services are available at every MFRC in the areas of Prevention Support and Intervention; Child Youth and Parenting Support; Family Separation and Reunion; and Community and Personal Development through the MFSP. Each year, MFRCs must submit funding applications outlining how they will offer programs/services in the four mandated areas. Based on an extensive funding review process, QOL/MFS allocates funds and monitors usage (implementation of the mandated programming) via quarterly reporting mechanisms. It is important to note that one of the service programs (Youth Model) was formally eliminated as part of departmental reductions. However, the funding for that enhanced program was cyclical, and both MFRCs and PSP Community Recreation will continue to offer programming for youth. Target Date: complete and ongoing

While the need for prescriptive recreation programming and mandated sets of activities have not been indentified, the PSP division has recently launched quality assurance programs that will ensure that all programming delivered at CF locations maintain a high level of quality. HIGH FIVE is a comprehensive quality assurance program for organizations providing community recreation and sport programming. Community Recreation and MFRC staff have been trained and are in the process of developing policies, procedures, risk management, program evaluation, and staff development strategies to reflect the HIGH FIVE framework. Target Date: complete and ongoing

OPI: CMP

Target: FY 2013/14

Management Action

A draft operational plan has been developed to support the Morale and Welfare strategy for the CF. A communications plan will follow, in coordination with the overarching Morale and Welfare communications plan, and in consultation with other service providers in the Department (i.e., Canadian Forces Housing Agency and DGHS).

A working group is being formed to leverage the single portal function of FamilyForce.ca for military families, and to extend its reach to the broader community of military families, honouring the philosophy of “One community, One million strong.”

OPI: CMP

Target Date: November 2013

Performance (Efficiency and Economy)

Management Action

See response to Recommendations 1 and 2.

OPI: CMP

Target Date: August 2013/April 2015

Management Action

In the current fiscal environment all programming will be further scrutinized to seek efficiencies while preserving core services.

OPI: CMP

Target Date: April 2015

Management Action

It is uncertain whether overhead costs can be materially reduced. One of the coordinating mechanisms may already exist in the form of the Social Sciences Research Review Board. If employed appropriately, this board could produce much of the effect being recommended by CRS.

OPI: CMP

Target Date: FY 2013/14

Annex B—Organizational Structure of the 12 MFS Programs/Services Evaluated

*Although Next of kin support was not included in the logic model, it was accounted for in the financial analysis for the evaluation.

Figure 1. MFS Programs/Services Evaluated. This organizational structure shows the 12 MFS programs/services included in the evaluation.

Annex C—Detailed Descriptions of MFS Programs/Services

1. MFSP. Program designed to help families manage the unique stresses of the CF lifestyle through personal, family, and community development. The aim is to promote health and well-being and provide needed information and referral to CF families by coordinating and providing a range of services to military families.

2. OSISS. The mission is to establish, develop, and improve social support programs for CF members, veterans and their families affected by operational stress. Additionally, the program aims to provide education and training to the CF community to create an understanding and acceptance of operational stress injuries.

3.HOPE. A national bereavement network that connects the families of deceased CF personnel and veterans.

4. IPSC. A network of support centres whose role is to provide integrated “one-stop service” to meet the needs of ill and injured personnel from the Royal Canadian Navy, Canadian Army, and Royal Canadian Air Force.

5. MFF. MFF assists military families faced with unforeseen and often immediate needs that have resulted due to conditions of service.

6. DEM. DEM manages the education compensation and benefits programs for DND members to ensure that their dependent children obtain elementary and secondary education.

7. Community Recreation. Personnel and Family Support Services deliver morale and welfare programs, services, and activities through three operating divisions – Canadian Forces Exchange System, SISIP Financial Services, and PSPs. Community Recreation fits within the PSPs. The mandate of the PSPs is to enhance the morale and welfare of the military community.

8. R2MR. R2MR is a pre- and post-deployment education and training program that includes a series of briefings/modules that address each stage of the deployment cycle for military personnel and their families.

9. Take a Stand Against Family Violence (Family Violence). A series of awareness campaigns that occur annually among military communities across Canada. Their purpose is to communicate the importance of the issue of family violence and promote awareness, responsibility, and prevention within the CF.

10. MFFC. MFFC formally recognizes the importance of family inclusion in the care of the member and provides the opportunity for family members to be seen in CF clinics in support of the member and to participate in psycho-education programs.

11. DSCs. The role of DSC is to support families and CF personnel during extended absences for professional reasons, including deployments, courses, exercises, domestic operations, temporary duties, and postings with imposed restriction.

12. Youth Model. The DGPFSS funds PSP – Community Recreation departments and MFRCs in all CF communities to provide programs and services that respond to the needs and interests of youth from military families.

Annex D—Logic Model

Figure 2. Logic Model for MFS Programs/Services. 1.MFSP; 2. OSISS Family Peer Support Coordinators; 3. HOPE volunteers/management; 4. IPSC Family Liaison Officers; 5. MFF; 6. DEM; 7. Community Recreation; 8. R2MR; 9. Family Violence; 10. MFFC; 11. DSC; 12. Youth Model.

Annex E—Evaluation Matrix

Table 2. Evaluation Matrix—Relevance. This table shows the lines of evidence that were examined to address each of the relevance questions for the evaluation.

Table 3. Evaluation Matrix—Performance (Effectiveness) – Immediate Outcomes. This table shows the lines of evidence that were examined to address each of the performance questions for the evaluation in relation to the immediate program outcomes.

Table 4. Evaluation Matrix—Performance (Effectiveness) – Intermediate Outcomes. This table shows the lines of evidence that were examined to address each of the performance questions for the evaluation in relation to the intermediate program outcomes.

Table 5. Evaluation Matrix—Performance (Effectiveness) – Ultimate Outcomes. This table shows the lines of evidence that were examined to address each of the performance questions for the evaluation in relation to the ultimate program outcomes.

Table 6. Evaluation Matrix—Performance (Effectiveness) – Unintended Outcomes. This table shows the lines of evidence that were examined to address each of the performance questions for the evaluation in relation to unintended program outcomes.

Table 7. Evaluation Matrix—Performance (Effectiveness) – CF Operational Objectives. This table shows the lines of evidence that were examined to address each of the performance questions for the evaluation in relation to the CF operational objectives program outcomes.

Table 8. Evaluation Matrix—Performance (Efficiency and Economy). This table shows the lines of evidence that were examined to address each of the performance questions for the evaluation in relation to the efficiency and economy issues.

___________________________________________________________________________________________________________________________

Footnote 1 These include: Operational Stress Injury Social Support (OSISS) Family Peer Support Coordinators, Helping Others by Providing Empathy (HOPE) volunteers/management, Integrated Personnel Support Centres (IPSC) Family Liaison Officers (FLO), Military Families Fund (MFF), Dependent Education Management (DEM), Community Recreation, Road to Mental Readiness (R2MR), Family Violence, Member Focused Family Care (MFFC), Deployment Support Centres (DSC), and Youth Model.

Footnote 2 Systemic issues result from separate administration (provincial/territorial) of health care, education, professional recognition, and inconsistencies across administrations.

Footnote 3 Interviewees (base/wing commanders, Canadian and Military Family Resource Centre (C/MFRC) board members, program staff) and survey respondents.

Footnote 4 Community engagement: integration and interaction (connectedness) of individuals and families with their local community (either CF or civilian).

Footnote 5 Although approximately half of the total $200 million allocated to this PAA element (FY 2010/11) was estimated to be expended on MFS, the results of this evaluation indicate that spending on MFS is $60 million (direct and indirect costs).

Footnote 6 A senior official, either civilian or military, who has direct accountability to the DM or the CDS.

Footnote 7 DND Report on Plans and Priorities. Reports for 2009-10 through 2012-13; Pepin and Sudom, 2008; Adams, Hall and Thompson, 2009; CF Family Covenant. (http://www.familyforce.ca/sites/Kingston/EN/About%20Us/Pages/Covenant.aspx)

Footnote 8 DND. Briefing Note: Morale and Welfare Programs in the Canadian Forces.A-PS-110-001/ACG-002, Section 3-44.

Footnote 9 The DND/CF use a risk-based approach to determine the timing of evaluations based on: 1) importance (criticality of the program to the achievement of Defence strategic outcomes); 2) likelihood of program non-performance; and 3) materiality (program expenditures and/or public visibility/sensitivity).

Footnote 10 Assistant Deputy Minister (Finance and Corporate Services), DAOD 5044-1, Families. (http://www.forces.gc.ca/en/about-policies-standards-defence-admin-orders-directives-5000/5044-1.page)

Footnote 11 CF Family Covenant. (http://www.familyforce.ca/sites/Kingston/EN/About%20Us/Pages/Covenant.aspx)

Footnote 12 Includes full-time CF members who are single parents with children.

Footnote 13 Includes married, common-law, and same-sex partners.

Footnote 14 Children and youth up to the age of 18.

Footnote 15 Note the “Policy Amendment: MFSP Populations Served,” issued by DMFS in 2011, emphasizes the inclusion of parents of deployed Regular Force members, families of civilians deployed on operations with the CF, and the families of the fallen. In addition, the amended policy extends the period of time for post-deployment support to parents of all deployed CF personnel from two to three years.

Footnote 16 CMP Morale and Welfare (web).

Footnote 17 Canadian Forces Personnel and Family Support Services: About Us (web). (https://public.cfpsa.com/en/AboutUs/CFPFSS/Pages/default.aspx)

Footnote 18 Note that relevant program data was obtained for the following programs: MFSP, OSISS, IPSC, MFF, Community Recreation, R2MR, and Family Violence.

Footnote 19 CRS (2009) Evaluation of Support to Injured CF members and their Families; CRS (2002) Follow-up Program Evaluation Study on the Military Family Services Program.

Footnote 20 Other military family members were not included in primary data collection as feedback had been frequently and extensively sought in the past. This existing data was reviewed for the current evaluation. Further explanation can be found in section 2.4 Limitations.

Footnote 21 These include Director General Naval Personnel, Director General Air Personnel, and Director Land Personnel Management.

Footnote 22 See for example: “Treatment at CFB Petawawa in ‘crisis’, report says; Authors issue scathing analysis of lack of care for Afghan war veterans,” Ottawa Citizen newspaper, 3 May 2012

Footnote 23 Non-public funds are acquired from fundraising, donations, grants, and fees for services or other revenue-generating activities of C/MFRCs. While non-public funds were not used in estimating unit costs, leveraging of these funds was considered an indicator of program efficiency.

Footnote 24 In most cases, one contact was provided for each survey respondent group at each of the 39 bases (i.e., three individual contacts per base).

Footnote 25 Not counting duplicates and instances where positions are not filled.

Footnote 26 For example: Family Quality of Life Study (every two years), MFSP community site visits (every three years), C/MFRC CNA (ongoing); MFSP participant survey (every year since 2006, 2012 re-tooling year, next survey 2013), Community Wellness Survey (every two to three years); CF Recreation Survey (ad hoc), Your Say Survey (semi-annually since 2005), CF Retention Survey (every two years), CF Exit Survey (ongoing), CF Health and Lifestyle Information Survey (every four years), Recruit Health Questionnaire (ongoing), Defence Ethics Survey (every two to three years). In addition, there are many single/unique application studies/surveys that have been conducted by DND/CF (i.e., Perstempo, R2MR, OSISS, etc.).

Footnote 27 Data derived from: Spousal/Partner Employment and Income Project: How Do Canadian Forces Spouses Compare? (November 2010) Director General Military Personnel Research and Analysis (DGMPRA), Table 7.

Footnote 28 DND Report on Plans and Priorities 2010; Departmental Performance Report 2010; CF Ombudsman Annual Report 2010.

Footnote 29 CF Ombudsman Annual Report 2011.

Footnote 30 Standing Committee on National Defence and Veterans Affairs (October 1998). Moving Forward: A Strategic Plan for Quality of Life Improvements in the Canadian Forces.

Footnote 31 CF Family Services Summit 2008; CRS. (July 2009) Evaluation of Support to Injured CF Members and their Families; CF Ombudsman Annual Report 2010-11.

Footnote 32 Percentages represent overall average across key informants and survey respondents, respectively, irrespective of location, isolation factor or centre size/population.

Footnote 33 As reported during evaluation consultation.

Footnote 34 CF Ombudsman Annual Report, 2011-12; key informant interviews and consultations.

Footnote 35 DND Report on Plans and Priorities 2012.

Footnote 36 DND Defence Priorities to the Government of Canada Priorities. Report on Plans and Priorities 2010.

Footnote 37 DND Briefing Note: Departmental Directives 2011-2016, Personnel.

Footnote 38 National Defence is ready to meet government Defence expectations” (PAA 2.0).

Footnote 39 “Defence operations improve peace, stability and security wherever deployed” (PAA 3.0).

Footnote 40 DND Report on Plans and Priorities 2012, Table “Defence Team Personnel Support.”

Footnote 41 Dursun, S. (2009) Spousal Support, Retention and Readiness of CF Members. Psychological Military Review.

Footnote 42 CF Family Covenant. (http://www.familyforce.ca/sites/Kingston/EN/About%20Us/Pages/Covenant.aspx)

Footnote 43 Tanner et al. (2008). Family Support in the Canadian Forces: An Overview of Research Conducted since 1990. Defence Research and Development Canada.

Footnote 44 Eighty-seven percent of key informants (41 of 47 who provided a response) and 91 percent of survey respondents (49 of 54).

Footnote 45 Twenty-eight percent, or 13 of 47 key informants.

Footnote 46 Systemic issues result from separate administration (provincial/territorial) of health care, education, professional recognition, and inconsistencies across administrations.

Footnote 47 DQOL. Initiatives Backgrounder; DQOL. Quality of Life Approved Business Plan 2012-13.

Footnote 48 DMFS Community Needs Assessment Toolkit. (https://www.cfmws.com/en/aboutus/mfs/governanceandaccountability/pages/communityneedsassessmenttoolkit.aspx)

Footnote 49 DMFS Community Site Visits (2012), MFS Participant Surveys (2008-2011), MFRC Evaluation (2008).

Footnote 50 Key informant interviews, online survey, review of previous surveys and studies, and other consultations.

Footnote 51 Recreation and Leisure Program Survey (2012): 41 percent of CF families reported involvement in volunteering; CF Retention Survey (2008): 73 percent reported family involvement in activities in their local community, some of which may have been volunteering.

Footnote 52 For example, MFSP, OSISS, Community Recreation, R2MR.

Footnote 53 Your Say Survey (all results since 2005); CF Ombudsman Annual Report FY 2011/12.

Footnote 54 MFS Participant Survey (all years: 2008-2011); MFRC Evaluation (2008).

Footnote 55 Dunn, J., Ford, K. and Flemming, S. (2005) PERSTEMPO Qualitative Data: CF Member Focus Group Findings, Operational Research Division Technical Report 2005-09.

Footnote 56 Eighty-five percent, or 46 of 54 of survey respondents.

Footnote 57 Sixty-five percent, or 13 of 20 key informant interviewees and 65 percent, or 35 of 54 of survey respondents.

Footnote 58 Sudom and Eyvindson. (2008) Effects of Personnel Tempo on Military Members, their Families, and the Organization: An Annotated Bibliography. DGMPRA; Pepin and Sudom (2008) Family Resilience: An Annotated Bibliography. DGMPRA.

Footnote 59 Dursun, S and Sudom, K. (2009). Impacts of Military Life on Families: Results from the Perstempo Survey of Canadian Forces Spouses. DGMPRA.

Footnote 60 Dursun, S. (2009) Spousal Support, Retention and Readiness of CF Members. DGMPRA; Dursun, S. (2006). Results of the 2005 Spouse PERSTEMPO Survey. DQOL; Sudom, K. and Dursun, S. (2007) PERSTEMPO in the Canadian Forces, DGMPRA; Dursun, S. and Sudom, K. (2009). Impacts of Military Life on Families: Results from the PERSTEMPO Survey of Canadian Forces Spouses. DGMPRA.

Footnote 61 Dursun, S. (2009) Presentation: Family and Individual Operational Effectiveness; Dursun, S. (2006), ibid.

Footnote 62 Most C/MFRC board members (80 percent, or 8 of 10) and 72 percent (39 of 54) of survey respondents felt there has been more family support for members’ military careers in the past five years, as compared to 47 percent (9 of 19) of base/wing commanders and 29 percent (4 of 14) of program staff.

Footnote 63 Most C/MFRC board members and base/wing commanders interviewed felt that military members were more confident (91 percent, 10 of 11; and 89 percent, 17 of 19 respectively). The large majority of survey respondents (89 percent, 48 of 54) also indicated that they thought military members were confident.

Footnote 64 Forty-seven percent, or 7 of 15 program staff members interviewed.

Footnote 65 MFSP Survey (2011); DGMPRA. (2011) Top Line Findings from the Military Wellness Survey.

Footnote 66 MFSP Survey (2011).

Footnote 67 Fifty-three percent (8 of 15) of program staff, 27 percent (3 of 11) C/MFRC board member and 21 percent (4 of 19) base commander interviewees did not know; 13 percent (7 of 54) survey respondents did not answer or did not know.

Footnote 68 Urban, S. (2007, 2008, 2009) Your Say Survey results. DGMPRA; Urban, S. and Yelle, M. (2010) Your Say Fall 2008 Results and Comparison across Administrations 2004-2008. DGMPRA.

Footnote 69 Adams, Hall and Thompson (2009) Military Individual Readiness. Humansystems Incorporated on behalf of National Defence; McCreary, Thompson and Pasto (2003) Predeployment Family Concerns and Soldier Well-Being: The Impact of Family Concerns on the Predeployment Well-Being of Canadian Forces Personnel. Can J Pol Sec Svcs, 1(1).

Footnote 70 Administered to select military occupations based on their attrition trends. Koundakjian, K and Michaud, K. (February 2011) Retention in the Canadian Forces: Descriptive Results of the 2008 Retention Survey. DGMPRA.

Footnote 71 Note that due to changes in the survey questionnaire, it is not possible to compare other results obtained for 2008 and 2010.

Footnote 72 DND (May 2010) Backgrounder: Recruiting and Retention in the Canadian Forces. BG 10.008; Sudom and Eyvindson (2008) Effects of Personnel Tempo on Military Members, their Families, and the Organization: An Annotated Bibliography. DGMPRA.

Footnote 73 Sudom, K. and Dursun, S. (January 2006) The Relationship Study: Qualitative Report. DGMPRA.

Footnote 74 Dunn, J. (2004) A State of Crisis? An Exploratory Examination of Family Breakdown in the CF. Director Strategic Human Resources and DQOL, DND.

Footnote 75 Dursun, S. (2009) Spousal Support, Retention and Readiness of CF Members. DGMPRA.

Footnote 76 Michaud, K. (April 2011) The Canadian Forces Exit Survey 2005-2008: Factor Analyses and Demographic Differences. DGMPRA.

Footnote 77 Treasury Board of Canada Secretariat. Policy on Evaluation, April 1, 2009. (http://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=15024andsection=text#cha4)

Footnote 78 Averaged over the last three years and based on census of 60 903 CF families (FY 2008/09 to FY 2010/11). Note: without accurate user information, this cost per family may be understated.

Footnote 79 According to the PAA allocation and averaged over FY 2008/09 to FY 2010/11.

Footnote 80 Basic Canada Child Tax Benefit. From National Child Benefit website. (http://www.nationalchildbenefit.ca/eng/06/cctb_children.shtml)

Footnote 81 Data was only available for two years.

Footnote 82 Canada Revenue Agency (2012) Fund Raising by Registered Charities, Guidance, Ref. No. CG-013. (http://www.cra-arc.gc.ca/chrts-gvng/chrts/plcy/cgd/fndrsng-eng.html)

Footnote 83 Average computed over 2008/09 – 2010/11. Data provided by DEM Financial Manager, CMP (email).

Footnote 84 DEM provides education at CF schools in Germany and Belgium, lodging for dependents in grades 11 and 12 to complete their schooling in province for those whose families are deployed out of province or out of country, tutoring in cases where there are changes in curriculums due to differences between provinces or countries, and support with the language if families are deployed to a province with a different language of instruction.

Footnote 85 MFFC Instruction Document (5100-07), 25 June, 2007.

Footnote 86 As per consultation with a service delivery coordinator from Director Mental Health Services.

Footnote 87 Counselling staff salaries comprise the full estimated expenditures for MFFC, as per consultation with the service delivery coordinator from Director Mental Health Services.

Footnote 88 HOPE was created in 2006 and MFFC in 2007.