Engaging Canadian Women with a Career in the Armed Forces
The Impact and Innovation Unit would like to thank:
- Dr. Nicole Robitaille, Assistant Professor
Smith School of Business, Queen’s University
- Dr. Monica Soliman, Research Advisor
Employment and Social Development Canada
- Lucie-Anne Besner, Director, Marketing and Advertising
Department of National Defence
- Kristen Davison, Acting Senior Communications Advisor
Department of National Defence
- Christel Gallant, Communications Advisor
Department of National Defence
In partnership with the Department of National Defence (DND), the Impact and Innovation Unit (IIU) conducted a marketing trial in an effort to understand what works in advertising aimed at increasing engagement among women with a career in the Canadian Armed Forces (CAF). The trial follows an earlier research project by the IIU which examined qualitative and quantitative data using a behavioural lens in an effort to identify barriers to the recruitment of women and provide recommendations to address those barriers. Behavioural insights (BI) is a multidisciplinary approach that applies knowledge and tools from the behavioural sciences, including behavioural economics, psychology, and social marketing, to encourage positive changes in behaviour.
Seven ads were tested on both Facebook and Instagram in both official languages and unique click through rates (UCTR) were used as a measure of engagement. UCTRs ranged between 0.48% and 1.04% on Instagram and 1.63% and 3.13% on Facebook, in some cases exceeding the industry benchmark engagement rate of approximately 1% for Government of Canada social media advertising. The best performing ads used facts to challenge misconceptions about women in the CAF, delivered a specific value proposition about the benefits of a career in the CAF, or included graphics that evoked traditional and patriotic imagery such as the Armed Forces badge. The trial also tested the importance of conceptual congruence between the images and text in social media advertising. The data suggests that conceptually aligning images and text is associated with increased engagement. There was little difference in engagement between ads when presented in English or French. The highest levels of engagement were measured in Quebec and Central Canada, while the lowest click through rates were seen in Northern Canada. Ads that promoted the CAF’s financial support for education performed better on Instagram than on Facebook. Each ad was designed leveraging behavioural science principles. The ads were viewed over 2.5 million times and differences in UCTR between the ads inform findings that can be used to design future campaigns.
This trial applies behavioural science principles and an experimental approach to marketing designed to help isolate key determinants of the performance of a given ad. Used in conjunction with numerous other efforts underway, this approach can assist DND in achieving its target of 25.1% women members in the CAF by 2026.
Recognizing the importance of an inclusive and diverse organization, the Canadian Armed Forces has made a commitment to increase the number of women to comprise a minimum of 25.1% of its workforce by 2026. As of December 2017, achieving this target will require the recruitment of approximately 10,000 women.
A study completed by the Impact and Innovation Unit (IIU) and published in June 2017 focused on three main areas in which behavioural insights can help increase the number of women in the CAF:
- The recruitment process;
- Recruitment marketing and communications;
- And policies and guidelines within the CAF.
To identify possible barriers to the recruitment of women, the IIU examined qualitative and quantitative data from numerous primary and secondary sources through a behavioural lens. This included data examined from surveys conducted with CAF members, jurisdictional analyses, and consultations with experts. Members of the IIU also went through the CAF application process and documented their experiences. Based on these findings, and barriers identified, approximately twenty recommendations were identified.
In addition, the report identified two key areas in which policy and program experimentation techniques could be used to test the effectiveness of CAF messaging targeting both current and potential applicants. The first was identifying the most effective format and process for applicant correspondence. The CAF continues to leverage behavioural insights principles to make the application process better and more inclusive.
The second area was to test the application of behavioural insights to recruitment marketing and identify the most effective messages in driving engagement with a potential CAF career.
Research commissioned by the CAF and DND suggests that familiarity with the CAF as a career choice among Canadians is limited. With this in mind, advertising campaigns like the one in this trial aim to increase overall awareness of the CAF as an employer of choice by delivering messaging not only to potential applicants, but to Canadians in general. The IIU worked with members of the CAF and communications professionals at DND to apply the principles of behavioural science to advertising messages with the aim of generating interest in the career options available.
This trial is innovative in its application of behavioural science principles to advertising but also for the experimental design and approach it employs. The campaign is designed to isolate the discrete effects of particular messages and images by comparing them against other ads. This approach represents a way to better understand what works in advertising through experimentation to ensure that policy objectives can be met.
The IIU and the Department of National Defence, designed and ran a social media marketing trial on two social media platforms – Facebook and Instagram from the weeks of September 18 to October 1, 2017. The objective was to determine best practices to increase engagement among Canadian women with a career in the CAF.
A total of seven ads were designed and tested over the three-week period. The ads were designed taking into account past research conducted by the IIU, knowledge and experiences from DND, and by leveraging behaviourally-informed marketing best practices. Copy and creative were kept as similar as possible across Facebook and Instagram in order to ensure maximum comparability between platforms. Sample sizes and outcome data were designed such that analysis of ads at the provincial and territorial level was possible.
Basic Training (Ad 1) - The IIU’s previous report identified apprehension over basic training as an important barrier to entry for women considering a career in the Armed Forces. This ad uses social norms that are specific to women to help break down the widespread assumptions that underlie the perception that basic training is difficult for women. The previous report identified the physical demands of training as a key concern for women considering a CAF career. The chosen image includes a woman leading men in a physical training exercise and seeks to challenge concerns over relative fitness levels.
Tuition (Ad 2) – The IIU’s previous report identified financial support for education as a key incentive for young women considering a career in the Armed Forces. This message is a value proposition that makes this incentive salient.
Badge Neutral (Ad 3) - The badge ad was designed to convey the most neutral message possible. It uses the Armed Forces badge instead of a picture of a service member and this traditional image is meant to evoke a sense of patriotism.
Challenge (Ad 4) - This ad frames a career in the armed forces as an exciting and challenging lifestyle. The use of ‘exciting challenges’ and the image of a woman in full combat uniform in this ad is intended to reflect this frame.
Humanitarian (Ad 5) – In contrast to ad 4, this ad frames a career in the Armed Forces in terms of helping others. The IIU’s previous research suggests that the humanitarian aspect of a CAF career is appealing to women. The chosen image of a woman helping an infant out of a helicopter during an aid mission is used to reinforce the humanitarian frame.
Challenge - Neutral Image (Ad 6) - This ad repeats the language from ad 4, with a more neutral picture. This allowed for the isolation of the relative effects of copy and creative in these ads.
Humanitarian - Neutral Image (Ad 7) - Similar to ad 6, this ad allowed for the isolation of the effects of copy and creative for the humanitarian message.
The campaign used Facebook and Instagram to target a national sample of 18 to 34 year old Canadian women and received over 2.5 million impressions. After a review of social media platforms, Facebook and Instagram were chosen because of their large user base among the target population. The total budget for the campaign was $25,000, which was evenly split between the two platforms. The seven ads provided the basis for two separate tests:
- Message Comparison Test (Ads 1-5)
Ads 1-5 each reflect a different message designed to attract interest in a career in the CAF.
- Image Message Congruence (Ads 4-7)
Ads 4-7 form an experiment which tests the importance of having the images and text in a given ad line up conceptually; how important is it that the image in an ad conveys the same message as the copy in the caption? To answer this question, both the challenge and humanitarian ads were displayed with a congruent picture (the woman holding the gun or the woman carrying the child – ads 4-5) as well as a neutral picture (the woman in the snow - ads 6-7).
Due to the Facebook and Instagram ad auction system, true randomization was not possible. The ad auction system identifies users eligible to see an ad and then enters them into an auction. Three factors determine which ad wins the auction and is displayed:
Advertisers bid on how much they would be willing to pay for a given outcome, such as link clinks.
Estimated Action Rates
Estimated action rates represent how likely Facebook thinks a user is to take action on a particular ad. This is affected by the user’s previous actions.
Ad Quality and Relevance
The overall quality of an ad as measured by feedback the ad has received in addition to a relevance score based on the users interests.
In order to maintain the validity of the test, we took the following approach:
- Advertiser bid was manually kept constant across ads
- Total budget allocated to each ad was kept equal
- Target population was the same for all ads
- All ads were designed to be of equal quality and relevance
In this study, engagement with advertising was measured primarily by unique click through rate (UCTR). While UCTR was available at an aggregate national level, it was unavailable for provincial/territorial analysis and so CTR was used instead. UCTR was the preferred metric because it helps avoid double counting if a particular user is repeatedly exposed to and/or clicks on a particular ad. All ads lead to the CAF recruitment page, which has information on a career in the armed forces and how to apply.
Engagement was measured by the proportion of users that clicked through on each ad and differences in these engagement rates inform the findings of this trial. Reach and impressions were approximately equally distributed between the seven ads. UCTRs ranged between 0.48% and 1.04% on Instagram and 1.63% and 3.13% on Facebook, in some cases exceeding the industry benchmark engagement rate of approximately 1% for Government of Canada social media advertising. Complete data on reach, impressions and click through rates are available in Appendix B.
The best performing ads as measured by unique click through rates were the badge, basic training, and tuition ads when combined across both Instagram and Facebook and in English and French. Regardless of whether they were paired with a congruent or neutral image, the challenge and humanitarian ads performed relatively poorly in comparison. (Figure 1)
On Facebook, we see a similar pattern as in the combined results, with the badge, basic training, and tuition ads taking the top three spots and the challenge and humanitarian ads performing relatively worse. In relation to the other ads, the tuition ad performed better in French than it did in English. (Figure 2)
On Instagram, we see that the badge ad achieves a high UCTR and that the tuition ad does considerably better than the basic training ad (EN: p=0.006 FR: p=0.002). In contrast, the humanitarian messaging paired with the congruent picture achieves a particularly low unique click through rate in both languages on Instagram. (Figure 3)
The test of congruence provides evidence that having text and imagery line up conceptually drives engagement with advertising. In the largest sample, English Facebook, there is a statistically significant and meaningful positive difference between the two congruent and two neutral ads (p=0.00). This means that the congruent ads achieved higher UCTRs than their neutral counterparts. In French Facebook, the difference is positive, but not statistically significant. On Instagram, higher UCTRs are not observed for the congruent ads compared to the neutral ads. The null and negative results are primarily driven by particularly low engagement with the humanitarian congruent ad, which was the worst performing ad in this campaign. (Figure 4)
The campaign achieved high levels of engagement in Central Canada and the Maritimes while Ontario and British Columbia were very close to the national average CTR for this trial. Much lower click through rates were observed in the territories. Analysis was conducted to determine if provinces or territories in which the CAF has higher levels of recruitment also engaged with this campaign at higher levels. However, no statistically significant relationship between representation in the CAF and click through rates by province or territory was found. (Figure 5)
Figure showing combined unique clickthrough rates (UCTR). The rate for the Badge ad is 2.25%. The rate for the Basic Training ad is 1.87%. The rate for the Tuition ad is 1.66%. The rate for the Challenge ad is 1.45%. The rate for the Humanitarian Neutral ad is 1.29%. The rate for the Challenge Neutral ad is 1.26%. The rate for the Humanitarian ad is 1.15%.
Figure showing unique clickthrough rates (UCTR) on Facebook. The rates for English ads are 3.13% for the Badge ad, 3.07% for the Basic Training ad, 2.22% for the Tuition ad, 2.03% for the Challenge ad, 2.01% for the Humanitarian ad, 1.67% for the Humanitarian Neutral ad, and 1.63% for the Challenge Neutral. The rates for French ads are 2.93% for the Tuition ad, 2.92% for the Basic Training ad, 2.61% for the Badge ad, 2.18% for the Challenge ad, 1.98% for the Challenge Neutral ad, 1.92% for the Humanitarian Neutral ad, and 1.76% for the Humanitarian ad.
Figure showing unique clickthrough rates (UCTR) on Instagram. The rates for English ads are 1.04% for the Badge ad, 0.84% for the Tuition ad, 0.79% for the Challenge ad, 0.79% for the Humanitarian Neutral ad, 0.77% for the Challenge Neutral ad, 0.74% for the Basic Training ad and 0.58% for the Humanitarian ad. The rates for French ads are 0.88% for the Badge ad, 0.82% for the Tuition ad, 0.74% for the Challenge Neutral ad, 0.73% for the Humanitarian Neutral ad, 0.62% for the Basic Training, 0.62% for the Challenge ad, and 0.48% for the Humanitarian ad.
Figure comparing the unique clickthrough rates (UCTR) of congruent versus neutral Challenge and Humanitarian ads. For English ads on Facebook, the congruent Challenge ad had a UCTR of 2.03% and the congruent Humanitarian ad had a UCTR of 2.01%. The neutral Challenge ad had a UCTR of 1.63% and the neutral Humanitarian ad had one of 1.67%. There is a statistically significant difference between the two congruent and two neutral English ads (significant at the 99% confidence level).
For French ads on Facebook, the congruent Challenge ad had a UCTR of 2.18% and the congruent Humanitarian ad had a UCTR of 1.76%. The neutral Challenge ad had a UCTR of 1.96% and the neutral Humanitarian ad had one of 1.92%.
For English ads on Instagram, the congruent Challenge ad had a UCTR of 0.79% and the congruent Humanitarian ad had one of 0.58%. The neutral Challenge had a UCTR of 0.73% and the neutral Humanitarian ad has one of 0.79%. There is a statistically significant difference between the two congruent and two neutral English ads (significant at the 99% confidence level).
For French ads on Instagram, the congruent Challenge ad had a UCTR of 0.62% and the congruent Humanitarian ad had a UCTR of 0.48%.
Figure showing click through rates (CTR) for the campaign by province or territory. The national average is just over 1.4%. The rates for Quebec (+0.09%, significant at the 99% confidence level), Nova Scotia (+0.05%, significant at the 90% confidence level), Alberta (+0.02%), Manitoba (+0.01%), New Brunswick (+0.01%), and Newfoundland and Labrador (+0.01%) were above this average. The rates for Ontario (-0.04%, significant at the 99% confidence level), British Columbia (-0.08%, significant at the 99% confidence level), Saskatchewan (-0.11%, significant at the 99% confidence level), Prince Edward Island (-0.10%, significant at the 90% confidence level), Yukon (-0.29%, significant at the 90% confidence level), Northwest Territories (-0.44%, significant at the 99% confidence level), and Nunavut (-0.81%, significant at the 99% confidence level) were below this average.
Badge Ad's Strong Performance
The badge ad (Ad 3) achieved a top three unique click through rate in all platforms and languages. Although further testing is required to isolate the determinants of this ad’s success, possible reasons might be that users identified with the regalia of the badge itself which may have evoked patriotic sentiments, or that including the Government of Canada logo drove engagement. The badge may also look different than other ads on the platforms, resulting in it standing out from other images.
Challenge Assumptions and Propose Value
The success of the basic training (Ad 1) and tuition (Ad 2) ads suggest that social media marketing campaigns that challenge assumptions or make a specific value proposition about a career in the armed forces are effective in driving engagement. Social media might be an ideal platform for challenging assumptions about a career in the CAF with facts and statistics in particular because it allows users to engage with that messaging through liking, sharing, and commenting.
In addition, the specific value proposition of subsidized tuition performed considerably better than the more general framing of the challenging or humanitarian nature of a career in the CAF. This finding suggests that more specific value propositions are one way of increasing engagement with social media campaigns.
The results of the congruence test suggest that congruent image-message combinations are better than having a neutral image with a given message. On English Facebook, a statistically significant positive effect of congruence was observed for both the challenge (Ads 4, 6) and humanitarian ads (Ads 5, 7). However, results from other platforms and languages suggest that congruence is not the only thing that matters.
The congruent humanitarian picture (Ad 5) in particular resulted in low levels of engagement, despite the fact that it was conceptually in line with the message in the text. Pre-testing images for use in future campaigns is a way of mitigating this issue.
Challenge Versus Humanitarian
While previous research has found that the use of words like ‘challenge’ and ‘exciting’ is likely to result in lower engagement from women with job ads, the results of this study suggest that this finding may not be universal. Overall, the messaging that framed a career in the CAF as challenging and exciting performed considerably better than copy and creative that framed a CAF career as a way to help and protect others, which is language that has been traditionally thought to resonate with women.
Instagram and Facebook
The relative performance of ads was similar on Facebook and Instagram, with the exception of the tuition and basic training ads. The tuition ad drove engagement more effectively on Instagram, which may be reflective of Instagram's younger demographic and the fact that the campaign was conducted at the beginning of the academic year. For this campaign, advertising on Facebook was cheaper than advertising on Instagram, and advertising in French was cheaper than advertising in English.
In partnership with the Department of National Defence, the Impact and Innovation Unit conducted a social media marketing trial with the aim of identifying best practices in driving engagement with a career in the CAF among Canadian women aged 18 to 34. The ads were displayed over 2.5 million times on Facebook and Instagram, and in both English and French.
Overall, click through rates for all the ads on Instagram and Facebook suggest high levels of engagement with the CAF and differences in these rates support evidence-based insights for future campaigns. The best performing ads either provided a specific value proposition to potential recruits, challenged assumptions about a career in the Armed Forces, or made use of the CAF badge and regalia. The trial also found that conceptual congruence between images and text in social media advertising is associated with higher levels of engagement. There were minimal differences in engagement patterns in French and English. However, there were large and significant differences in engagement among provinces and territories. High engagement was measured in Central Canada and the Maritimes, while Northern Canada displayed relatively low levels of engagement.
Applying experimental techniques to Government of Canada advertising campaigns is an effective way of improving marketing outcomes. Designing campaigns with built-in comparison cases and comparing outcome measures between ads in order to create an evidence base of what works would allow the government to improve value by making advertising more effective in achieving its goals. Future testing in the area of CAF recruitment could involve other social media platforms, like Twitter and LinkedIn, as well as applying experimentation techniques to advertising beyond social media. Social media advertising is one tool to increase engagement among Canadian women with a career in the Armed Forces, and is being used in conjunction with many other approaches to achieve the ultimate goal of recruiting more women into the organization.
The IIU would like to thank our colleagues at the Department of National Defence and the Canadian Armed Forces for making this trial possible. This trial, and the IIU’s previous report, is part of a larger ongoing effort to achieve greater gender equality in the CAF that the IIU is proud to support.
Glossary of Terms
- Behavioural Insights (BI)
- A multidisciplinary approach that uses the principles and tools of behavioural economics, psychology, and social marketing to influence behavioural change in a positive way.
- Clickthrough Rate (CTR)
- CTR is calculated by dividing the number of clicks each ad receives by the number of impressions.
- Unique Clickthrough Rate (UCTR)
- UCTR is calculated by dividing the number of unique users that click on an ad by the number of unique users that are exposed to the ad.
- Impressions are the total number of times an ad is displayed.
- Reach is the total number of unique users exposed, at least once, to an ad.
|Ad||Unique Clicks||Reach||UCTR||95% Confidence Interval||Unique Clicks||Reach||UCTR||95% Confidence Interval|
|Basic Training||3819||124469||3.07%||[2.97%; 3.16%]||1190||40762||2.92%||[2.57%; 3.08%]|
|Badge||4328||138346||3.13%||[3.03%; 3.22%]||1068||40923||2.61%||[2.70%; 3.02%]|
|Tuition||2857||128693||2.22%||[2.13%; 2.30%]||1180||40282||2.93%||[2.76%; 3.09%]|
|Challenge||2601||127983||2.03%||[1.95%; 2.10%]||866||39793||2.18%||[2.03%; 2.31%]|
|Humanitarian||1897||94517||2.01%||[1.91%; 2.09%]||547||31034||1.76%||[1.61%; 1.90%]|
|Challenge - Neutral||2140||131027||1.63%||[1.56%; 1.70%]||665||33574||1.98%||[1.83%; 2.12%]|
|Humanitarian - Neutral||2092||125102||1.67%||[1.60%; 1.70%]||751||39040||1.92%||[1.78%; 2.05%]|
|Ad||Unique Clicks||Reach||UCTR||95% Confidence Interval||Unique Clicks||Reach||UCTR||95% Confidence Interval|
|Basic Training||992||134311||0.74%||[0.62%; 0.78%]||203||32972||0.62%||[0.53%; 0.70%]|
|Badge||943||90640||1.04%||[0.97%; 1.10%]||172||19537||0.88%||[0.74%; 1.01%]|
|Tuition||958||114583||0.84%||[0.78%; 0.88%]||281||34461||0.82%||[0.70%; 0.91%]|
|Challenge||939||118740||0.79%||[0.74%; 0.84%]||196||31772||0.62%||[0.53%; 0.70%]|
|Humanitarian||763||131923||0.58%||[0.53%; 0.61%]||173||36252||0.48%||[0.40%; 0.54%]|
|Challenge - Neutral||856||110733||0.77%||[0.72%; 0.82%]||274||37257||0.74%||[0.60%; 0.82%]|
|Humanitarian - Neutral||834||105076||0.79%||[0.74%; 0.84%]||266||36199||0.73%||[0.64%; 0.82%]|
|Impressions||Clicks||CTR||95% Confidence Interval|
|British Columbia||287,842||3,873||1.35%||[1.30%; 1.38%]|
|New Brunswick||73,394||1,057||1.44%||[1.35%; 1.56%]|
|Newfoundland and Labrador||39,130||563||1.44%||[1.32%; 1.55%]|
|Northwest Territories||7,766||77||0.99%||[0.70%; 1.21%]|
|Nova Scotia||95,016||1,405||1.48%||[1.40%; 1.55%]|
|Prince Edward Island||14,165||182||1.28%||[1.09%; 1.47%]|
Report a problem or mistake on this page
- Date modified: