Statistics Canada started sending out its 2016 Census letters this week. Shortly after taking office, the recently elected federal government once again made it mandatory for survey respondents to complete the long form questionnaire, presumably restoring the census.
Contrary to promises it made last year while still sitting in opposition, the current federal government did not make any changes to the Statistics Act, which appears to have been last amended in 2005. That means Statscan can, and, if history is anything to go by, will once again be threatening non-respondents with jail time. Effectively, the clock has been set back to May 2006, when the conditions were already in place for the eventual long form census cancellation.
2006 was the first census conducted using software and services contracted from giant US defense contractor Lockheed Martin. That year, a protest campaign against the federal government’s questionable, and extremely unpopular, decision to reward the contract to the US firm saw hundreds of thousands of Canadians either refuse to complete or otherwise spoil their census questionnaires.
Since that episode appears set to play itself out again, it’s worth reviewing why and how it happened, as well as the lessons obviously not learned.
Census contract background
Contrary to what Canadians were told back in 2010, when the federal government of the day decided to make response to the long form questionnaire voluntary, there in fact was a great deal of concern over security and privacy.
A quick Hansard search shows these concerns were expressed by Members of Parliament (MPs) at least as early as 2003, when it first came to light that Statscan was proposing to contract the US defense giant for the upcoming census.
Citizens had in fact contacted their federal MPs to voice their concerns, and those concerns were repeatedly expressed in the House of Commons (HoC).
Some MPs had even threatened to boycott the census over the contract decision, not only over security and privacy concerns but over the issue of ethics as well. One MP noted the US defense contractor produced landmines – problematic since a few years earlier Canada had played a key role in bringing about the United Nations Mine Ban Convention and Treaty.
In April 2004, a motion was submitted in the HoC asking the federal government to rescind the controversial census contract.
In May 2004, under the mistaken impression that the government was reconsidering its unpopular census contract decision, a prominent MP acknowledged and credited the protest movement for its effort in the HoC.
Later that year the government justified proceeding with the disputed census contract by claiming the 2006 census would be jeopardised if it changed course so far along in the process.
The typical government assurances regarding security and privacy had been given to assuage public concerns. However, those assurances quickly came under suspicion when the CBC reported that employees of the defense contractor were directly involved in census processing.
A ‘Count Me Out‘ boycott campaign, buoyed by public distrust and ethics concerns, encouraged citizens not to cooperate with Statscan (on the upcoming census, as well as future surveys). While the agency publicly downplayed the effect of the boycott, the incomplete and spoiled questionnaires did negatively impact the 2006 census results (released late-2007 to mid-2008).
Supposedly to avoid a replay of the unfortunate 2006 census episode, the federal government in 2010 made response to the long form questionnaire (renamed for 2011 the National Household Survey, or NHS) voluntary and promised that no ne would go to jail for refusing to complete it.
Prosecuting ‘Refuseniks’
In 2008, Statscan referred a number of 2006 census refuseniks to the Attorney General of Canada for prosecution.
Going by what was reported at the time, it seemed Statscan targeted average Canadians who had vocally supported the census boycott over the US defense contractor’s involvement. Cases involving middle-aged and elderly folks and students, predominantly from lower and middle income households, were widely reported.
However, vocal critics with political influence, predominantly from higher income households, who had publicly stated their refusal to comply didn’t appear to be targeted.
Using discretion
Which brings up a point worth discussing: The (ab)use of discretion.
As previously alluded to, the 2011 census fiasco in many ways mirrored a previous Statcan misstep, with the Survey of Family Expenditures (FAMEX) in the 1990’s. That survey, which at one point had been mandatory, was changed to a voluntary survey in the 1980’s. Response rates dropped off following the change, but still remained in the high-70 percent range. Statscan’s efforts to revert back to a mandatory survey involved threatening non-respondents with jail time.
The effort was ultimately abandoned after public backlash. While the agency had hoped to increase response rates by reverting to a mandatory FAMEX, the survey ended up being scrapped altogether and replaced with another, the Survey of Household Spending, that was also voluntary; the SHS has since produced response rates significantly lower than even the voluntary FAMEX had (less than half of those surveyed bother completing the food expenditure diary, and response rates for the questionnaire have been in the mid-60 percent range).
While response to the 2011 long form questionnaire was not mandatory, response to the 2011 census short form still was. Statscan still went after those who refused to complete the short form with threats of prosecution, including an 89-year old war veteran (elderly case previously mentioned). The judge in that case came up with an incomprehensible reason to acquit the elderly war veteran and openly questioned Statscan’s judgement for even considering going after such a person.
It would have been wiser – not only in terms of public relations, but data quality as well – to target non-respondents from higher income households: There are far fewer households at the top of the income distribution, so the absence of only a few of their responses would have a significantly greater impact on the income statistics than the absence of thousands elsewhere along the distribution. (While income data is available from administrative sources, one of the main purposes of the long form survey is contrasting income across demographic, social and other economic traits not otherwise readily acquired from administrative records).
It’s quite likely that the ‘fine and/or prison’ penalty was contemplated for the Statistics Act to address the potential problem of having higher income households avoid completing their questionnaires, opting instead to pay what would be a relatively negligible sum for the privilege.
Often it’s not the law that’s the problem, it’s the discretion used in its application.
Compromised co-operation
The 2011 long form census fiasco demonstrated an important fact often glanced over: One in three Canadian households could not be convinced to complete the census long form voluntarily, even with repeated ‘follow-up’/harassment from Statscan. That number rises to one in two – which was the non-response rate for the voluntary 2008 census test that preceded it – without the harassment.
That’s significant because it begs the question how accurately/completely those who are not inclined to complete the questionnaire tend to respond when coerced into doing so.
Notably, as an alternative to complete non-compliance, the ‘Count Me Out’ campaign counseled minimum cooperation. This would entail providing either arbitrary or minimal/incomplete responses to the census questions.
As reported back in 2011, Statscan had instructed enumerators to accept questionnaires with as little as 10 of 84 questions completed. The agency has never denied the report.
Most of the long form census questions (and some on the short form) are personal and can’t be deemed inaccurate even if the responses appear suspect.
Will forcing Canadians to respond this go-around, knowing that only half would voluntarily do so without harassment, really improve data quality?
It also begs two other questions: How much less reliable were the 2011 NHS data, given a quarter of the responses that were received had been obtained after ‘follow-up’/harassment of non-respondents? And likewise, how much less reliable were the 2006 census data, since it’s impossible to know how many census protesters chose the ‘minimum cooperation’ route to spoil their questionnaires?
To the point, forcing people to respond to personal questions with legal threats when meaningful responses require their willful participation and co-operation is a fundamentally flawed approach.
Add to that Statscan’s view that practically empty questionnaires obtained through legal coercion are sufficient, as incomplete/inconsistent responses can be ‘corrected’ through imputation, and you have a recipe for bad data.
Put another way, non-response is the known unknown; coerced response is the unknown unknown, the more problematic scenario.
International context
It’s also worth pointing out that the census boycott over the involvement of US defense contractor Lockheed Martin is not an isolated, quintessentially Canadian quirk. There’s a Count Me Out campaign in the United Kingdom over the same firm’s involvement in its census of population; the UK 2011 census protest story mirrors Canada’s, complete with prosecutions and a federal government promising to scrap the survey.
Besides the US, Canada and the UK are the only other national governments so far to have adopted the US defense contractors’ Census Systems. It seems every other major industrialised country has managed to conduct its census without it and avoid the unnecessary, easily foreseen public backlash.
Lessons not learned
In addition to the initial $40+ million dollar contract, Statscan has continued to pay the US defense contractor tens of millions of dollars for its continued work with the agency instead of investigating an alternative solution. (Then again, inertia is the hallmark of public administration – once something is done, rarely is it undone, irrespective of how ill-advised it may have initially been.)
Whether the data is as secure and privacy as well guarded as represented is impossible to tell. It’s not unheard of for government reassurances regarding security and privacy to turn out to be false (see CBC exposé referenced earlier). As previously discussed, Statscan proceeded to collect social insurance numbers during 2014 census testing (for the 2016 census) without express authorisation to do so and despite the Office of the Privacy Commissioner having expressly advised the agency against doing so. And Lockheed Martin’s own secure networks were hacked – ironically, at the same time the 2011 census was being conducted.
Beyond security and privacy, the general public rarely gets a chance to directly protest its displeasure with public officials for rewarding billions of dollars in contracts to large foreign defense contractors, often with little to no competition or public consultation. As noted earlier, the issue of ethics behind the Statscan census contract was debated in the HoC no less. The backlash from rewarding the controversial contract in the face of widespread public disapproval should have been readily foreseen.
Yes, low response rates do – in the case of the 2011 NHS and numerous other recent Statscan surveys, already did – result in less reliable data. But, as discussed, so does forcing people to respond against their will when co-operation is required to obtain accurate, meaningful survey responses. While non-response is easy enough to measure, its negative impact on survey data is not. Inaccurate responses on the other hand are neither easy to measure nor assess in terms of data quality.
When it’s known why a significant segment of the population is wary of responding to a survey, and where circumstances have already demonstrated that a significant proportion of that same population is not inclined to voluntarily respond to it, the reasonable solution would be to address the ‘why’ instead of resorting to coercion, especially when pursuing the latter entails even greater unknowns with respect to data quality.
As was the case in 2006, there will be hundreds of thousands of Canadians who will refuse to complete their 2016 census questionnaires voluntarily. Some will respond when subjected to legal coercion by Statscan; how meaningful those responses will be is impossible to say. Others may register their protest through ‘minimum co-operation’, leaving the agency to rely on ‘imputation’ to guesstimate and fill in the blanks. Either way, data quality will continue to be compromised, regardless of how much the headline response rate figure improves.
It’s tempting to end this off with a cliché, such as how it’s never too late to do the right thing. But given the confluence of federal politics, bureaucratic inertia and military-industrial interests involved here, good will and the public interest don’t seriously stand a chance.