Governance Transparency

2011 NHS: Notes on the CVs release

If it weren’t so confusing, this would be titled ‘Notes on the note accompanying the 2011 NHS coefficients of variation (CVs) release’. We completely missed the October 3, 2013 release, along with the accompanying note posted a week later. Neither was mentioned in The Daily. We only received a heads-up from a friend (who’d also initially missed the release) late last week. If the media intentionally ignored it, good job. CVs on garbage data are, well, garbage. We’ve refrained from referencing the 2011 NHS data as such, going so far as to suggest a very limited-case use for it back in July. After reading the notes accompanying the CVs release, we’ve reconsidered.

A few interesting snippets from the notes:

In 2011, the  estimate of the CV includes both the sampling variance and the variance due to  household non-response. In both cases, variability was measured assuming that  the estimation methods had corrected the biases.

Statistics Canada assumed to be true that which it knows to be false. See the notes accompanying the 2011 NHS Immigration and Income releases. To be fair, it’s not possible to calculate a CV otherwise. StatsCan should have just acknowledged as much and not released this nonsense, the same way it didn’t publicly release the low-income data.

Furthermore, the target population in 2011 included only persons living in  private dwellings, while that of 2006 also included persons living in  non-institutional collective dwellings (approximately 1% of the population).

We completely missed this point previously, and it’s not a trivial one. The ‘approximately 1%’ is interesting given the supposed surge in boomer retirement – one of the main differences between the two concepts is seniors in independent living residences.

The rate of non-response to the 2006 Census long form  was 6%. By comparison, the rate of non-response to the 2011 NHS weighted by the  sampling weight (to take account of the follow-up subsample) is 23%.

To date, we have no idea if StatsCan calculated the ‘global non-response’ rate (GNR) for the 2011 NHS the same way it did for the 2006 long-form Census, other than this note on reweighting. As noted previously, the 2008 test of the 2011 long-form Census was also a voluntary survey – as the 2011 long-form non-Census turned out to be. The global response rate for the voluntary 2008 Census test was 45%. Given that StatsCan doubled its acceptable non-response rate and reportedly accepted near-blank questionnaires as acceptable responses, we have no confidence in the reported 76% response rate – and won’t until StatsCan discloses how it was calculated, along with the (pre edit-and-imputation) total and item non-response.

In 2006, only  geographic information and household size were used for the non-response adjustment, while in 2011, essentially all census variables as well as some administrative data  were used.

As far as we’re aware, this is the first time StatsCan has admitted to reweighting the 2011 NHS data using administrative data. In previous years the long-form Census data was compared to administrative data as a quality control test, but it was not adjusted using it. This point alone renders it garbage data. The whole point of carrying out the long-form Census was to capture the socio-demographic / economic breakdown of the Canadian population not otherwise captured by administrative data.

Guess we’ll have to continue waiting for the 2011 NHS technical reports (which apparently will not include a report on coverage) to see if StatsCan comes clean on how much effort it put into trying to sell this garbage data as useful.

Though we’d touched on it previously, it’s  worth once again noting that Chief Statistician Wayne Smith admonished critics of the 2011 NHS data for doing a  “disservice to Canadians.” As he put it, “it’s irresponsible to try and dissuade Canadians from using what is an extraordinary rich and powerful database… that is I think irresponsible.” The debacle that was the final 2011 NHS release, and now the slow trickle of details showing just how unreliable the 2011 NHS data is, speaks to who’s really been doing Canadians a disservice.

Leave a Reply

Your email address will not be published. Required fields are marked *