Source: The Importance of the Long Form Census to Canada, David A. Green and Kevin Milligan, Canadian Public Policy – Analyse de politiques, vol. xxxvi, no. 3 2010
This Thursday, Statistics Canada will be updating the Consumer Price Index (CPI) basket weights with the latest results from its 2013 Survey of Household Spending (SHS). Hopefully its corresponding CPI update ‘Ask an Expert’ Q&A, scheduled the following day, proves more informative than its 2011 National Household Survey (NHS) Q&A session.
While this CPI basket update will be the second undertaken since a major SHS redesign in 2010, little information about the survey has been made publicly available since then. Statscan stopped producing public use micro data as well as data quality reports for the SHS immediately following the redesign. The official reason: “There will be no public use microdata file (PUMF) for SHS 2010 due to resource constraints.”
That Statscan regularly cites ‘resource constraints’ even as it contends they haven’t affected the agency is one thing. That it regularly does so to justify withholding data from the public is a concern. Other national statistical agencies likewise facing resource constraints, including those in the US and the UK, seem loathe to target public data dissemination. Not only does doing so inconvenience data users, it undermines public confidence.
Since the release deals with the specific use of the SHS to weight the CPI basket, the main question is whether / how well the survey results are representative of expenditure patterns of different households across Canada. The two common sources for potential problems are sampling error (whether the selected sample households accurately represent the target population) and non-sampling error (like under-coverage and non-response).
The 2013 SHS was based on the Labour Force Survey (LFS) sampling frame in place at the time, which was based on the 2001 census. In addition to accounting for geography (its ‘primary sampling unit’), the LFS sampling frame includes ‘special strata’ to better account for aboriginal, immigrant and high-income households (and yes, that data is, or rather, was, derived from the long-form census – more on that in an upcoming post).
Assuming the LFS sampling method provides a sufficiently representative cross-section of non-institutionalised Canadian households (it doesn’t, but more on that another time), that leaves potential sampling errors as a possible source of bias in the SHS.
For a selected household to be part of the survey, it has to be living in the residence selected in the sample at the time the survey is conducted, with at least one household member able to respond on its behalf available to do so (coverage). That member would also have to agree to take the survey (response) as the SHS is voluntary. Statscan also has to find the survey response ‘acceptable’.
The SHS is conducted in two parts. The interview, administered to all surveyed households, collects information on broader expenditure as well as income components. To make it more like the US Consumer Expenditure Survey (CEX), in 2010 Statscan introduced a diary, distributed to half of surveyed households, which it uses to collect more detailed expenditures (and to check against the interview data for verification).
The response rates for the 2013 SHS survey components were 67% and 46%, respectively. For comparison, the response rates for the 2013 CEX survey components were 67% and 61%, respectively.
Overall response rates are less informative than response rates by different household characteristics, since different types of households living in different parts of the country have different consumption patterns, which can have a significant impact on CPI basket item weights.
While Statscan provides some information on geographic disparities – for example, only five hundred usable diaries were received from Ontario, its nearly 5 million households comprising the most populous and diverse province in the country.
Wealth/income inequality in Canada sees a disproportionately greater share of income, along with a disproportionately even greater share of discretionary income, concentrated among fewer households. As such, not having a representative share of high-income households in the SHS could significantly impact the survey results, and ultimately the CPI basket weights.
Statscan makes a point of emphasising the priority given to sampling high-income households: “The high-income household strata are allocated a larger share of the sample than the other strata, where an allocation proportional to stratum size is used.” However, that’s the first and last mention of high-income households in the 2013 SHS Guide. It’s worth noting the LFS sample frame defined ‘high-income’ as total household income over $125,000 based on the 2001 Census.
Statscan used to publish somewhat more detailed data quality indicators. From its 2009 SHS data quality report: “This report… covers the usual quality indicators that generally help users interpret data, such as coefficients of variation, response and non-response rates, slippage rates and imputation rates.” The 2009 report included non-response and under-coverage rates for high-income households (for the interview; the diary was rolled out in 2010).
Under-representation of high-income households (as well as their respective incomes, which exclude capital gains) in the national household spending survey is not a trivial / trifling matter. In fact, economists from across US federal statistical agencies recently published a report seeking to address this very issue with the US household spending survey. Is the Consumer Expenditure Survey Representative by Income? (NBER, 2013) notes: “Given the plutocratic nature of the CPI, the relationship of income and spending on different types of categories suggests that under-representation of high income families in the CE could be biasing the CPI.”
The solution seems simple enough: Just make the SHS mandatory. Unfortunately, there’s a long history that precludes such a possibility. The predecessor of the SHS, the Survey of Family Expenditure (FAMEX) was once a mandatory survey, before over-zealousness and questionable competence on Statscan’s part killed it. (A valuable lesson for the ongoing National Household Survey / long-form census debate, that’s obviously not been learned.)
There are far more questions than there are answers regarding the SHS at this point. As it stands, it’s not too far a stretch to say that using the SHS to weight the CPI basket could be biasing the Canadian index. Unfortunately, questionable Statscan decisions of the past have all but assured there’s no easy fix to the problem going forward.