Methodology — Benchmark Canada

Methodology

How a metric gets chosen, how its peer set is picked, where the figures come from, and what a verdict does and does not claim.

Every claim on this site is a comparison, and every comparison rests on choices: which countries, which measure, which year, which source. This page sets those choices out so that readers can check them and challenge the specific ones they disagree with.

For why the project exists and how the five pillars fit together, see about.

How a metric is selected

Before a metric is added to the site, it has to pass five tests:

It measures what the country achieves. Announcements, budgets, and stated intentions do not count. Spending qualifies only when the spending is itself the commitment, as with defence expenditure against the NATO target. It is never used as a stand-in for the outcome it was meant to buy.
There is a like-for-like comparison. Peer countries publish the same measure on a comparable basis, or Canada publishes a standard of its own that it can be held to. If neither exists, we do not build the metric, however important the subject.
It comes from a primary source. We prefer series published on a regular schedule, so the page can be refreshed as new data comes out. Some sources publish less often, and those pages are updated when the next edition is released.
It measures the thing itself. A proxy that answers a different question from the one being asked is rejected. Seizure counts, for example, show how active enforcement is. They cannot tell you how large the production problem is. Where both questions matter, each gets its own metric.
It can be broken down. A composite index folds someone else's weighting choices into a single number, so we use the underlying series wherever we can get it. When an index is the only measure available, the page identifies which subcomponents drive Canada's score instead of reporting the headline alone.

Related measures get separate pages. Where two measures of the same subject move differently (a level and its rate of change, or the assessment stage and the licensing stage), each has its own page and the two link to each other. Averaging them into one figure would hide the disagreement that makes the pair worth reading.

Choosing the peer set

The peer set is chosen on the merits of the comparison and before the result is known, and it is stated on the page. The default is the G7: countries of comparable income and institutional type, against which Canada already measures itself in ordinary public debate.

There are two exceptions, and the page says when either one applies.

A wider or different comparator group, where the G7 series does not exist for all seven countries, or where the G7 is the wrong comparison. A measure that turns on resource endowment, federal structure, or population density may be better read against a group selected on that characteristic. The page names the group and the reason for it.
No international comparison at all, where no other country publishes on a basis that can honestly be set beside Canada's. In that case the benchmark is Canada's own published service standard: the target the state set for itself, and how often it meets it. Passport processing times and veterans' disability claim backlogs are measured this way. We would rather hold Canada to its own commitments than build a false international ranking out of mismatched definitions.

Like for like. A cross-country comparison must hold constant everything except the country: the same profession, the same stage of the same process, measured in the same units. A six-week turnaround at one country's nursing regulator set beside a fifteen-week one at another country's engineering body compares two professions, not two countries. Where the like-for-like series does not exist, the page compares within a single profession or process and states what it therefore cannot conclude.

How to read a verdict

Every metric carries one of three verdicts, describing where Canada sits within that metric's peer set, on that measure, at the vintage stated on the page.

Strong

Canada ranks in the top third of its peer set.

Watch

Canada ranks in the middle third. No crisis, but no advantage either, and often a direction worth watching.

Weak

Canada ranks in the bottom third of its peer set.

The verdict follows Canada's rank in the peer set, counted from the desirable end of the measure (the lowest debt, the shortest wait, the most capacity). With N members including Canada, ranks up to one third of N are Strong, ranks beyond two thirds of N are Weak, and the ranks between are Watch. In the G7 that means ranks 1 and 2 are Strong, 3 and 4 are Watch, and 5 to 7 are Weak. Every metric page ends with a Verdict section that shows the peer set, Canada's position, the direction of travel, and the working.

Some measures have no honest international comparison. For these, Canada is judged against its own past record, from the baseline the page states. The verdict is Strong if the latest reading is better than that baseline and not getting worse, Weak if it is worse and not improving, and Watch if the record is mixed. Where a government has published a service standard, the page reports performance against it alongside.

A few pages cannot be ranked cleanly, because the peers are compared on design rather than on a single number, or because the page's measures point in different directions. There the Verdict section says which evidence the verdict follows and why. Where a measure's level and its direction disagree (a strong position eroding quickly, or a weak one improving steadily), the page says which of the two the verdict reflects. The verdict sums up the page in one word. The reasoning is on the page itself.

A verdict is not a grade for any government, because most of these measures move over decades and across administrations. Verdicts also cannot be compared across metrics. Each metric has its own peer set, so Weak on one page and Weak on another are not necessarily the same distance from the front, and the metrics should not be added into a composite or ranked against one another.

Pillar-level verdicts on the metrics overview are computed from the metrics currently in view, so filtering the dashboard recomputes them rather than leaving a stale headline in place.

Where the figures come from

Sources are used in order of preference, and the page says which level it is drawing on.

The primary series, from the agency that produces it. Statistics Canada, the Bank of Canada, the IMF, the OECD, the World Bank, departmental performance reports and departmental plans, and the equivalent national agencies in peer countries.
The underlying records, assembled here. Where no published series exists but the case-level or administrative records do, the series is constructed from those records and the construction is described on the page, so that someone else could rebuild it.
A named institutional source, cited and flagged. Where neither of the above is obtainable (a research institute, an industry association, a private data holder), the holder is named, the page says the figure depends on them, and the limitation travels with the number.

Not used: figures taken from news reports, values read off someone else's chart, aggregator sites and encyclopaedias, and any number that cannot be traced back to a body prepared to stand behind it. Every chart on every metric page carries its source, the specific table or series identifier, and the vintage of the data.

Sources and licences

Most of the data here is published openly and carries its own terms of reuse. Those terms are set out below, in the words the publisher uses. Every metric page also carries a short licence line naming the licences that page relies on.

Openly published data is used under the terms its publisher sets, with the source cited on the page that carries the figure. No permission was sought, because none is required. No licence, citation, or acknowledgement anywhere on this site implies that the publisher endorses Benchmark Canada, its framing, or its verdicts. Several of the licences below say so explicitly, and it holds for all of them.

Statistics Canada

Used under the Statistics Canada Open Licence. Almost everything here is a re-basing, per-capita conversion, ratio, or chart built on a published table rather than a verbatim reproduction, so the adapted-data acknowledgement applies: “Adapted from Statistics Canada, [name of product], [reference date]. This does not constitute an endorsement by Statistics Canada of this product.” Each page names the specific table, survey, or publication it draws on.

Federal departments and agencies

Departmental results reports, service standards, and agency data (CMHC, Natural Resources Canada, IRCC, Veterans Affairs Canada, the Treasury Board Secretariat, the Canada Revenue Agency, CIPO, CBSA, and others) are used under the Open Government Licence – Canada, which requires the statement “Contains information licensed under the Open Government Licence – Canada” and prohibits any suggestion of official status or endorsement. Statistics Canada is not covered by this licence. It has its own, set out above.

Bank of Canada

Under the Bank's terms of use, content may be freely used provided the Bank is attributed as the source and any changes are indicated, and not in any way suggesting the Bank endorses the use. Where a Bank series has been indexed, re-based, or otherwise recalculated here, the chart note says so. Bank note images, the Bank's logo, and third-party content on its site are excluded and are not used here.

OECD

Under the OECD Terms and Conditions, OECD data may be extracted, adapted, and redistributed with appropriate credit, cited as OECD (year), dataset name, data source, DOI or URL (accessed on date). Charts here are built from OECD data rather than adapted from OECD written content, so the adaptation disclaimer attaching to OECD publications does not apply. Where an OECD publication is quoted or its analysis restated, the page attributes it directly. The OECD notes that some data in its holdings is owned by third parties.

World Bank

World Development Indicators and other World Bank datasets are used under CC BY 4.0, which requires the dataset be credited to the World Bank and prohibits representing or implying that the World Bank has sponsored, approved, or endorsed the use.

International Monetary Fund

Published IMF statistical data (principally the World Economic Outlook database) is used with the attribution the IMF specifies, “Source: International Monetary Fund”, per the IMF's terms. Where a figure is an IMF projection rather than an actual, the page says so.

Eurostat

Reused under the European Commission's reuse decision of 12 December 2011, as set out in the Eurostat copyright notice, which authorises reuse for commercial and non-commercial purposes provided the source is acknowledged and any changes are indicated.

NATO

Defence-expenditure figures come from NATO's annual press release. NATO's content rules permit republication in objective and balanced content provided all content taken from NATO is clearly credited or sourced to NATO, and prohibit selling NATO material or using it for advertising or promotional purposes. The NATO logo, name, and emblems are not used here.

Other sources with published terms

Basel Institute on Governance — Basel AML Index public edition, used under CC BY-NC-ND 4.0 as set out on the Basel AML Index downloads page, and credited to the Basel AML Index / Basel Institute on Governance with a link to its site.
Canadian Institute for Health Information (CIHI) — used under CIHI's terms of use, which allow CIHI material to be freely used and reproduced for non-commercial research provided CIHI is credited as the source.
European Central Bank — euro reference exchange rates, credited to the ECB.
International Energy Agency — used under the Creative Commons licence attached to the source publication, which the page names.
Tax Justice Network — Financial Secrecy Index, used under CC BY-NC-SA 4.0 as set out in the Financial Secrecy Index copyright licence, and credited to the Tax Justice Network.
U.S. Geological Survey — USGS-authored material is in the U.S. public domain; credited to the U.S. Geological Survey.

Sources used by express permission

Two sources are not open data and are used because their holder agreed to it. Fraser Institute figures, tables, and tabulations are reproduced with the permission of the Fraser Institute, and the page carrying them says so. Leaders Fund has granted permission for its Specter-sourced data to be used as a metric here. In both cases the acknowledgement is not an endorsement: neither organisation has been asked to approve the framing or the verdicts.

Charts

Missing data is left missing. A gap in a series is drawn as a gap. We do not interpolate, carry a value forward from the previous period, or put a placeholder where the source published nothing.
Every chart states what it plots and in what units, and carries a source line with the citation and the vintage.
Projections are labelled as projections. Where a value is a forecast or an agency outlook rather than an observed figure, the chart and the surrounding text both say so.
Country labelling is consistent across the site, so the same country reads the same way on every page it appears on.

Vintage, updates, and revisions

Figures carry the vintage of the data, not the date of the page. A metric published this month may show a figure from two years ago because that is the most recent complete period its source has released. When that happens, the page says so rather than substituting a more current but less comparable figure.

Metrics are updated when their sources publish, not on a fixed calendar. A page is not refreshed with a partial period presented as a complete one. Where a source revises history, the revised series is adopted. If a revision changes a finding, the page notes the change.

Corrections

If you believe a figure here is wrong, please say so. Corrections will be published.

A challenge to a specific figure will be answered specifically, with the source it came from, the judgement calls made in constructing it, and any revisions since publication. A number that cannot be defended in that detail should not be on the site in the first place.

How metrics are tagged

Besides its pillar, each metric carries two sets of tags, used by the filters on the metrics overview.

Cross-cutting lenses group metrics by the question a reader arrives with rather than by the structure of the site. Seven are in use: cost of living; economic competitiveness; energy and resources; infrastructure delivery; innovation and technology; national security and sovereignty; and social fabric. Most metrics carry one or two, and none carries more than three. A lens is applied only where the metric tells that reader something useful.
House of Commons standing committees map each metric to the committee, or two committees, whose mandate would actually study it. Thirteen committees currently appear in the filter, and the list is self-curating: a committee is added only once a metric is mapped to it, so the dropdown never offers an empty view.

What this method cannot tell you

Position, not cause. A benchmark establishes where Canada sits. It does not establish why, and the causal claim is usually the contested one. The pages set out what the evidence supports and stop there.
Comparability is imperfect even at its best. Official series defined similarly are still collected differently, and national statistical practice varies. Where that materially affects a comparison, the page says so. The ordinary caution that applies to any cross-country comparison applies here too.
Coverage is a judgement. The 44 metrics currently published do not cover the whole of national performance. They reflect a judgement about what matters and what can be sourced defensibly, and more will be added.
A benchmark is a starting point. It tells you which questions are worth asking, and it makes an unsupported claim harder to get away with. Answering those questions is separate work.

A project of the Macdonald-Laurier Institute Led by Senior Fellow Kevin Vuong

The five pillars

The site