How do you achieve quality in a 2050 Calculator?

quality check

What checks should be done to ensure quality and how can quality considerations be built into the climate calculator process?

How do you achieve quality in a 2050 Calculator?

Quality assurance/quality control processes

“Quality means doing it right when no one is looking.”
- Henry Ford, American industrialist

Henry Ford’s quote encapsulates something very important about quality. It doesn’t provide a magic solution to achieve ‘quality’, but it does describe a state of mind that everyone working on a Calculator should have. When developing a Calculator, you shouldn’t have to wait for your work to be checked; rather you should know that you’re responsible for the quality of your own work.

What checks should you do? How should you do them? How should you record them? Quality considerations should be built into the thinking behind creating and updating Calculators as well as generating data input. Simple principles can be put into practice to help you to achieve high quality work. 

What do we mean by quality?

You’re likely to have heard the terms ‘quality assurance’ and ‘quality control’. They are often used interchangeably, but they don’t mean the same thing. The Intergovernmental Panel on Climate Change (IPCC) has adopted the following definitions to clarify activities, roles, and responsibilities:

Quality assurance (QA) is a planned system of review procedures conducted by personnel not directly involved in the creation or update of the Calculator. Reviews, preferably by independent third parties, are performed on a completed Calculator following the implementation of quality control (QC) procedures. QA activities are extremely useful because they provide an independent assessment of the functioning of the Calculator – and provide confidence that the Calculator is working as it should.

Quality control (QC) is a system of routine technical activities to assess and maintain the quality of the Calculator as it is being compiled. It is performed by personnel directly involved in creating and updating the Calculator. The QC system is designed to:

• Provide routine and consistent checks to ensure data integrity, correctness, and completeness
• Identify and address errors and omissions, and
• Document and archive material related to the creation and updates of the Calculator and record all QC activities.

General QC procedures include generic quality checks related to calculations, data processing, completeness and documentation. Documenting and archiving material related to the creation and update of a Calculator is critical to serve as a record of activities for future members of the Calculator team. If documentation and archiving are not undertaken, information can be easily lost and the institutional memory is jeopardised.

It’s important to have a simple quality system for the Calculator and it’s a good idea to nominate a QA/QC manager who understands what their responsibility means in practice.

Creating a simple QA/QC system

The following bullet points summarise the major elements of a QA/QC system that’s suitable for the Calculator project:

• A QA/QC plan which sets out quality objectives
• General QC procedures which can be applied to the whole Calculator
• Source-specific QC procedures
• QA and review procedures, and
• QA/QC system interaction with uncertainty analyses.

 What errors typically occur?

We’ve seen from our own experience that it’s too easy for small errors in Calculators to cause big problems. In the Calculators launched to date, we’ve also found a range of issues which can occur in the Excel file, such as:

• Simple typing mistakes
• Missing units when entering data
• Labelling not being updated (e.g., still refers to the UK Calculator version)
• Documentation that isn’t complete or easy to understand
• Formula errors, which are often subtle to spot.

 What checks should you do?

• Units – order of magnitude of errors are very common
• Implementation of methods – are the equations coded correctly?
• Input data – for scope, completeness, consistency with data in recent years, time series consistency and (where available) verification against other independent datasets
• Mass balances checks – does the fuel use in the Calculator agree with national statistical data?
• Sector expert and greenhouse gas expert checks
• Recalculation checks – compare the latest Calculator dataset against the previous data set. Are you sure about the reasons for the changes?
• Implied emission factors – calculate emissions from categories, for example, perhaps on a per capita basis and benchmark against other countries. Are your results reasonable?

 What does a QA/QC system look like?

The main elements of a QA/QC system are shown in the diagram below. The diagram could be used to explain the principles and practice of your Calculator QA/QC system to stakeholders involved in the 2050 Calculator, e.g., data suppliers, the 2050 Calculator team and users, including government policy makers.

Most quality problems are preventable with a simple and effective QA/QC system. Document and archive your findings.

Consider appointing a QA/QC manager

A QA/QC manager coordinates all QA/QC activities and manages the contributions from data suppliers, sector experts and independent experts, while also undertaking cross-cutting QA/QC activities. They maintain the QA/QC plan and co-ordinate action across the Calculator team to set quality objectives, communicate and implement QA/QC activities, and identify training and development needs (individual and systematic).

Further support

For more detail about QA/QC systems in principle and in practice, please refer to the IPCC 2019 Refinement, Volume 1.

The UK-funded 2050 Calculator programme supports creation of country Calculators and aims to provide ongoing support to existing Calculator teams – including QA/QC checks on existing Calculators.

You can visit the 2050 Calculator website and join us on LinkedIn for discussion.

• John Watterson, Greenhouse Gas Emissions Inventories Knowledge Leader, Ricardo