This is a report and set of recommendations from our research project for 2020, which forms part of the Council’s wider work programme.
Our research topic was trust in digital and data-driven technologies. This is a vital foundation for achieving an equitable, thriving digital future for Aotearoa. As this is such a big topic, we chose automated decision-making (ADM) as a case study to help focus our research.
Broadly speaking, automated decision- making refers to decision-making systems where parts of the process are carried out by computer algorithms.
ADM is everywhere and has big impacts on our lives. It informs what we see on our social media feeds and what price we are offered when booking a hotel online. It also informs big decisions, like whether people are approved for parole. ADM is not just about computers and automation.
ADM systems are built by people, who also decide what data to feed into them, and how to make decisions based on the outputs. Those people, in turn, are part of larger institutions and systems with reputations, power relationships and histories. All these elements influence how people see, and whether they trust, ADM systems.
There has been research on many aspects of trust and ADM, but most was produced in an academic context and does not usually include hearing directly from people who are most affected by ADM. We wanted to help fill this gap. Our team ran community workshops to hear from a diverse range of people, most of whom do not usually get a seat at the table when it comes to designing and implementing ADM systems. Their voices are central to this report.
We learned that many workshop participants were uncomfortable with aspects of the way ADM systems are used today, especially when personal data is used to make big decisions. We know there are examples of good practice around ADM already, but as far as we can tell, these do not yet address all the concerns and suggestions raised in the workshops.
We built off what we heard at the workshops and from a range of other experts to develop recommendations to increase people’s level of trust in the use of ADM. These are practical suggestions, but they are also bold. They ask people designing and implementing ADM systems to fundamentally rethink how things are done. We recognise that making big changes is not easy, nor is it fast. But it’s worth it, to build a better future for Aotearoa and all New Zealanders.
Summary of Findings:
We spoke to over 180 people throughout Aotearoa about different situations where ADM 1 has specific impacts on the lives of individuals, whānau and communities. We heard loud and clear that ADM and other decision-making systems should be built for — and with — the people who are impacted. This is essential for ensuring trusted and trustworthy systems.
When workshop participants talked about ADM, they focused on more than the technology itself. Instead, they talked about algorithms as being part of a much wider system that also included the way data is collected and used, the people and organisations that develop the systems, and the interventions resulting from decisions. Participants thought that ADM, with its ability to process data fast and at scale, is well suited to some situations. However, they were clear that it can be harmful in other situations and can intensify pre-existing bias and discrimination — especially when the decision has major impacts on the lives of individuals and their whānau.
Participants provided clear and concise suggestions of what would make them feel more comfortable in situations where ADM is used. They want systems that are built to meet the needs and reflect the values of the communities impacted. To achieve this, it is important to participants that people who have similar lived experience to them are involved in the development of decision-making systems and the interventions that result from them. Participants told us they would be more comfortable if there was transparency and clear communication about how the government uses ADM and how it is used to make decisions.
We took these clear and urgent suggestions and used them as a basis to develop a set of recommendations to the government.
We looked at work already underway and the barriers preventing systemic change, and gathered input from experts to inform our thinking.
To build towards a thriving and equitable digital future for Aotearoa, the Digital Council recommends that the government:
- RECOMMENDATION 1 – Fund community groups to lead their own data and ADM projects
- RECOMMENDATION 2 – Fund and support a public sector team to test and implement ADM best practice
- RECOMMENDATION 3 – Establish a public sector ADM hub
- RECOMMENDATION 4 – Work collaboratively to develop and implement private sector ADM rules and best practice
- RECOMMENDATION 5 – Build ADM systems from te ao Māori perspectives
- RECOMMENDATION 6 – Build a diverse digital workforce
- RECOMMENDATION 7 – Increase the digital skills and knowledge of public sector leaders