New York City Local Law 144 of 2021 (also known as the Bias Audit Law, Bias Audit Mandate, and AEDT Bias Audit Law) requires annual independent and impartial bias audits of automated employment decision tools (AEDTs).
However, compliance with Local Law 144 is not as simple as solely commissioning an audit from an independent bias auditor. The law also imposes notification and transparency requirements for employers and employment agencies using automated employment decision tools.
Given that the law targets AEDTs used to evaluate candidates for employment or employees for promotion that reside in NYC, the law has far-reaching implications, where any company with either employees or prospective employees residing within the city that uses an AEDT must comply with Local Law 144. As such, companies around the world are within the scope of the law.
Here, we take a deep dive into the NYC Bias Audit Law and explores the essential components you need to know for Local Law 144 compliance.
New York City Local Law 144 was first introduced by the New York City Council on 27 February 2020 before amendments were approved by the Council on 10 November 2021 and the law was enacted on 11 December 2021.
The law’s target is to prevent discrimination against protected characteristics caused by the use of automated tools. It aims to hold employers and employment agencies accountable for the decisions made by AEDTs, as well as provide greater transparency around the use of the tools.
To support the enforcement of the law, New York City’s Department of Consumer and Worker Protection published proposed rules on 19 September 2022 to clarify who qualifies as an independent auditor, in addition to details of how to conduct bias audits, and guidance around what must be included in the summary of the bias audit.
Following this, a public hearing was held on 4 November, where stakeholders could provide feedback on the proposed regulation. Based on the feedback received during the hearing, the DCWP updated their proposed rules at the end of December 2022 and pushed back the enforcement date from 1 January 2023 to 15 April 2023.
A second public hearing was held on 23 January 2023 on the second version of the proposed rules before the enforcement date was once again pushed back to 5 July 2023 for the final time to allow the DCWP to address the high volume of comments that they had received on the proposed rules. This announcement came with the publication of the final version of the rules for the enforcement of Local Law 144 in April 2023.
According to the final version of the DCWP’s rules, an automated employment decision tool (AEDT) is a computation process that is derived from machine learning, statistical modelling, data analytics, or artificial intelligence. These processes are used to issue a simplified output, such as a score, classification or recommendation, which is used to substantially assist or replace discretionary decision making.
Specifically, machine learning, statistical modelling, data analytics, or artificial intelligence are a group of mathematical, computer-based techniques used to generate a prediction or classification. For these techniques, a computer is used at least in part to identify the inputs, relative importance of such inputs, and other model parameters to improve the accuracy of the model.
To be deemed to substantially assist or replace discretionary decision-making, an AEDT should either be the sole determinant in decision-making, the primary factor, or have the capability to override decisions based on other factors, which may include human judgment.
Importantly, the output of tools used to translate or transcribe text would not be within the scope of Local Law 144.
The New York City Bias Audit Law has three key requirements for organisations using AEDTs:
A bias audit is an impartial evaluation by an auditor that assesses whether an AEDT results in disparate impact against individuals based on race/ethnicity and/or sex/gender, where disparate impact refers to disproportionately negative outcomes for a particular group.
The protected categories that must be included in the audit are male, female, and optionally other for gender, and Hispanic or Latino, White, Black or African American, Native Hawaiian or Pacific Islander, Asian, Native American or Alaska Native, and two or more races for race/ethnicity.
The audit must be carried out using specified metrics depending on whether the AEDT is a regression system that results in a continuous score or classification system that has a binary output.
For classification systems, the metric is used to calculate an impact ratio that compares the selection rate for a category to the selection rate of the highest scoring category:
The scoring rate refers to the proportion of people in each group designated to the positive condition.
For regression systems, outcomes must first be binarised to calculate the scoring rate, where individuals are designated to pass/fail based on whether their score is above or below the median score for the dataset used to complete the audit. The impact ratio can then be calculated using the scoring rate in a similar way to with classification systems:
These impact ratios must be calculated based on standalone groups (e.g. male, female) and intersectional groups (e.g., black male, black female).
An independent bias auditor is someone – an individual, group, or company – that is capable of exercising objective and impartial judgement about a system to conduct an independent evaluation. An independent auditor is not anyone that is or was involved in using, developing, or distributing the AEDT, has an employment relationship with the employer/employment agency or vendor of the tool during the course of the bias audit, or has a financially direct or materially indirect interest in the employment agency, employer, or vendor of the AEDT during the bias audit.
The preferred type of data to conduct AEDT audits is historical data collected from the real-life use of the tool. However, where this data is insufficient to conduct a bias audit or the tool has not yet been used in practice, test data can be used instead, providing this is disclosed.
An employer or employment agency can also rely on a bias audit of an AEDT conducted using aggregated historical data that combines data from other employer or employment agencies that use the same tool if they also contributed to this aggregated data or if they have never used the tool themselves.
Auditors can exclude groups with a small sample size that represent less than 2% of the audit data from the analysis, but they are still required to calculate the scoring rate or selection rate of the group.
Once a bias audit has been conducted, employers or employment agencies using an AEDT must provide a bias audit summary on their website before using the tool.
This summary of results must include:
This summary must be updated annually with details of the most recent bias audit and kept online for 6 months after the tool is retired.
Candidates and employees being evaluated by an AEDT in New York City must be informed of this at least 10 working days before the tool is used. The notice must contain all of the following:
For candidates, this notice can be provided through the employment section of the website in a clear and conspicuous manner, in a job posting, or through mail or e-mail. For employees, notice can be given in a written policy or procedure, in a job posting, or via mail or e-mail.
New York City Local Law 144 is enforced by New York City’s Department of Consumer and Worker Protection (DCWP). Any suspected violations should be reported to the DCWP by contacting 331 or visiting the DCWP website and should include details on the complaints, job posting or position, the name and type of AEDT being used, and any notices given for the use of the AEDT.
Penalties for non-compliance with Local Law 144 start at up to $500 for the first default and each additional violation occurring on the same day and increase to up to $1500 for subsequent defaults.
It is important to note that obtaining a bias audit and providing notice for the use of an AEDT are separate requirements and are considered separate violations.
New York City Local Law 144 already appears to be having a snowball effect, with New Jersey proposing a similar law in Assembly Bill 4909, which seeks to require annual impartial disparate impact analyses of automated employment decision tools.
However, the New Jersey Bias Audit Law takes a slightly different approach, targeting vendors rather than employers or employment agencies. It would also prohibit the sale of AEDTs in New Jersey unless they have been audited for bias, where a bias audit should be included at no extra cost. In other words, the New Jersey Bias Audit Law targets developers of AEDTs, whereas the NYC Bias Audit Law targets deployers of AEDTs.
However, New Jersey is not the only state in the U.S. following in New York City’s footsteps. New York State has proposed two laws, A00567 and A07859, which are essentially Local Law 144 in two halves. Specifically, A00567 seeks to impose bias audit requirements while A07859 seeks to impose notification requirements for the use of AEDTs.
Similarly, in the insurance industry, Colorado Senate Bill 169 of 2021 was enacted to prohibit discrimination based on protected attributes. Recently, Colorado Department of Regulatory Agencies’ Division of Insurance proposed a framework for internal bias audits of algorithms and predictive models and algorithms used in life insurance.
With several bias audit laws already proposed or enacted in the U.S., this trend is likely to continue, meaning if an AEDT is not yet within the scope of Local Law 144, it is likely to fall under the scope of another similar law in the future.
Getting compliant early is a competitive necessity to promote trust and avoid legal liability. Compliance with Local Law 144 has five key steps:
In the event that the bias audit does flag exceptions for particular subgroups, indicating that the tool has the potential to result in biased outcomes, then an additional step should be taken to mitigate unjustified subgroup differences.
Holistic AI’s Bias Audit Solution is a one-stop-shop for NYC Bias Audit Law Compliance, including a fast and efficient bias audit, prepared and hosted Summary of Results page, education and knowledge exchange on the bias audit ecosystem, support with providing notifications, and custom mitigation recommendations.
Our holistic approach combines expertise in computer science, AI ethics, law, policy, and business psychology to understand the system, the context in which it is used, and the legislative and regulatory requirements.
Join global companies already trusting Holistic AI to get Bias Audit ready.
