Risk and impacts

AI system harms and impacts pre-assessment

Understanding what harms and societal impacts an AI system may create is a crucial precondition for sustainable AI system development. The intensity of potential harms and impacts will vary significantly across different AI systems. Industrial AI systems with no direct effects on individuals or the environment will likely create limited harms and impacts. An AI system that makes irreversible decisions that affect the rights and obligations of individuals will have profound impacts and may generate significant harms. The AI System Owner should ensure that the organization conducts a harms and impact preassessment at the outset of AI system development. The pre-assessment outcomes should be documented. The pre-assessment should cover a wide range of potential harm and impact creation pathways. The pre-assessment should consider the harms the AI system may create and the impact it may have on its users, possible decision-making subjects, other affected parties, society at large, and the environment. AI system risk assessments often focus on the direct harms the systems may create. The harms and impacts pre-assessment should, however, aim at also identifying the potential system-level harms and impacts. These include the social action affordances the system may create or modify, its potential wealth and power distribution implications and effects on equality. The ethical advisory function should be involved in the pre-assessment of the harms and impacts if it is likely that the AI system will create a non-negligible risk of harm to individuals or the environment. The AI System Owner should ensure that harms and impacts pre-assessment is conducted if the design parameters of the AI system undergo fundamental changes.

Algorithm risk assessment

Algorithms constitute the backbones of AI systems. AI system performance driven by algorithm performance. Possible AI systems biases and unfair outcomes often emanate from algorithm design. If the organization has access to the algorithms in the AI systems, identifying the possible algorithm risks and assessing their gravity is key to sustainable AI system development and operation. Algorithm risk assessment should cover, to the extent possible, a wide range of algorithmrelated risk sources and causes. The Algorithm Owner should at least ensure that the organization 1) explores and documents how the algorithm affects the operations of the entire AI system 2) explores and documents the possible risk of biased and, in particular, discriminatory outcomes, and 3) explores and documents the risk of unfair outcomes and harms the algorithm may generate. As identifying biases and unfairness is often complex and contentious, the reviews should involve ethical and legal experts. Particularly if the organization intends to use the algorithm in a high-risk use case. In machine learning algorithms, testing algorithm outputs may be necessary for identifying biases and discriminatory outcomes. In addition, if a machine learning algorithm incorporates inferences made from training data, the risk assessment should review and assess 4) the risk of detecting non-existing patterns and correlations in the data, 5) the level of algorithmic scrutability and explainability.

AI system health, safety, and fundamental rights impact assessment

AI system impacts on the health and safety of humans will likely remain the most important concerns that organizations should address when developing and using AI. These impacts together with fundamental rights impacts will likely be the centerpieces of future regulatory instruments. If the AI system harms and impacts pre-assessment (T40) indicates that the AI systems will likely have non-negligible impacts on the health, safety, and fundamental rights of individuals, The AI System Owner should ensure that the organization undertakes and documents 6) a health impact review to identify the potential health impacts the AI system may have on the physical and psychological well-being of its users, subjects, and other affected parties, 7) a safety impact review to the identify the potential safety risks the AI system may impose on individuals' and organizations' tangible assets, and 8) a fundamental rights impact to identify the potential impacts that the AI system may have on the protection and realization of individuals' fundamental rights. The legal advisory function should be involved in the assessment.

AI system non-discrimination assurance

Many jurisdictions have non-discrimination laws and impose equal treatment requirements. Non-compliance with the non-discrimination laws and equal treatment requirements is incompatible with sustainable AI operations and may create significant legal and reputational risks. The AI System Owner should ensure that the organization conducts and documents a nondiscrimination assurance process to ensure that the AI system outputs are compliant with non-discrimination laws and equal treatment requirements. The legal advisory function should be involved in both designing and conducting the assurance. Ensuring that an AI system creates no discrimination risk is challenging due to the nature of the non-discrimination and equal treatment. For example, under the Finnish Equality Act, an AI system would directly discriminate against a person if the system threated a person less favorably than other based on their age, nationality, language, religion, belief, opinion, political activity, trade union activity, family relationships, state of health, disability, sexual orientation, or other personal characteristics. Less favorable treatment is discrimination even if based on an apparently neutral rule. Despite the prima facie ban, differential treatment can be justified if mandated by law or the treatment has an acceptable objective in terms of basic and human rights, and the measures to attain the aim are proportionate. Conducting a diligent non-discrimination assurance is particularly important for AI systems with algorithms developed using machine learning approaches. Machine learning approaches may result in inadvertent discrimination. As the algorithms are often unexplainable, detecting discriminatory bias may require the use of post-hoc analysis tools and real-world data AI system output testing.

AI system impact mitigation

Minimizing the AI system impacts is an important phase in sustainable AI system development and deployment. Minimizing the impacts requires first that the potential impacts are analyzed and appropriate measures are taken to eliminate or reduce adverse impacts where possible. Second, minimization requires that the organization mitigates the effects of the adverse impacts that it cannot eliminate or, third, manages their consequences. To arrive at an acceptable AI system impact, the AI System Owner should ensure that the organization 1) conducts a thorough analysis of potential impacts the system may have on its users, subjects or affected parties, or the environment, 2) develops and implements a risk minimization plan. The risk minimization plan should be designed to guarantee that the AI systems are acceptable and aligned with the organization's values and risk tolerance. The risk minimization plan should outline 1) appropriate measures to eliminate adverse impacts to the extent possible, 2) appropriate measures to reduce adverse impacts that cannot be eliminated, 3) appropriate measures to mitigate the effects of the residual adverse impacts, and 4) appropriate measures to manage the adverse impacts that cannot be mitigated.

AI system impact metrics design

Acceptable AI system impacts performance can only be ensured by deploying appropriate metrics to measure them. The AI System Owner should ensure that the organization defines and documents metrics for monitoring the AI system impacts during its operational use. The AI System Owner should ensure that the impact metrics align with the organization's values and risk tolerance.

AI system impact monitoring design

Monitoring AI impact is crucial to ensuring that its impacts remain acceptable. The monitoring must be systematic and metrics-based to achieve consistency over time. The AI System Owner should ensure that the organization defines, documents, and entrenches 5) workflows and technical interfaces to facilitate the monitoring of AI system impact, including for example 6) automated or manual production and reporting of impact metrics data, 7) alarm thresholds, and 8) workflows that allocate monitoring responsibilities. 9) workflows to address issues detected during health checks. The AI System Owner should ensure that the AI system performance monitoring process aligns with the organization's values and risk tolerance.

AI system impact monitoring

The AI System Owner should ensure that the organization implements the planned AI system impact monitoring processes. If the system version control processes disclose a breach of impact standards or indicate a value or risk tolerance misalignment, the AI System Owner should initiate appropriate measures to address the breach or regain alignment.

AI system impact health check

The AI System Owner should ensure that the organization performs the regular planned impact health checks. The reviews should assess whether the AI system impacts align with the organization's values and risk tolerance. If a review discloses a misalignment, the AI System Owner should initiate appropriate measures to regain alignment.