Design

"Data preprocessing choices: ML systems often preprocess the raw input before passing them into their modeling components for inference. Examples include tokenization, image transformation, and data imputation and normalization. Additionally, data from multiple sources and modalities (image, text, metadata, etc) may be combined and transformed in ETL (extract, transform, load) pipelines before being ingested by the model. The choices made here will have consequences for the training and operation of the ML model. For example, filtering words based on a predefined list, as was done for Copilot. Such simplistic filtering does not account for the sociolinguistic nuances of slurs and offensive words, and could unintentionally marginalize the very communities it was intended to protect."(p. 11)

"Modeling choices: The act of operationalizing an abstract construct as a measurable quantity necessitates making some assumptions about how the construct manifests in the real world. Jacobs and Wallach show how the measurement process introduces errors even when applied to tangible, seemingly straightforward constructs such as height. A mismatch between the abstract construct and measured quantity can lead to poor predictive performance, while confusing the measured quantity for the abstract construct can have unintended, long-term societal consequences. In contrast to recent end-to-end approaches for processing unstructured data (e.g., image, text, audio), ML systems that operate on tabular data often make use of hand-engineered features. The task of feature selection then rests on the developer. Possible risks here include: 1) Training the ML component on spurious features; 2) Using demographic attributes (e.g., race, religion, gender, sexuality) or proxy attributes (e.g., postal code, first or last name, mother tongue) for prediction. The former could result in poor generalization or robustness, the latter, entrenching discrimination against historically marginalized demographics. For example, the automated essay grading system used in the GRE was shown to favor longer words and essays over content relevance, unintentionally overscoring memorized text. Other automated grading systems have proven to be open to exploitation by both students and NLP researchers."(p. 11)

"Specificity of operational scope: Designs are often created based on requirements and specifications. Consequently, failing to accurately specify the requirements and operational scope of the system increases the risk of encountering phenomena it was not designed to handle. This risk factor is likely to be most significant for ML systems that are high stakes or cannot be easily updated post-deployment."(p. 11)

"Design and development team: Although software libraries such as PyTorch and transformers are increasing the accessibility of machine learning, a technical understanding of ML techniques and their corresponding strengths and weaknesses is often necessary for choosing the right modeling technique and mitigating its flaws. Similarly, good system design requires engineers with relevant experience. A team with the relevant technical expertise may be able to identify gaps in the design requirements and help to improve them. Conversely, the lack of either increases the risk of an ML system failing post-deployment or having some unforeseen effects on the affected community. There have been calls for mandatory certification of engineers to ensure a minimum level of competency and ethical training, though they are largely voluntary. Additionally, the diversity of a team (in terms of demographics) will affect its ability to identify design decisions that may disproportionately impact different demographics, such as using proxy attributes in modeling or training an international chatbot only on White American English."(p. 11)

"Stakeholder and expert involvement: Since the development team is unlikely to be able to identify all potential negative consequences, other experts (e.g., human rights experts, ethicists, user researchers) and affected stakeholders should be consulted during the design process. This involvement helps to mitigate the team’s blind spots and identify unintended consequences of its design choices, allowing them to be addressed before anyone is harmed. In some cases of participatory machine learning, affected stakeholders can directly influence the system’s design as volunteers."(p. 11)