Type 4: Willful indifference

Example "A tech company called X-corp uses an automated “A/B testing” system that tries out new parameter values to expand its user base. Like in the Corrupt Mediator story, their system learns that they can get more users by causing their users to create problems for each other that only X-corp’s tools can solve, creating a powerful network effect that rapidly expands X-corp’s user base and earns X-corp a lot of money. Some concerned X-corp employees complain that they have inadequate checks in place to ensure their A/B development process is actually benefiting their users, but it never seems to be a convenient time to make major changes to the company’s already profitable strategy. One employee manages to instigate an audit from a external non-profit entity to assess the ethics of X-corp’s use of AI technology. However, X-corp’s A/B testing system is opaque and difficult to analyze, so no conclusive evidence of ethical infractions within the company can be identified. No regulations exist requiring X-corp’s A/B testing to be intelligible under an audit, and opponents of the audit argue that no technology currently exists that could make their highly complex A/B testing system intelligible to a human. No fault is found, and X-corp continues expanding and harming its user base."(p. 12)

"Black-box” machine learning techniques, such as end-to-end training of the learning systems, are so named because they produce AI systems whose operating principles are difficult or impossible for a human to decipher and understand in any reasonable amount of time. "(p. 13)