Risks from product functionality issues

"Risks may arise where general- purpose AI models and systems fail to comply with general tenets of product safety and product functionality. As with many products, risks from general- purpose AI - based products occur because of misunderstandings of functionality and inadequate guidance for appropriate and safe use. In that respect, general- purpose AI- based products may be no different (430)."(p. 47)

"Impossible tasks arise from instances of an attempt to accomplish goals with a general- purpose AI system that goes beyond the general- purpose AI system’s capability. It can be hard to say definitively what constitutes an impossible task in a modern setting. Historically, large language models have not been able to consider events or developments that occurred after the end of their training. However, enabling AI products to retrieve information from databases has improved their ability to consider what happened after their training - although models still perform worse on tests that require novel information (431). Another potentially impossible task may be tasks requiring data that is inherently inaccessible -- such as information that does not exist in the format of computable media, or data not available for training due to legal or security reasons."(p. 47)

"Impossible tasks pose risks because often, salient types of failure-- including many of the engineering failures, post- deployment failures and communication failures (see Table 1) -- might be the by- product of mismeasurements, misapprehensions or miscommunication around what a model can do, and the misinformed deployments that result. For instance, the GPT- 4 model achieved results of “passing a simulated bar exam with a score around the top 10% of test takers” and being in the 88th percentile of LSAT test takers (2*). Confidence in this result even led some lawyers to adopt the technology for their professional use (432). Under different circumstances, such as changes to test- taking settings or when comparing to first- time bar examinees who passed the exam, the model achieved substantially lower percentile results (433). Those who were attempting to make use of the model in actual legal practice encountered these inadequacies, facing severe professional consequences for the errors produced by these models (i.e. inaccurate legal citations, inappropriate format and phrasing, etc.) (434). Similar misapprehensions regarding model performance are thought to apply in the medical context (435), where real world use and re- evaluations reveal complexity to the claims of these models containing reliable clinical knowledge (436) or passing medical tests such as the MCAT (2*) or USMLE (437). More generally, some deployed large language models struggle under some linguistic circumstances: They might, for instance, have trouble navigating negations and consequently fail to distinguish between advising for and against a course of action – though some research suggests these issues are addressed by general capability gains (438, 439)."(p. 48)

"Some shortcomings are only revealed after deployment. Although many thorough evaluations have examined large language model use for code generation (440*), including in relevant real- world tasks (441), instances of real- world deployment of large language models for coding suggest that the use of these models could lead to the potential introduction of critical overlooked bugs (442), as well as confusing or misleading edits (443) that could be especially impactful when guiding engineering programmers, particularly in applications that automate parts of the workflow (444)."(p. 48)

"In general, for many machine learning based products, it can be unclear exactly which context of deployment is well represented in the data and suitable for the model. However, more general purpose AI tools specifically are more difficult to vet for deployment readiness than lower- capability or narrower AI systems: With General Purpose AI, it can be difficult to clearly define and restrict potential use cases that may not be suitable or may be premature, although substantial progress on restricting use cases is feasible."(p. 49)