On March 13, 2024, the EU Parliament approved the EU AI Act (adopted text), the details of which are the focus of this analysis. In my previous article, I provided an overview of the essential aspects of the adopted text. Now, I've dedicated myself to a detailed study of the regulation, which, despite its length and complexity, hinges on whether a system qualifies as AI. This article delves into the Act's definition of AI, uncovering a significant oversight: as currently defined, even ChatGPT might not qualify as AI. Clearly, all the comparably simpler algorithms and projects that the Act intends to regulate, would not qualify as well, making this a regulation that completely misses its objective.
At first, let's consider the key definition in Title I, Article 3, Clause 1:
‘AI system’ means a machine-based system designed to operate with varying levels of autonomy, that may exhibit adaptiveness after deployment and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments;
I will now elaborate on three key terms:
Inference
Autonomy
Adaptiveness
Inference
Most of this definition is rather clear. The traditional view of inferring machines is used. Whether you are coming from a statistical school of thought, or rather machine learning philosophy, we can all agree that the models that we build learn on a sample of data and generalise to a wider population. For instance, you are interested in the average height of a person in a country. You do not measure everyone living in this country, you only collect a representative sample and infer the parameter of the population. In other instances, this could be a bit tricky to conceptually imagine. Does the same principle hold if we are trying to estimate market movements? Yes. Just in this case, the sample is our historical data, and we are inferring what might happen in the future. The regulators have made an effort to be explicitly inclusive—or perhaps slightly pedagogical—by listing various applications of inference, such as generating predictions, content, recommendations, or decisions.
Autonomy
Until this point, the analysis has been relatively straightforward. However, the concept of "autonomy" introduces significant complexity. According to Recital 12:
AI systems are designed to operate with varying levels of autonomy, meaning that they have some degree of independence of actions from human involvement and of capabilities to operate without human intervention.
At first glance, this definition appears clear-cut: the regulation aims to oversee systems demonstrating autonomy. Consider an automated loan approval algorithm that independently decides on loan approvals; such a system is autonomous and potentially impacts human well-being significantly. Conversely, if the algorithm only suggests to a representative whether to approve a loan, its autonomy—and the associated risk—is considerably reduced, potentially diminishing the need for regulatory oversight.
The problem from my perspective lies in how broad the range between the two aforementioned extremes is, and how ambiguous this range is. For instance, let's say we are at the recommendation option (the latter). Imagine your algorithm is highly accurate. What if we would find out that only at the rarest of occasions, company representatives deviate from the recommendation of our loan approval algorithm? Over time, company representatives stop scrutinising its recommendations, essentially rubber-stamping the outcome. This wasn't intentional, but the system has inadvertently gained de-facto autonomy. Does it now fall under the AI Act? The view of autonomy is no longer about technical capability of the system, but how humans interact with the system. This creates uncertainty for developers and those deploying the algorithm. How would this criteria apply to the AI systems you are currently working with?
Adaptiveness
Adaptiveness is another crucial trait for a system to be considered AI under the EU AI Act, as detailed in Recital 12 of the adopted text:
The adaptiveness that an AI system could exhibit after deployment, refers to self-learning capabilities, allowing the system to change while in use.
This provision reveals a significant loophole in the AI Act's definition of AI. It effectively excludes many use cases and projects that are widely regarded as embodying AI, highlighting a disconnect between the Act and practical applications of AI technology. Let's delve into this issue further.
Imagine that you want to create a machine that plays chess. You can go one of two ways. Your first option is to explicitly program moves, decisions and board positions into the machine. This is cumbersome as this explicit programming would simply take too much effort. The other way, which has been an intriguing promise of AI for decades already, is that you will rely on self-learning. You start by collecting a dataset, referred to as training data. In our case, this would be hundreds, thousands or even more chess games which were played by humans and recorded. Secondly, you need an appropriate self-learning algorithm. The realms of AI, or its subsets of machine learning offer plenty of possibilities. Finally, you combine these two - you let the algorithm self-learn how to play chess on the training data you collected. As a result, there is no need to perform explicit programming - thanks to the self-learning capability, the system will be capable of playing chess.
At first glance, Recital 12's explanation of adaptiveness, emphasizing self-learning, seems to make a clear distinction between traditionally programmed machines and those we classify as AI. However, the critical phrase "allowing the system to change while in use" significantly narrows the scope of what is considered adaptive.
Our self-learning chess playing machine would not be regarded as adaptive. The machine (or system) was changing, but NOT changing while in use. The life of our chess playing machine is broken into two large phases - training and utilisation. Within the training phase, we start with the data (set of games from the past) and a raw algorithm. Within this phase, the algorithm changes as it observes the training data and learns from them how to play chess through imitation of human player moves. At a certain point, we are going to conclude the training phase. The self-learning, and changing of the algorithm stops as we now want to utilise the algorithm. This breaking point between the two phases is commonly referred to as “deployment”, also mentioned in the recital of the regulation.
Deployment can take on many forms. Let's say that for our needs we would place this trained machine on a server and publish it through a website. Visitors to our website can now test their chess skills and play against our machine. The machine is in use, but it is not changing anymore. It just showcases its learned capabilities, while playing chess against the visitors of our website. Hence, our chess playing machine
Now you might ask - but what if rules of the game change, or there are new trends among players. Wouldn't we want our machine to account for these? Yes, and this is fairly common (maybe not in chess). This step is referred to as “retraining” of the algorithm/system. We essentially go back to the drawing board. We would take the original data that the algorithm was trained on and enhance it with new samples, or change the data to reflect only the recent needs, such as chess games with newly introduced rules. Here, one might argue that the chess playing machine is “in use” and we start to “change it”. This argument does not quite hold, because we are designing a new chess playing machine that will replace the old one. Thus, this step of “retraining” is happening outside of the utilisation and would not contribute to that the system is considered adaptive. If you would like to draw a simple parallel, imagine a software that does not use any self-learning algorithm, such as a website. Occasionally, you want to redesign the functionality or appearance of the website, so you simply publish a new version of this website.
Now, let's generalise from the chess playing machine that we have built to various industries. The same principle of self-learning and accompanying change that we have utilised, applies in a vast array of AI applications across the EU.
In simplest cases, we start with risk rating, or creditworthiness algorithms, where we commonly use machine learning models that can learn for instance, on historical cases of loan defaults to predict who is unlikely to pay back requested loan. Just like our chess machine, this model learned on the historical cases, became fixed at the deployment step and after that point, we simply utilised its fixed state. It is a common practice that these models get retrained regularly, for instance every year to account for recent trends in the financial market.
Let's now take it to an extreme - large language models, with the example of ChatGPT. Again, just like our chess machine, it learns within the training phase. Namely, it learns our language patterns and word interpretations. At the deployment step, these become fixed and if you now open up conversation with ChatGPT and interact with it, it is not self-learning right away from your inputs.
Both examples, from the simple to the complex, illustrate a common pattern: AI systems undergo a phase of learning and adaptation but remain static in their application until they are retrained. This process does not align with the AI Act's narrow definition of adaptiveness, which requires ongoing learning during use, raising questions about the legislation's applicability to many AI technologies in practice.
Online learning
To be fair, there is a subset of models within the realm of machine learning which “changes while in use”. A common reference for these algorithms is online learning. If we stick to our chess playing example, we would design the machine to learn after every game (or a small batch of games) it plays. My practice as a data scientist and consultant however taught me that in practice, we rarely utilise this set of algorithms in real world use cases. We could find usefulness for this continuous self-learning practice in stock-market predictions, where recent trends might hold tremendous importance for predictions of where the market will move. If we would stick to the previous approach of, for instance, only retraining the algorithm once a year, it would be terribly inaccurate as it would never account for recent trends.
One of the prime reasons why data scientists rarely utilise online learning, is that these algorithms are prone to various kinds of dangers, ranging from technical difficulties, up until straight up cybersecurity issues. For instance, imagine that someone wants to hurt our website where people can come and challenge our machine in a game of chess. This attacker is aware that we use self-learning and the machine will learn from every game it plays. The attacker will purposely play irrational moves within several games, hoping that the self-learning algorithm will pick these up, and deteriorate in performance.
To sum up this brief extension to my text, online learning algorithms represent a tiny fraction of currently productive AI use cases. If the AI Act was aiming to regulate just these, it would be quite pointless regulation.
Does ChatGPT qualify as AI?
For the fun of it, I asked ChatGPT itself, whether it would view itself as fitting the ideas of “adaptiveness” and “autonomy”.
User:
I would like you to elaborate on the following. Today is March 15, 2024. Tell me, since when do you consider yourself being "in use"? Secondly, when do you consider that was the last point in time when you exhibited self-learning capability?
ChatGPT:
Since I consider myself "in use" from the moment I was deployed by OpenAI, and that happened in various stages over the last few years, with significant updates and improvements along the way. The version you're interacting with now, based on the GPT-4 architecture, was introduced before my last knowledge update in April 2023. So, in a broad sense, you could say I've been "in use" since the deployment of the initial GPT model by OpenAI, but more specifically, in my current form, since shortly before April 2023.
As for self-learning capability, it's essential to clarify that I don't learn or update my knowledge in real-time or after deployment. My learning occurred during the training phase, which happened before I was made available for use. During that phase, I analyzed a vast dataset of text from the internet and other sources up to my last update in April 2023. After this training phase, I do not update my knowledge or learn from interactions or new information. Thus, the last point in time when I exhibited self-learning capability was during my last training cycle before April 2023. After that, my knowledge and responses have been static, only reflecting the information available up to that last update.
User:
So assume that I am now evaluating whether you are AI system. I consider a system being AI, if it fulfills the condition of "adaptiveness" defined as follows:
"The adaptiveness that an AI system could exhibit after deployment, refers to self-learning capabilities, allowing the system to change while in use. "
Based on what you just said, and what you see, would you consider yourself fitting this definition of adaptiveness?
ChatGPT:
Based on the definition of adaptiveness you've provided, I would not fit this definition of an AI system. My capabilities do not include self-learning after deployment, meaning I do not adapt or change based on interactions or new information encountered while in use. My learning and knowledge acquisition were completed during the pre-deployment training phase, up to the point of my last update in April 2023. After that, my responses and knowledge remain static, without the ability to autonomously update or adapt.
User:
And finally, would you consider yourself "autonomous"? If yes, to what degree? For instance, following this definition of "autonomy":
AI systems are designed to operate with varying levels of autonomy, meaning that they have some degree of independence of actions from human involvement and of capabilities to operate without human intervention.
ChatGPT:
Under the provided definition of "autonomy," I exhibit a certain level of autonomy but with limitations. Here's a breakdown of how that applies:
Independence of Actions from Human Involvement: I can generate responses, perform tasks, and create content based on the instructions given to me without the need for human intervention in those specific tasks. Once given a prompt, I operate independently to produce an output based on the patterns, rules, and information I learned during my training.
Capabilities to Operate Without Human Intervention: While I can perform a wide range of tasks without ongoing human intervention, my operation still relies on several layers of human involvement, including:
Design and updates: Humans designed my architecture, determined my training data, and update my software and models.
Setting objectives and tasks: I require humans to provide prompts or tasks for me to respond to. I do not self-initiate tasks or determine my objectives.
Oversight and moderation: Human oversight is necessary to review and moderate my outputs to ensure they meet certain standards and are appropriate for various contexts.
Limitations on Autonomy:
I cannot learn or adapt post-deployment based on new information or feedback unless I am updated by humans.
My responses are limited to the knowledge and capabilities embedded in me up to the last update.
In summary, while I possess a degree of operational autonomy allowing me to execute tasks and generate responses independently once directed by a human, this autonomy is bounded. It does not extend to learning, adapting post-deployment, or operating outside the scope of my pre-determined programming and capabilities. My autonomy is fundamentally constrained by the design, objectives, and boundaries set by human developers and operators.
Conclusion & Implications
As we've seen, the EU AI Act, despite being ratified by the European Parliament, harbours a significant oversight. The act's current definitions fail to encompass systems like ChatGPT—undeniably a product of artificial intelligence—under its AI umbrella. This discrepancy suggests that numerous applications and projects within the EU, commonly recognized as AI, similarly do not meet the legislative criteria.
The EU has repeatedly expressed its intention to regulate sophisticated algorithms akin to ChatGPT. This oversight, therefore, seems more like an administrative lapse than a deliberate exclusion, leading me to anticipate a necessary revision of the AI definition or its clarifying recitals. Such an amendment would ensure that a broad spectrum of algorithms integral to our daily digital experience in the EU are rightfully acknowledged as AI. It's crucial to note, however, that aligning with the AI definition doesn't automatically subject an algorithm to stringent regulations. To fall under more rigorous scrutiny, an algorithm must also meet specific criteria for high-risk AI or be classified as General Purpose AI—topics I intend to explore in subsequent articles.
Need help with EU AI Act compliance?
If you're looking for support, I offer training, consulting, and audits. You may send your inquiries directly to my email robert@barcik.training .
If you are an individual who would like to simply get informed about this regulation in more detail, I have published a Udemy course, which I am committed to keeping up to date with the regulation progress.
Comments