Rational Agents and Good Behavior

background-shape
feature-image

It’s probably not that hard to define what a Rational Agent is on a surface level. It’s an agent that makes the right decision, does the right thing. That’s what we want, an AI that makes the right choices because that’s better than one that makes the wrong choices. However, what does it mean by making the right choice or decision? How do we define it? Is it even possible to define something as Absolutely Rational?

Defining Good Behavior

It’s certainly impossible for me discuss the various philosophical definitions of Ethics in this post for two reasons, I tend to have a word limit and I am not educated enough regarding it. Therefore in this section, I am going to just discuss a few common approaches in defining ethics, namely Deontology and Consequentialism.

Immanuel Kant's Tombstone in Königsberg.

Deontology defines a choice as rational based on the intent or motive behind the action rather than the consequence of it. Under deontology an act is considered to be right or rational even if it has negative consequences, provided it follows the moral laws. This idea is resonated in Immanuel Kant’s theory of ethics, which many considered deontological. Kant argued that its not the consequence of the action that determines it’s nature but rather the motive of the person carry it out.

On the other hand, Consequentialism, as the name suggests, determines the nature of an action based on its consequences. A classic example of this is lying to save someone’s life. While lying is considering as morally wrong, the consequence of the action makes the rational choice. G. E. Moore, an English Philosopher and among the founder of Analytic Philosophy, gave a standard formulation of consequentialism based on these three rules:

  • An action ought to be performed if and only if its outcome is better than that of every alternative.
  • An action is right if and only if its outcome is at least as good as that of every alternative.
  • An action is wrong if and only if it is not right.

Both these approaches have their own merits (and it must be noted here that these two are not the only approaches out there). However given it is easier to quantify a consequence of an action compared to the motive behind it, Consequential approach has taken the front seat when it comes to Rational Agents in AI.

Performance Measure

Evaluating an action based on its consequences does seem more intuitive when dealing with algorithms, because consequence is quantifiable compared to motive. Let’s consider a simple example; I bought shares of two companies, say A and B. After a few days I sell the shares of A for $50 profit while I make a loss of $50 on B. The consequence of buying shares of A and selling it is a profit of $50, while for B it is a loss of $50. Clearly A was a better choice than B.

We can train a stock market agent, using the consequence of the action as a metric. Similar concept can be applied in other domains as well; classification models rely on accuracy of prediction, chess engine try to choose actions that maximize their chance of winning etc. It however is important here to recognize that each task requires a different performance measure that’s suited for it.

Deciding on Performance Measure

The above video shows a Hide and Seek Reinforcement Learning model trained by OpenAI. The task is rather simple, hiders try to find strategies to be not found while the seekers' objective is to find the hiders. While the objectives of the player are simple, the games remains interesting as the players can’t see the whole domain, unlike say chess.

The Hide & Seek Environment here is a partially observable domain; at any given time the player can only see a part of the map and not all of it. On the other hand, chess is fully observable domain, as the whole board is available for the player at all times. In partially observable domain like Hide & Seek, we can only make the optimal decision based on the information available to us, therefore it needn’t be the best decision according to an omniscient perspective.

Let us now consider what could be a good performance measure for seekers in this case. The objective is obviously to find the hiders within the time constraints, therefore doing so will be rewarded with maximum points. Furthermore we can guide the seeker further by giving it some points for exploring the map, analogous to searching in real life Hide & Seek games.

Conclusion

While it is certainly not easy to define ethical or rational behavior in humans, the idea of consequentialism does seem to do a fine job in AI systems, as it is more quantifiable. Many of the algorithms that power the modern AI are trained based on this concept, by providing the AI with a score for it’s action or choice; probably more prominently visible in Reinforcement Learning models. Maybe in some future posts, we can discuss this in more detail along with examples.

Reference