Here is a recent NPR story about self-driving cars. Shankar Vedantam sums up the dilemma as follows:

Adam Millard-Ball… modeled what’s going to happen when self-driving cars start showing up on the road…[A]t its core, driving is not just about the physics of moving objects and the law of who can do what on the road. It’s also about psychology. People have developed really complex and often unspoken rules of how to interact with one another on the road. You can teach a self-driving car all the rules and give it the tools to navigate around obstacles, but can these cars deal with all the psychological games that human drivers and pedestrians play on the roads?

Vedantam then frames the issue in terms of a distinction between the “perfectly rational” behavior of an ideal self-driving car and the “irrational” psychology of human drivers and pedestrians:

we often make a big assumption and that big assumption is that rational behavior is always the right course of action on the road.…but when you think about it, a good part of driving today involves the unspoken assumption that other drivers may not always behave rationally. When it comes to interacting with self-driving cars, humans will know that the robot who’s operating the car will always do the rational thing.

It’s not hard to see that rationality is not what’s at stake here. And in fact Vedantam has already conceded that the real challenge for the AI at the heart of self-driving car technology lies in the network of tacit, intersubjective rules that govern drivers’ relationship to the explicit rules of the road.

Sadly, the primary example he offers of the conflict between car AI and human psychology concerns moral hazard, which is at best a form of collectively self-defeating behavior, not of either irrationality (since it routinely arises out of perfectly rational choices) or tacit rules.

The key thing to notice about these intersubjective rules is that they are, in fact rules. The social sciences have developed refined accounts of their operation. These include metapragmatics, game theory, behavioral economics, etc. In each case, the tendency is to explain the ways these tacit rules parallel, contravene and sometimes even govern the explicit ones. For example:

1. Metapragmatic analysis concerns the rules governing the use of the formal—grammatical and syntactical—rules of language use. Anthropologist Greg Urban sums it up as follows: “‘Metapragmatic’ refers to linguistic signs that are about the pragmatic code, about how to interpret the extrasemantic meanings encoded in speech.”
2. This attention to the ideological assumptions about social life embedded in the tacit rules governing the application of explicit rules overlaps with sociolinguistic analysis, which focuses on the way ordinary language is “diverted” from its ostensibly inherent aim of converging on some shared meaning and information and used to orchestrate, maintain and alter relationships of membership, status, power, obligation, and the like.
3. Meanwhile, behavioral economists and psychologists assert that a decisive gap separates “fast” and “slow” forms of cognitive processing, such that official or formal rules are derived on the basis of the latter even as much routine behavior is governed largely by the former.

These and other approaches capture most of the crucial features of what Lacan called the Symbolic Order. But they overlook at least one crucial element: the way this “extra” set of rules supplements or sustains the formal ones by enabling us to violate them and to manage our access to mandatory forms of enjoyment.