After one of my finance professors declared that there was insufficient literature on any financial theories that contradict the CAPM, I realized that the academic establishment did not simply ignore behavioral finance; it ignored any theory that did not resemble a mathematical proof. While I do enjoy a particularly clever proof now and then, mathematics does not offer sufficient accuracy for market prediction and I doubt that it ever will. The problem exists in both the tools that we use to tackle the fundamental problems of finance as well as the philosophy that determines the assumptions of those models.
Much of what underlies the belief in efficient markets is based on the Expected Utility Hypothesis formulated by Daniel Bernoulli in 1738. The theory states that the value of an asset is equal to the probability weighted expected values. Using a lottery example, one would expect to pay $1 for a ticket where the payout is $1 million and there are 1 million tickets sold. Expected utility works fine when the probabilities are known but fails to explain why investors are averse to ambiguity.
The Ellsberg paradox describes violations to the expected utility hypothesis due to uncertain probabilities. The existence of ambiguous probabilities separates the market from the casino exposing investors to Knightian uncertainty and results in real world decision making that does not conform to expected utility. The missing ingredients in the expected utility hypothesis are the miscalculations of risk about risk and the psychological costs or benefits associated with extreme events.
In 2002, Daniel Kahneman received the Nobel Prize in Economics for his work with Amos Tversky on behavioral finance. Kahneman and Tversky introduced Prospect Theory in 1979 which was later developed into Cumulative Prospect Theory through the use of rank-dependent weightings in 1992. The big new idea presented in CPT is that people base decisions on a reference point rather than absolute utility and people exhibit risk aversion and risk seeking behavior as the magnitude of the outcome increases. This leads to a utility function that has both concave and convex portions while also exhibiting overall risk aversion since more utility is gained from minimizing losses than maximizing gains. In the concave portion, people exhibit risk aversion and engage in behavior such as buying insurance against a low probability event with a large negative effect. In the convex portion, people exhibit risk seeking behavior such as buying a lottery ticket that has a low probability large payout. The key point of CPT is that the same individual will engage in both behaviors because there is additional utility gained or lost when a large effect from a low probability event occurs. This is why gamblers prefer bets on an unlikely winners than expected winners. For a more quantitative explanation, read up on stochastic dominance. Here’s Daniel Kahneman addressing the Georgetown University class of 2009 via Fora.
One problem I do have with behavioral finance is that there seem to be many theories that explain everything while predicting nothing. This is fine for philosophers in search of Truth but not very good for those of us managing money. Perhaps someone a little more or a lot less intelligent than I might conclude that human behavior is inherently unpredictable, but that is likely what draws most of us to study finance; the promise of predictability is just over the next mountain.