How investment markets can be both efficient and inefficient depending on the environment.
In this episode you’ll learn:
- What is the efficient market hypothesis.
- Why we are not optimizers but satisficers.
- How investors use emotion and heuristics to navigate complexity.
- How hedge funds are indicator species to financial market turmoil.
Welcome to Money For the Rest of Us, our personal finance show on money, how it works, how to invest it and how to live without worrying about it. I’m your host, David Stein, and today is episode 170. It’s titled “Are Financial Markets Efficient?”
The Efficient Market Hypothesis
When I was in graduate school and an undergrad I studied finance, and one of the theories that was really conveyed as a doctrine, and there was really no questioning it, was the efficient market hypothesis. Some of the early academics that developed the theory include Paul A. Samuelson, who won the Nobel Prize in Economics in 1970, and Eugene Fama, who won the same prize in 2013.
Eugene Fama in 1965 – which gives you an idea how old this theory is – wrote:
“An efficient market is defined as a market where there are a large number of rational profit-maximizers actively competing, with each trying to predict future values of individual securities, and where important current information is almost freely available to all participants. In an efficient market, on the average, competition will cause the full effects of new information on intrinsic values to be reflected instantaneously in actual prices.”
So the information, everything that has happened in the past, is reflected in the price, and the current price also reflects all the expectations of the market participants, such that it then becomes impossible to predict future prices in terms of what’s going to happen next, because everything – expectations of investors, what’s happened in the past – is reflected in the price.
Now, Fama outlined different forms: the weak form, the semi-strong form, and the strong form of the efficient market hypothesis. The weak form says prices fully reflect all available information contained in past prices.
The semi-strong form essentially says, well, it does reflect everything in the past, but if you use public information like a company’s earnings, sales, the price/earnings ratio to pick stocks, it’s pointless, because that’s also already reflected in the price.
The strong form is even private inside information, if you try to develop a trading strategy based on that, it’s just not effective; it’s also reflected in the price. So why spend time trying to pick individual securities if everything is already reflected in the price? We might as well index the portfolio. That’s what I was taught, and I only sort of believed it… And I didn’t believe it really at all once I took a job as an investment advisor, an institutional investment consultant, because one of our responsibilities was to pick managers – stock manager, bond managers, hedge funds who could outperform the market. I met with these managers and I saw that many did actually outperform the market.
Now, proponents of the efficient market hypothesis would say that they were lucky, and we’ll explore that a little more… But I had this contradiction, as an investment advisor, because I’d been taught one thing all during my academic career, and now as a professional investment advisor, I was essentially doing the opposite. It would be like going through medical school, learning all the doctrine, how to go about it, and then getting out and practicing some medieval form of medicine; bloodletting, or something like that.
It’s Not as Clear Cut
In 1996 I attended a conference – it was the Institutional Investor Roundtable for Consultants and Institutional Investors. This was about a year into my tenure as an investment professional. One of the speakers – I don’t remember very few of the speakers, I only remember one… And I don’t even remember what he spoke about, but it was Andrew W. Lo. He is a professor of Finance at the MIT Sloan School of Management. He was the first academic I had ever heard suggest that maybe the efficient market hypothesis wasn’t as clear-cut, sound doctrine as many in academic fields, as I’d been taught in my textbook, that all my professors had shared – maybe it isn’t quite like that.
I was also starting to experience that in — while I didn’t necessarily believe it, I started to see examples of maybe investors aren’t quite as rational as Fama suggested. In 1997 we had the Asian financial crisis. I saw emerging markets lose 30% of their value from July 2007 through December 2007. Investor panicked, I saw that. I saw my clients that had emerging markets exposure start to question “Why are we doing this?” It was a tough time.
In 1998 Long Term Capital Management, a hedge fund, essentially lost pretty much 87% of its value in about a three-month span. The Federal Reserve had to organize a bail-out because they had used so much leverage, but that was also spawned by investors panicking – both their investors in their fund, as well as investors generally. Russia defaulted on its debt, and there was another panic. The S&P 500 index (large company U.S. stocks) fell 18% from mid-July to early September 1998.
Then we had the internet bubble. We saw extremely high valuations for internet stocks, and U.S. stocks overall. Indexing was becoming all the rage, and clients wanted to index — we were doing some indexing in terms of the large company stocks because frankly we were having trouble finding managers that could outperform in the large company space, and so we were using active management in small cap stocks, in international stocks… But large cap – we were thinking indexing would be a good thing, but then the valuations of those index funds kept getting higher and higher. The S&P 500 index, the price/earnings ratio was approaching 35 times earnings, which was an extreme overvaluation.
At the same time, some examples of behavioral finance, where that showed evidence that investors weren’t always rational. I talked a little bit about that in episode 138 of this show, titled “Should you sell your stocks before Trump takes office?” I discuss a book by Michael Lewis, a very excellent book, The Undoing Project; he profiles a Nobel Prize winning psychologist, Daniel Kahneman, and Amos Tversky.
So I had all this evidence – or not even evidence, just anecdotes – that something was wrong here. I was seeing panic and euphoria, extremely expensive asset classes and very cheap ones… And my question was that the efficient market hypothesis suggested that there was wisdom of the crowds that collectively, while individuals might make mistakes in terms of what they thought was going to happen, there were enough other investors that could profit from those mistakes, that mispricing, that overall the emerging market — that the markets were efficient, that prices of a stock reflected its true, intrinsic value.
But Andrew Lo wrote a book which just came out, called The Adaptive Markets: Financial Evolution at the Speed of Light. In the book, he discusses the adaptive market hypothesis, which is his theory that he believes captures the efficient market hypothesis, but expands it. And he says it several times in the book – it takes a theory to beat a theory.
Academics had drunk the Kool-Aid. They were so convinced of the efficient market hypothesis that anyone that pointed out perhaps some flaws in it… I’m thinking of examples. Well, Benoit Mandelbrot did a lot of research in the ’60s that showed that maybe markets weren’t entirely efficient, as he looked at future prices; he discusses that in his book, “The (mis)behavior of markets”, which I’ve referenced in earlier episodes… But he was kind of ostracized; he was not from traditional finance, and essentially, he was ignored, and other academics were ignored.
Discovering the Cracks
Lo says the reason why is that finance academics – they have physics envy. 99% of what happens in the physical world can be predicted, or at least can be described based on Newton’s three laws of motion. So it’s very, very mathematic-driven (they’re driven by mathematics), and finance professionals want the same thing as the academics, so they developed CAPM, efficient market hypothesis, modern portfolio theory. But Lo states in his book while 99% of what happens in the physical world can be explained by Newton’s laws, there’s probably 99 rules that explain only 3% of the behavior of investors. While you can have math to sort of describe what’s going on in the financial markets, there’s a lot more going on that makes it very, very difficult.
And investors are not rational, they’re irrational. Lo writes:
“The wisdom of the crowds depends on the errors of individual investors canceling each other out. But if we all exhibit certain behavioral patterns that are constantly irrational in the same way, sometimes the errors don’t cancel out. If you use a defective scale that’s biased upwards, averaging your weight across multiple readings on the scale won’t give you a more accurate measure of your weight.”
In the case of irrational investor behavior, the errors can compound across individuals, replacing the wisdom of the crowds with the madness of the mobs. He goes on:
“While arbitrage and profit motive can exploit a misjudgment, they still rely on the ability of investors to recognize when a mistake has taken place. In many cases, this expectation is simply unrealistic. The history of markets is filled with rational investors going wrong with utter confidence in the soundness of their judgment, until brought down by information just beyond their range of consideration or understanding.”
So what is going on? Well, the world is too complex, and our mental capacity too limited for us to see six or seven steps down the road. If prices are to reflect everything, those expectations of investors, and those expectations should cancel them out if an investor maybe gets a little bit off, so the wisdom of the crowd takes hold – that suggests that most investors should be able to foresee surprises and expectations; it should be reflected.
In fact, they should be able to optimize their portfolio decisions, because that’s what portfolio optimization is – we get the inputs, we can optimize it, and there’s no way that a price of a security can get out of line from its intrinsic value. So we all must index.
But there was an academic in the ’40s and mid ’50s, Herbert Seidman – he was an amateur chess player. As a chess typical, he calculated that a perfectly rational player that is playing chess, over a 16 move sequence would have to examine a trillion trillion variations, whereas a typical player, even a grandmaster such as Gary Kasparov, only looks three to five moves ahead in a typical game… And only moving 3-5 moves ahead in the investment markets does not give investors insight, in my opinion, to accurately reflect prices.
So Simon proposed that we’re not optimizers. We can’t. It is too mathematically taxing to be able to do that. Instead, we “satisfice” (it’s the term he came up with). It’s a combination of the word “satisfy” and “suffice”. So when we make individual decisions, we calculate — there’s a cost to any decision. Lo points out if you’re doing taxes, deciding whether you’re gonna do your taxes or not – there’s a mental tax to doing taxes, and sometimes it pays to have somebody else do that. Well, we do that with all our decisions. We’re not optimizing, we’re deciding how much effort should we put into making this decision so that we get a good enough solution.
Andrew Lo gives a great example of how we satisfice, or get good enough solutions. He mentions in his closet he has 10 shirts, 5 jackets, 20 ties, 4 belts, 10 pairs of socks, 4 pairs of shoes. Together, combined, they can create over two million unique outifts, and it would take him 23.3 days if he spent 24 hours a day, trying to optimize or solve the problem what is the most optimal outfit for that day’s work. Obviously, he doesn’t do that. He uses heuristics, rules of thumb, and that is how he satisfices.
For example, in his case his pants match his jacket. He has suits, so that simplifies the problem. Plus, he has experience; he’s gone to dinner parties where he was in jeans and everybody else was wearing a suit, and that emotional experience, how he felt (bad, because that was a wrong choice) allows him to come up with these rules of thumb, and that is how we live.
We talked about rules of thumb in earlier podcasts. He writes:
“How do you know in the satisfying process to stop optimizing? You don’t. You develop rules of thumb by trial and error. You usually don’t know whether a decision is optimal. Over time, you experience positive and negative feedback from your decisions, and you alter your decisions in response to this feedback.”
Basically, you learn and adapt to the current environment. Now, Herbert Simon came up with this theory, and the academic establishment didn’t believe it. Their point was “Well, how could you know you have a good enough solution if you didn’t know the optimal decision ahead of time?” Like, you can’t compare “good enough” unless you know the “optimal.” But Brennan, who was a co-author with Andrew Lo (Thomas Brennan), in his paper – they realize you don’t have to know the optimal decision… That our emotions, the feedback we get from our emotions through trial and error is what allows us over time to learn and adapt and then come up with these decisions.
“Emotion is the primary feedback mechanism that causes us to update our heuristic. Love, hate, sympathy, jealousy, anger, anxiety, joy, grief and embarrassment all serve useful purposes in telling something about our environment.”
I had this experience yesterday, I was out biking. I often bike along our greenway belt that goes along the river. It’s a multi-use trail. I got to a stoplight; I’ve gone through this stoplight dozens and dozens of times, so I usually wait until — well, it turns red, but if it’s turning yellow and I don’t see any cars, and I see the cars are stopping, I’ll go ahead.
Well, when I was biking yesterday, there was a couple – sort of a portly couple. They had a baby in a stroller, they had a little (I think it was a) Beagle puppy on a leash; they were enjoying their walk, but actually they were staring at their phones. So the light turned yellow, I saw the cars were fine, they were stopping, so I went.
About 20 minutes later, coming back toward them on the other side of the river trail, they were kind of walking in my lane. I slowed down a little bit, I let them get over, and then as I go past, he lets out an expletive; he says “You are a______” (you can fill in your blank). I passed him and I stopped, and I turned around, and I could see them stiffening up. He had let out, for whatever reason, an emotional response, and didn’t really think about it. It was his emotion. He was upset at something, and he took it out and he called me a name.
So I turned around and I said “What did I do wrong?” Because I thought maybe they thought I approached them too close when I was coming toward them… But no, he was upset that I had crossed on the yellow light. And I pointed out, “You actually weren’t even paying attention, because you were looking at your phone.” I thanked them for being concerned about me, but I didn’t get angry, which I think he was — he thought we were gonna get in a fight; you could see him stiffening up, and I was just… I don’t get called names very often. But I think in terms of satisficing, he gave an emotional response. It was a trial and error. He probably will not be as emotional at a biker, because he realized “Hey, they actually might stop and turn around and have a conversation.” And the fact that I was so calm actually threw him off-guard a little bit, but he learned from that experience; we all do that.
I’ve given emotional outbursts before, often while biking, because somebody cut me off, some car… But we learn and we adapt, and these emotions become our feedback mechanism to learn.
“Under the adaptive market hypothesis individuals never know for sure whether their current heuristic is good enough. They come to this conclusion through trial and error. Individuals make choices based on their past experience and their best guess as to what might be optimal, and they learn by receiving positive and negative reinforcement from the outcomes.
As a result of this feedback, individuals develop new heuristics and mental rules of thumb to help them solve their various economic challenges. As long as those challenges remain stable over time, their heuristics will eventually adapt to yield approximately optimal solutions to those challenges.”
The adaptive market hypothesis doesn’t label behavior as irrational, but sub-optimal. If something becomes irrational, they’ll label it sub-optimal because the heuristics being used might not fit the environmental context. He gives an example, “A great white shark on a beach.”
As Things Begin to Change
Now, the idea of indexing in efficient markets – that’s actually a subset of the adaptive market hypothesis. In a stable investment environment, which he points out we had from the 1930’s through kind of mid-2000’s – a fairly stable environment, where the higher the risk you took, the higher your reward was, and it was a context where the efficient market hypothesis worked very well; that was optimal. But since 2005, he’s found the environment has changed significantly, and then those old heuristics don’t work as well as they once did.
“In the case of the U.S. market, our environment has changed so much in the last two decades that the errors from assuming stationarity and rationality have greatly increased, and for reasons we can explicitly identify. They’ve reached a critical point where they can no longer be ignored.”
What he’s referring to is just sheer globalization, and how central banks have influenced markets more and more… And now it’s not enough anymore to — like you could 40 years ago do your due diligence on a stock, find that maybe it was mispriced because of the other investors (typically, it was a retail investor on the other side), and you could get some type of informational edge.
Now markets are efficient in the sense that there’s a lot of computing power trying to find these misprices. So when we talk about the market not being efficient, we’re not saying it’s easy to outperform the market. In fact, it is still very, very difficult, perhaps more difficult than it ever was because of the firepower that investment firms bring to the table. But at the same time, because of increasing complexity in the financial markets, the influence of many more countries – the U.S. is no longer the dominant economy; there are other economies out there – and just the sheer connectedness of markets, or people, the changing cultures, the politics, the world is getting more and more complex, which means it’s getting more and more difficult for investors to look six or seven steps down the road. There are more unintended consequences, and as a result, often times our heuristics don’t fit the current environment. That can lead to sub-optimal behavior, or irrationality. It can lead to bubbles. It can lead to what he calls “freaking out”, this run to quality or liquidity.
One of the characteristics he’s seen in the markets is that if you look over the long-term, a 90-year period, the higher the risk in terms of volatility, the higher the return. But if you look over rolling five-year periods, during periods of high volatility the returns were actually lower for stocks, because investors are freaking out and they’re running to the safety of lower-risk assets.
One of the questions he asks, which is pretty profound… He writes:
“Must passive investment always accept risk passively and never have the benefits of active management?” The answer is no, because he points out that active managers are looking at the environment and they’re trying to make changes, but index fund investors – or certainly the index managers – are just trying to replicate an index; they’re not worried about the market being overvalued, like we saw during the internet bubble. They’re just trying to replicate the S&P 500.
So if you’re a purely passive investor and you just have index funds and you ignore the environment, then you’re taking on a great deal of risk. Now, maybe you’re comfortable with that, maybe you’re comfortable going up and down in terms of riding that rollercoaster, but I’m not… And he’s not. He says:
“One of the most mind-numbing aspects of professional portfolio management is monitoring the portfolio in real time, and deciding when to act in response to rapidly deteriorating market conditions.”
That’s what we do on Money for the Rest of Us Plus. We’re not trying to outperform the market; we’re trying to be risk managers. We recognize that investors invest using heuristics, and that we can’t see 8 or 9 steps down the road. We have rules of thumb, and irrationality can creep into the markets to where over-valuations can occur, investors can panic and get fearful, particularly during economic slowdowns or other disruptions… So all we do is we try to monitor market conditions and be willing to take less risk when risk of some type of regime change or market meltdown – particularly related to a slowing economy – is there.
So are financial markets efficient? They’re not. Lo writes:
“Prices don’t automatically reflect all available information. How could they? We’ve noted buyers and sellers don’t use all available information to make their decisions; they use some information and heuristics instead, and the can very sophisticated.
As markets become more competitive, investors have to adapt their heuristics to maintain a profit. Under stable conditions, a virtuous circle of increasingly efficient strategies can evolve to take advantage of any scrap of information, misplace or arbitrage opportunity.”
So in stable environments, markets do get more and more optimized and efficient, si they adhere to the efficient market hypothesis. But as the dynamics change, as markets become more complicated, then you have a situation where the heuristics don’t work anymore, and we have to be willing to change… And the change that I’m willing to do is to monitor market conditions and be willing to risk-manage, and not say purely passive investing works all the time. I’m just not a buy and hold investor, that’s not how I invest anymore.
A couple other interesting points… He points out that hedge funds are a great indicator species. There’s so many hedge funds out there, some very, very talented, but when there is a dislocation in a market, usually it shows up first in hedge fund. We talked about that a few episodes ago (I forget the number) on “Signs of the next financial crisis.” We talked about some of the first indications in August 2007 occurred because some hedge funds were having difficulty.
Then the other really interesting point that sort of contradicts the efficient market hypothesis is that the efficient market hypothesis says that prices reflect all available information, so you can’t get any type of competitive edge, and you can’t outperform the market. But if that’s the case, then investors wouldn’t try to do so. Some investors, particularly hedge funds, are able to do that, so there’s a profit motive. If there was no profit motive, they couldn’t do it, then investors would stop and then markets would indeed be inefficient.
So the efficiency comes in a stable environment because some investors are actually able to earn excess profits and outperform… But in a stable environment, things get more and more efficient, it becomes more difficult to do. But as primarily buy and hold investors, what we can do – we shouldn’t spend our time trying to pick individual stocks; it’s very, very hard to do. It’s hard to get that informational edge. But we can look for when investors are becoming sub-optimal and irrational, when segments of the market are getting overvalued, or other areas of the market are undervalued.
Right now most everything is overvalued, so we’re looking for other market conditions – economic trends, we’re looking for the level of fear and greed, and then adjust based on that. That’s how invest.
That’s episode 170. You can get links to the articles and books I mentioned in today’s episode at moneyfortherestofus.com. While you’re there, if you’re not yet signed up, sign up for my free Insider’s Guide. This is a weekly e-mail I send you with those same links and show notes, but more importantly, I send a newsletter, an essay, something that only goes to you; it doesn’t get published anywhere else. It’s the best writing I do all week. It puts into context what we’re discussing in this week’s episode. It will include things that I forgot to put in the episode, since I write it the day after I record it. So some of my best writing, directly to you; it helps you put the episode into greated context. You can sign up for that also at moneyfortherestofus.com, or if you’re a U.S.-based listener, just text the word “insider” to the number 44222.
Finally, if you haven’t actually subscribed to the podcast on your favorite app, you just happened to look at the episodes, hit the Subscribe button and it gets delivered to you automatically. I would very much appreciate that.
Everything I’ve shared with you in this episode has been for general education only. I’ve not considered your specific risk profile, I’ve not provided investment advice. This is simply general education on money, investing and economy. Have a great week!