Sampler postings republish pieces previously available only to paid subscribers.
This was originally published in two parts during April, the first in the
Asset Manager category and the second in Due Diligence.
Sign up for a free or paid subscription plan here.
For more than a quarter century, Michael Mauboussin has been producing research on decision making and investment process. Dan Callahan has been his co-author on articles for much of the last decade. Their latest essay is “Feedback: Information as a Basis for Improvement,” which fits neatly with The Investment Ecosystem’s goal of “continuous improvement for investment professionals and organizations.”
That piece will be used to look at investment practice from two points of view — first regarding the work of asset managers and then that of the allocators who do due diligence on them.
Disconnects
Despite the widespread popularity of Mauboussin’s work among investment professionals, many of the ideas and principles presented therein don’t result in actual implementation within organizations.
Furthermore, it’s perplexing that investment professionals — who are so open to new companies, strategies, and vehicles from an investing standpoint — can be anything but innovative when it comes to organizational design, investment process, and the like. That sort of bounded creativity stands in stark contrast to Mauboussin’s approach, which seeks to apply insights from other disciplines in order to improve investment decision making.
Those disconnects are on full display in “Feedback.” The authors aggregate a dozen “facets of process improvement” in three categories: people, organizations, and feedback during process execution.
People
This section opens with a fairly innocuous summary:
If you want to build an excellent, repeatable process you need to start with having the right people. You then need to train them to be effective, figure out how you should deploy their skills, and institute a policy of practice.
If, as the authors then suggest, the “first step is to identify the skills that can lead to success,” do asset management firms routinely examine them in detail, and do they analyze potential hires on that basis? In large part, no. Specifically regarding those with some years of experience, previous performance is more heavily weighted than it should be (given the inherent imprecision in performance as a measure of skill, as well as the often overlooked importance of the candidate’s previous environment in producing it).
The authors use the characteristics of good forecasters found in Superforecasting, the book by Philip Tetlock and Dan Gardner, to illustrate the kind of skills inventory that firms should be using — a list that is particularly appropriate given the importance of forecasting in so many investment roles and the lack of effort put forth to try to improve the results from that activity.
They also delve into the differences between intelligence quotient (IQ) and rationality quotient (RQ), noting that “how people think is more important than raw measures of intelligence.” That leads to an examination of the differences between “fluid rationality” and “crystalized rationality” — including what can be tested for (and what can’t), as well as in what categories improvement is most likely (and where it isn’t).
Can training provide that improvement? If so, how? The report addresses ways to improve forecasting skills; to identify and institutionalize standards for analytical concepts that are prone to sloppy definition and application; and to assess “which aspects of investment process can be addressed systematically and which must be in the realm of judgment.” Also worth noting are the leverage effects that come from training on effective communication skills and data visualization techniques.
Given the nature of the investment realm, “it is easy to get drawn into a pattern of all playing and no practicing.” But how can one “practice” this craft? With the computing power and gamification techniques of the day, you’d think that advanced simulations could be developed to both hone desired skills and provide exposure to a range of possible scenarios to experience. In the meantime, the authors offer some more prosaic options.
The section closes with this (an apt segue into the next topic):
There is a lot of opportunity to improve coaching in the investment business . . . it starts by professionals acknowledging that there is room to improve.
Organizations
Many of the shortcomings in the areas noted above can be traced to the culture of the industry as a whole and asset management in particular. “Values and norms are influenced by governing objectives, incentives, teamwork, and the overall organizational environment,” so examinations of those come next.
Asset management firms “have tension between two potential objectives,” one related to the goal of producing long-term returns for their clients and the other focused on the financial success of the firm and its employees. Most often, that tug of war leads to playing the game as it is currently being played; the industry is marked by similarity rather than differentiation and inertia instead of innovation.
Incentives can play an important role, but “effective incentives are very difficult to design, and incentives by themselves rarely promote the proper behaviors.” In fact, many firms cling to incentive structures that reflect the noise of the market — and most equate the performance of an individual to the “numbers” that appear to represent their worth. That leads to a not-so-merry-go-round of hopes that the results will be better next time, instead of an orientation for improvement grounded in the need for feedback on the fundamental drivers of long-term value-added.
Over time, there has been a massive increase in the proportion of assets that are managed by teams versus individuals — at least superficially, since it’s unclear how many are in fact what could be called real teams. Usually lacking in their construction are careful consideration of the importance of collective (and multifaceted) intelligence on a team, the elements that lead to team success, and the kinds of training that are helpful. Yet those factors likely are more important than traditional drivers of team composition (previous areas of experience and track records).
The “environment,” or culture if you will, of an organization is the soil in which everything grows. It should be “committed to ongoing learning,” should “welcome different points of view,” should audit the decision-making process in ways that lead to specific and ongoing improvement, and should engender an atmosphere of equanimity:
All organizations go through ups and downs, so removing peaks and valleys is essential. A focus on proper process with a long-term view contributes to this attitude.
Feedback during process execution
The final section includes some specific feedback techniques to improve investment processes:
Providing feedback to fundamental investors is inherently difficult because the process to make decisions is often poorly defined and the outcome, the rise and fall of asset prices, is noisy in the short run. One way to solve this problem is to break down decisions into measurable components. But it all starts with documenting decisions and how they are made. Without a record of what you expect to happen, it is difficult to measure the accuracy of your predictions.
That means identifying in advance the “variant perception, a well-founded view that the market has not priced properly” which is motivating a particular decision. Using a “linchpin analysis” to flesh out that variant perception can help to reveal the weaknesses of the proposition, and it also serves as a roadmap for the decision process that can be evaluated after the fact.
One specific way to track the accuracy of predictions about probabilistic events is called the Brier score. Is it widely used? Not at all:
Most investment organizations have the raw material to keep Brier scores but fail to do so. Part of it is that keeping track of decisions requires some discipline. But the bigger issue is that when we are wrong, we generally cope by crafting a story to explain what happened in a way that avoids making us look bad. We are poor at predicting but great at explaining the past.
The same can be said for other pieces of what could be called a process attribution. The information is there to analyze whether the individual components of that process diagram in the pitch book actually add value, but very few managers perform such analyses and fewer still reveal what they have found.
Many aspects of process feedback have gotten easier; there are more opportunities for “a machine giving feedback in order to improve a human’s game.” What’s lacking, though, is the commitment to identify the feedback loops that are important, to communicate how they can help to move the organization forward over time, and to clear away the cultural impediments that inhibit progress.
Your scorecard
If you work at an asset management firm — and especially if you are in a leadership role — take the time to walk through the items mentioned above in some detail and grade your organization. What are you doing and what are you not doing to build the kind of feedback culture that yields the ongoing improvements that you will need to compete in the future?
For those of you that evaluate asset managers, stay tuned for the next posting, which deals with the implications of all of this for your efforts to separate the wheat from the chaff.
~~~~~~~
The above posting summarized the essay by Michael Mauboussin and Dan Callahan in order to question whether the kind of ideas expressed within it are being applied by asset management firms.
This time, we will consider how the same ideas can be used by those doing due diligence to gain greater insight into the organizations that they analyze. Such avenues of inquiry offer the opportunity for explanatory depth beyond the narratives provided by the firms.
(Many of the ideas presented here are expanded upon in the Investment Ecosystem Academy course on due diligence and manager selection. The need for “explanatory depth” is one of the core concepts within it; “look both ways” is another tenet of the training.)
Look both ways
While investment organizations across the spectrum differ from each other in fundamental ways — asset managers are not like asset owners or investment advisory firms, and even within those categories there are recognizable subsets — many of the principles that can be used to analyze them are remarkably consistent.
Take the categories of evaluation that were the historical “4Ps” of philosophy, people, process, and performance. All of them are relevant no matter where you go or what kind of organization you’re looking at. Many of the finer points in those categories and others apply universally too.
So “look both ways” is a reminder that insights that allow for a more intensive look at asset managers also open doors to evaluating your own organization. Therefore, the piece by Mauboussin and Callahan can first be used in that way, to examine your practices and consider opportunities to get better over time.
An uncomfortable topic
Since a large part of a due diligence analyst’s job is to crack open the narrative of an asset manager and determine what’s real and what’s not, one goal should be to accumulate topics and techniques that facilitate that effort.
A fruitful line of inquiry involves asking about what things need improvement. It is surprising how often people struggle with questions like that, even senior people, who are most expert at reciting the narrative and are often reluctant to share anything that seems negative. For example, they tend to be protective of the investment process as advertised, using the industry catchphrase of “consistent and repeatable” to describe it, when words like learning, evolving, and improving would represent the qualities that are likely to win over time.
Their inability (or unwillingness) to identify concrete areas for improvement indicates one or more of these dynamics at work: a) they haven’t really thought about it much; b) they genuinely don’t believe that the organization needs to improve; c) there is a cultural aversion to talking about shortcomings within the organization; and/or d) they don’t want to be open with investors about what they need to do to get better (not a great foundation for a long-term partnership).
Whether an interviewee is cooperative or intransigent in response to general (and more detailed follow-up) questions on potential areas for needed improvement, there are opportunities to unearth unusual insights about a firm by asking them. The fact that those exchanges may be uncomfortable are an indication of their value.
Angles of discovery
That brings us back to the ideas put forth by Mauboussin and Callahan. They show ways in which feedback can be used in organizations guided by an ethos of continuous improvement, providing a convenient set of variations on a theme for engaging with an asset management firm.
The suggestions below also apply broadly to the use of other sources of information — from within the investment world and outside of it — that can be used to reveal aspects of an organization that otherwise tend to remain hidden. The Mauboussin/Callahan piece is a particularly good example of how it can work.
Like any organization, an asset management firm makes choices, large and small, that define it. Many of them happened years ago, while others (especially investment decisions) occur on an ongoing basis. Much of the engagement with those doing due diligence on the firm focuses on what the manager has done (and what it intends to do): Here’s the philosophy, here’s the structure, here’s the people, here’s the process.
Much more important in getting to the heart of the matter are the why questions along each dimension — and that’s where the advantage can swing in favor of someone doing due diligence, if they are more well versed in those topics than those they are interviewing.
Consider just a few of them referenced in the last posting: forecasting, training, team composition, and environment/culture. It is not at all difficult to build a base of knowledge to be able to ask revealing questions (in search of that explanatory depth) in any of those areas. Asking why or why not questions about well-researched organizational topics can lead to a dialog that takes you beyond the narrative descriptions on which most due diligence reports are based.
The items on the Mauboussin/Callahan list can be used that way, allowing you to move past the rote topics to more fertile ground, where there usually aren’t pat answers at the ready.
But, to take it a step further, imagine showing up for a due diligence meeting and saying, “I know your meetings with allocators aren’t usually like this, but I’d like to spend the time today talking about how improvement happens in your organization. That will help me to understand how your firm got to this place and how you might grow and adapt over the coming decade or more of our relationship together.” Such an approach would result in more differentiated insights than come out of the typical manager meeting.
Another good time to use outside ideas like this is in those boring quarterly update calls where not much is said of value. Instead, they should be seen as opportunities to expand your knowledge of the firm. Exploring uncovered ground and introducing new concepts is a good way to do that.
Your scorecard
In the previous posting, asset managers were asked to complete a scorecard on where they are on the various means of improvement that were discussed. Now it’s time for those of you doing due diligence to fill out your own report card.
How often do you incorporate ideas into your process that would be considered uncommon topics in due diligence but which can provide you with deeper insight into the organizations you are trying to understand?
What are the impediments to incorporating angles of discovery like that into your work?
If you look at the twelve facets of process improvement identified by Mauboussin and Callahan, how much do you know about what the firms you cover think about the possibilities around each and what they actually do in practice?
Published: August 23, 2022
To comment, please send an email to editor@investmentecosystem.com. Comments are for the editor and are not viewable by readers.