A commonly-asked question from those considering due diligence training from The Investment Ecosystem is, “What proof is there that this intensive focus on qualitative research results in better manager selection?”
The straight answer is that there is no “proof,” in the sense that there are no studies that have attempted to do such an analysis in any kind of a thorough way. We know what doesn’t work — performance chasing — although that hasn’t stopped allocators from gravitating toward those managers that have the best historical performance.
There are a few other quantifiable factors that have been studied which provide important evidence, the strongest of them being the level of costs charged to the investor. But none of those indicators is fail-safe, so “proof” is evasive. Furthermore, the body of evidence for some quantitative rules of thumb is often limited in scope, being primarily based upon the attributes of equity mutual funds, especially those in the United States. Other asset classes and geographies have not been studied to the same extent.
Analyzing qualitative factors is exponentially more difficult. While allocators talk extensively about the importance of the qualitative assessment of a manager, very few of them codify or analyze their findings in an in-depth, systematic way. Therefore, while the impressions of those qualitative factors may be leveraged in the selection process, they aren’t tracked by individual allocator organizations (and if they are they aren’t shared externally), making more global examinations of manager characteristics out of the question. There is no “proof” because the raw information needed to buttress such a claim has never been collected or disseminated.
Analyzing private equity funds
Before we return to that theme, let’s look at a new paper, “Limited Partners versus Unlimited Machines; Artificial Intelligence and the Performance of Private Equity Funds,” by Reiner Braun and four co-authors. They analyzed around four hundred private placement memorandums (PPMs) from private equity firms, using econometric and machine learning methods to study fund performance and fundraising success.
The evaluation of private equity firms presents some hurdles for investors:
Private markets are characterized by non-standardized disclosures and significant information asymmetries between managers and investors.
Unlike mutual fund databases, private equity datasets are thin. Moreover, PE fund managers have considerable degrees of freedom to frame their track records at the time of fundraising.
The authors find that the quantitative information provided in the PPMs does not predict future performance. But that lack of corroboration does not hinder allocations, since “PE firm reputation (as proxied by size and number of funds previously raised) as well as past performance are significantly related to fundraising success.” Unfortunately, the research also shows that fundraising success is “unrelated to future performance.” Popular fund offerings don’t outperform.
On the other hand, using natural language processing and machine learning techniques allowed the researchers to report “the central contribution of our paper: the analysis of qualitative information.” Some of their conclusions:
Our main results show that the three machine learning algorithms are remarkably effective at predicting fund performance.
One interpretation of the results is that institutional investors do not incorporate qualitative information but do incorporate the quantitative information provided to them. And as a direct consequence, quantitative information is unrelated to future performance, but qualitative information is.
In our view, this pattern of finding provides evidence that qualitative information is a valuable tool to learn about fund manager skills and may be one of the reasons why some LPs are better at this exercise than others.
Our findings suggest that the average investor does not seem to take into account relevant qualitative information when selecting fund managers.
Methodology and future analyses
The paper outlines in detail the approach taken by the researchers. Their process concentrates on the strategy part of the PPMs. Most of the other sections show a high degree of consistency in content, since “lawyers tend to largely copy-paste across PPMs.” In contrast, firm principals are involved in laying out the strategy description, so there is more variability from which to draw inferences.
As the authors state:
The application of new techniques to the analysis of private markets is just beginning. . . . There are certainly other areas of potential improvement as the disciplines of textual analysis and machine learning are growing rapidly and will provide even more powerful methods to be applied in the context of private capital markets.
Larger datasets are needed, closer inspection of some of the non-strategy sections of PPMs might yield discoveries, and other kinds of documents from managers which provide additional qualitative information may prove to be valuable. Also, the “marginal influence of features” on ultimate outcomes needs to be analyzed further; everyone wants to know which manager attributes truly matter.
This is just one study, so its conclusions shouldn’t be overemphasized, but doesn’t it whet your appetite?
Untapped possibilities
Which brings us back to the opening theme.
Allocators are sitting on a gold mine of information that can be analyzed. Materials sent by managers across the years can be studied in ways that they couldn’t be before — and the same can be done with investment memos about those managers. What would such an analysis reveal about the process of selection by allocators and the attributes of manager success?
All of this points out the need to start documenting qualitative characteristics, so that selection preferences can be judged and modified over time. While we are a long way from the ever-elusive “proof,” the techniques to evaluate qualitative attributes of managers (and of allocators) are now at hand.
Here’s that list of training options. You might also take The Basis Point Test.
Published: July 27, 2023
To comment, please send an email to editor@investmentecosystem.com. Comments are for the editor and are not viewable by readers.