Competition Is a Discovery Procedure: When Schumpeter Meets Hayek

Innovation, entrepreneurial judgment, and the discovery value of competition

There is a basic and probably familiar tension in the economics of innovation: competition is regarded as desirable because it disciplines incumbents, reallocates resources toward more productive firms, and limits the rents associated with market power; however, innovation itself is an activity undertaken in pursuit of rents. If the innovator cannot expect to appropriate at least some of the surplus created by a successful innovation, why incur the cost and risk of producing it? The question whether competition promotes or retards innovation has generated a massive theoretical and empirical literature, much of it organized around the incentives facing individual firms.

Pedro Bento’s 2014 American Economic Journal: Macroeconomics paper, “Competition as a Discovery Procedure: Schumpeter Meets Hayek in a Model of Innovation,” approaches this old problem from a angle that is novel from the point of view of standard economics (it may be more familiar to those who know the work of Nelson and Winter … ). Bento argues that innovation is not simply an investment whose payoff is uncertain, but a search process in which the quality of the technological opportunities themselves is unknown before firms experiment with them. Once that distinction is introduced, the number of independent innovators becomes economically significant in its own right. Competition matters not merely because it changes each firm’s incentive to innovate, but because it changes how many technological experiments the economy conducts.

That sounds like a modest modification of the standard innovation problem. It is not. It introduces into Schumpeterian growth theory a Hayekian function for competition and, in the process, separates three relationships that are too easily conflated: competition and innovation at the firm level, competition and technological progress at the industry level, and the effect of competition policy on long-run growth. Bento’s most interesting result is that these relationships need not even have the same sign.

The usual Schumpeterian question

The modern literature on competition and innovation has inherited a recognizably Schumpeterian problem. Innovation requires costly effort today in return for an uncertain stream of rents tomorrow. Market structure matters because it changes both the rewards from successful innovation and the consequences of failing to innovate. Competition can therefore sharpen incentives, but it can also erode the rents that make innovative investment worthwhile.

Aghion, Bloom, Blundell, Griffith, and Howitt’s (2005) paper in the QJE showed how these forces can generate an inverted-U relationship between product-market competition and innovation. At low levels of competition, an increase in competition can induce firms to innovate in order to escape their rivals. As competition becomes sufficiently intense, however, the incremental rents from innovation shrink and innovative effort eventually falls. The attraction of the argument is precisely that it escapes the sterile choice between “competition is good for innovation” and “market power is good for innovation.” Both can be true over different parts of the relevant range.

Bento retains this incentive problem but adds another margin. In the usual formulation, competition affects how intensively firms pursue innovation; in Bento’s formulation, competition also affects the number of independent technological draws available to the economy. That matters because innovations differ in their realized productivity, and firms do not know ex ante which experiment will generate the superior outcome. The market is therefore not merely eliciting effort toward a known technological objective; it is sampling from a set of uncertain technological possibilities. Enter Friedrich von Hayek.

Intensive and extensive margins of experimentation

Economists often invoke Hayek’s description of competition as a “discovery procedure,” but usually (in my experience) rhetorically rather than substantively, i.e., as a general, loose claim that markets are dynamic, entrepreneurs are clever, or decentralized economies somehow generate information. Bento adds real content to the rhetorics.

Suppose innovative outcomes are uncertain and heterogeneous. A firm undertakes research, but neither it nor anyone else knows beforehand exactly how productive the resulting innovation will be. If several firms undertake independent innovative activities, society obtains several draws from the relevant distribution of technological possibilities. The technology that ultimately determines productivity is selected from this collection of realized innovations. Increasing the number of firms therefore changes the distribution of the best available technology even if the innovative effort of each individual firm does not increase.

The distinction is between the intensive and extensive margins of experimentation. Schumpeterian reasoning directs attention primarily toward the intensive margin: how much innovative effort does each firm undertake given the rents available to a successful innovator? The Hayekian mechanism adds an extensive margin: how many independent experiments are undertaken, and hence how many opportunities does the economy have to discover an unusually productive innovation?

This is not merely “more firms means more innovation.” Rather, the point is that, under technological uncertainty, the social value of an additional experiment depends partly on the order statistic it generates. Society cares about the best technology discovered, not simply the average quality of all research projects undertaken. Increasing the number of independent draws can therefore improve expected technological performance even while the amount of innovative activity undertaken by the representative firm declines. That is the key idea in the paper.

The average innovator and the technological frontier

Consider what happens as competition increases. Greater competitive pressure changes the rents associated with innovation and therefore changes the innovative effort chosen by each firm. Industry productivity depends on more than the innovative effort of the representative firm; it also depends on the number of firms generating independent innovations and on the quality of the best innovation among them. More competition can mean a larger pool of realized technologies from which the frontier technology emerges.

Suppose an industry moves from a relatively concentrated structure to one containing substantially more independent innovators. Competitive pressure may reduce the resources devoted to innovation by each firm. The expected innovative output of the representative firm may fall. But, with sufficiently many additional experiments, the expected quality of the best realized innovation can rise. Industry productivity growth may therefore increase precisely over a range in which firm-level innovative effort is declining.

This allows Bento to neatly reconcile seemingly conflicting empirical possibilities. Competition may display an inverted-U relationship with firm-level innovation while exhibiting a positive relationship with industry-level productivity growth. There is no paradox once one recognizes that firm-level innovative effort and the economy’s rate of technological discovery are different objects.

This matters empirically. R&D expenditure per firm, patenting per firm, or similar measures of innovative intensity do not necessarily provide sufficient statistics for the technological consequences of market structure. A concentrated industry may contain firms undertaking large research programs, while a less concentrated industry may contain a wider set of independent technological experiments. If realized innovation quality is heterogeneous, the latter structure may generate faster movement of the technological frontier even when its representative firm appears less innovative.

Competition as an epistemic institution

As already suggested, a key aspect of Bento’s paper is that competition does more than alter incentives or resource allocation. It also performs an epistemic function: it helps determine what the economy comes to know about its technological possibilities. If nobody knows which technology will prove best, allowing several independent experiments can reveal something that no participant knew beforehand. The relevant knowledge is generated through the competitive process rather than merely transmitted through it.

This is close to Hayek’s conception of competition as a discovery procedure. Hayek’s point was not simply that markets aggregate information already possessed by dispersed individuals (the focus of his 1945 paper). Competition also produces knowledge through action: prices are tried, products introduced, technologies tested, and conjectures about consumer demand or productive possibilities exposed to market selection. (the focus of his 1946 paper and o/c his 1968 paper ). Some economically relevant knowledge becomes available only because competing actors attempt different things.

Bento captures a tractable version of this idea within his model. Firms undertake uncertain technological experiments, and competition affects how many independent attempts occur. The best realized technology can then determine industry productivity. Competition matters not merely because it changes the innovative effort of existing firms, but because it changes the experimental process through which superior technologies are discovered.

There is, however, a limit to how Hayekian this mechanism can be. Bento represents discovery as stochastic technological search: firms draw innovations whose quality is uncertain beforehand. Hayek’s stronger argument concerns circumstances in which economic actors may not even know the relevant possibilities in advance. Discovery then involves more than learning which outcome is realized from a given distribution; it involves forming and testing conjectures about opportunities whose value, and sometimes whose very existence, cannot yet be specified.

This doesn’t weaken Bento’s result. The simplification is what makes the discovery mechanism analytically tractable. But, it raises a further question that the model itself need not answer: where do genuinely different experiments come from? One way of approaching that question is through the judgment-based account of entrepreneurship, which shifts attention from uncertain technological draws to the heterogeneous judgments that lead economic actors to commit resources to different visions of the future.

From technological draws to entrepreneurial judgment

The JBA offers something that is largely missing from Bento’s otherwise elegant Hayekian mechanism: an account of where heterogeneous experiments come from. In Bento’s model, competing firms generate independent technological draws which suffices to establish the formal result. If innovation quality is uncertain, increasing the number of independent experiments can increase the expected quality of the best realized technology even when innovative effort per firm declines. Competition therefore has a discovery value independent of its effect on the incentives of the representative firm.

However, from a judgment-based perspective, entrepreneurs do not simply draw technologies from an urn. They form judgments about which technologies to pursue, which consumer wants will become important, which resources are undervalued in their present uses, and which combinations of heterogeneous assets might generate future value. They then acquire or control resources and expose those judgments to the market process.

This distinction matters because the judgment-based approach, following Knight, reserves a central role for uncertainty that cannot simply be represented as risk with known probabilities. If the entrepreneur knew the complete set of technological alternatives and the probability distribution governing their returns, the decision problem would largely be one of optimization. Judgment becomes important when neither probabilities nor, in the stronger case, the relevant future possibilities themselves can be specified adequately in advance.

Seen this way, Bento’s competing innovators can be interpreted as entrepreneurs exercising rival judgments about the technological future. Their investments are not merely multiple attempts at solving the same known problem. They are conjectures about what the problem is, which resources will become valuable, and which combinations of those resources will produce value in circumstances that do not yet exist.

This gives competition an even stronger discovery function than Bento’s formal model requires. Competition does not merely increase the number of draws from a distribution of technological outcomes. It permits different theories of the future to be enacted simultaneously.

That is precisely where ownership becomes important in the JBA. Judgment is economically consequential because entrepreneurs can acquire, combine, and deploy heterogeneous assets according to their own beliefs. Ownership provides the entrepreneur with the authority to rearrange resources without having to establish beforehand that the proposed use is objectively superior. Indeed, if superiority could already be demonstrated, much of the entrepreneurial problem would have disappeared.

A decentralized ownership structure therefore permits disagreement to become productive experimentation. Entrepreneurs who hold different judgments can organize resources differently and allow subsequent market outcomes to discriminate among their conjectures. Profit and loss do not eliminate uncertainty ex ante; they provide an ex post selection mechanism among resource combinations generated under uncertainty.

This puts additional flesh on Hayek’s idea of competition as a discovery procedure. What competition discovers is not simply which of several known firms has lower costs. It discovers which entrepreneurial judgments about unknown resource uses survive contact with consumers, technologies, rivals, and complementary investments.

Bento and the JBA therefore illuminate different parts of the same process. Bento provides the formal logic for why a plurality of independent experiments can improve technological outcomes. The JBA explains the microfoundations of that plurality: heterogeneous capital, heterogeneous beliefs, entrepreneurial judgment, ownership, and the authority to commit resources under uncertainty.

The synthesis also sharpens the policy problem. Concentration may matter not simply because it changes conventional measures of market power or R&D incentives, but because it can alter the number of genuinely independent judgments being exercised over productive resources. Ten nominally separate research projects inside one corporation are not necessarily equivalent to ten entrepreneurs risking resources on competing visions of the future. Internal projects ultimately confront a common hierarchy, a common capital-allocation process, and potentially a common strategic conception of what the organization should become.

Conversely, merely increasing the number of firms does not guarantee meaningful experimental diversity. If firms imitate one another, rely on the same information, respond to identical subsidies, or pursue the same technological trajectory, an industry may contain many legal entities but relatively few independent entrepreneurial conjectures. The economically relevant variable is therefore not firm count as such, but something closer to the number and heterogeneity of judgments that can actually command resources.

This suggests a more explicitly entrepreneurial interpretation of Bento’s discovery mechanism. The social value of competition lies partly in preserving the possibility that somebody is wrong in a different direction. Under genuine uncertainty, we do not know which direction will eventually prove wrong, and we therefore cannot rationally select the winning experiment in advance. What decentralized entrepreneurship provides is a mechanism for maintaining rival conjectures long enough for economic experience to select among them.

The implication is subtle but important. Entrepreneurship is valuable not because entrepreneurs know the future better than everyone else. It is valuable because an entrepreneurial economy permits people with different judgments about an unknowable future to act on those judgments. Competition then becomes the institutional process through which those judgments are tested.

What should we measure?

Bento’s argument becomes harder to operationalize once we take entrepreneurial judgment seriously. If competition promotes discovery by sustaining multiple experiments under uncertainty, what exactly should we measure? Conventional indicators—R&D expenditure, patents, firm counts, concentration ratios—capture innovative inputs or market structure reasonably well. They tell us much less about the diversity and independence of the judgments underlying innovative activity.

Bento naturally directs attention to the number of independent technological experiments. The JBA suggests a further question: what makes those experiments genuinely independent? From a judgment-based perspective, what matters for discovery may therefore be not simply how many firms or research projects exist, but whether decision-makers are able to act on genuinely different judgments about an uncertain future.

Firm counts are an imperfect proxy for this. Ten nominally independent firms may pursue essentially the same technological trajectory, imitate one another, or respond similarly to common incentives. Conversely, a large firm may sustain competing research programs based on quite different technological conjectures. Organizational boundaries nevertheless matter because projects within a firm ultimately confront a common hierarchy and internal capital-allocation process. Common ownership can facilitate experimentation by allowing resources to be redeployed among projects, but it can also narrow experimentation if rival conjectures are screened out before markets have had much opportunity to test them.

Ideally, then, we would like to know how much genuinely independent judgment exists within an industry: how many decision-makers can commit resources to distinct technological conjectures, and how different those conjectures actually are. Patent similarity, technological distance among research programs, inventor mobility, venture-capital portfolios, and the organization of internal capital markets might provide partial evidence, but I don’t think there is any obvious sufficient measure.

The case for pluralism

The deepest implication of Bento’s paper is that the welfare consequences of competition cannot be inferred solely from what competition does to the innovative effort of the representative firm.

A concentrated industry can contain formidable research organizations, exploit economies of scale in R&D, and offer large prizes for successful innovation. Those are genuine advantages. However, concentration can simultaneously reduce the number of independent technological bets being placed. Whether the resulting system generates faster technological progress depends on the balance between the scale and incentive advantages emphasized by Schumpeterian reasoning and the experimental diversity emphasized by the Hayekian mechanism.

That is also why the paper cannot really be easily claimed by any either side of contemporary competition-policy debates. It provides no theoretical warrant for the proposition that breaking firms up necessarily accelerates innovation. Equally, impressive R&D spending by dominant firms does not settle the case for concentration. Both arguments focus on only one part of the innovation process.

Rather, the paper suggests that the real question is institutional. What arrangements preserve sufficient rents to induce costly experimentation while maintaining enough independent centres of decision-making to generate technological variety? How do entry barriers, intellectual-property rules, merger policy, research subsidies, capital markets, and organizational boundaries alter not simply total innovative expenditure but the distribution of experiments? And, under what technological conditions does the value of an additional independent draw outweigh the loss of innovative intensity induced by stronger competition? Those questions are perhaps tougher than asking whether competition increases R&D. They are also closer to the economics of innovation under genuine uncertainty and entrepreneurial judgment.