The Cosmological Constant Problem

Since the mid-1980s, astronomers and astrophysicists have been accumulating evidence that the expansion of the universe is accelerating (see the article by Saul Perlmutter in Physics Today, April 2003, page 53 ). The simplest way to incorporate that acceleration into the description of cosmology, within the framework of general relativity, is to add a cosmological constant (CC) term to the Einstein equations. Before Edwin Hubble discovered the expansion of the universe, Albert Einstein had originally introduced such a term to obtain a static solution of his cosmological equations. After the cosmic expansion was discovered, Einstein considered his introduction of the CC to be the greatest mistake of his career.

Many physicists were reluctant to consider the CC as an explanation for astronomical data, because the value it would need to have is ridiculously small compared to current theoretical expectations—some 10¹²⁰ times too small. Theorists interpreted that discrepancy as an indication that they would one day find an elegant explanation for why the parameter was exactly zero. Although some still cling to that hope, I conclude that observation has once again upset the expectations of overconfident theorists.

The framework that gives rise to the enormous mismatch between calculation and observation is called effective quantum field theory in background spacetime, or EFT for short. EFT always involves a short distance cutoff scale below which the approximations of EFT break down. The natural length scale introduced by quantum gravity (QG) is the Planck length—the combination of Newton’s gravitational constant, Planck’s constant, and the speed of light that has units of length. Naive dimensional analysis and explicit calculations in EFT suggest that the cosmological constant should be proportional to the fourth power of the corresponding Planck energy of about 10²⁸ eV. That’s 10¹²⁰ times too big.

Any dynamical solution of the CC problem within EFT should involve particles whose mass is on the order of the energy scale of the CC, about 10⁻³ eV. There have been many published attempts to resolve the problem by invoking such particles, and I can attest personally to many more unpublished ones, but all of them have failed. EFT does provide a loophole for resolving the CC problem: Apart from calculable contributions (see figure 1), there are contributions from energy scales higher than those corresponding to the cutoff. In principle, those two types of contributions can cancel, but from the EFT point of view, the cancellation to 1 part in 10¹²⁰ would be incredibly fortuitous.

I believe that the resolution of the CC problem does not involve some clever trick in EFT. Rather, QG will force on theorists a fundamental revision of the rules of the game. My belief is not yet the accepted dogma of the field. There are as many ideas about how to solve the CC problem as there are theorists who think about it. I will talk about only two of them.

These new insights into the nature of QG come from classical and semiclassical results about black holes in general relativity combined with results from string theory—our most successful attempt to build a mathematical model of QG. One important insight concerns the connection between energy scales and length scales. According to the Heisenberg uncertainty relation applied to excitations whose size is independent of their energy, high energy is related to short distance. That high energy–short distance connection lies at the heart of EFT, but it changes dramatically in QG when energies get higher than the Planck energy. It is replaced by a connection between high energies, large spatial distances, and large entropies. Once that new feature of QG is understood, one’s view of the cosmological constant problem is completely transformed.

Quantum gravity and measurement

Quantum mechanics appears to make precise mathematical predictions for the values of complete sets of commuting observables, but the measurement of those values seems to require classical measuring devices. The key to resolving this apparent dichotomy is recognizing that certain large quantum systems have a special class of observables that have robust, almost classical properties. Such “pointer” observables take on the same value for a large subspace of states of the system. An example is the magnetization of a large sample of a quantum ferromagnet. If one sets up a correlation between, for example, the microscopic spin of a spin-½ particle and the magnetization of a large ferromagnet, then one has measured the particle’s spin. There is some probability that the large ferromagnet will experience a quantum fluctuation of its magnetization, but that probability is very small. It would be zero in the limit of an infinitely large magnet.

The problem in any quantum theory of gravity is that every physical apparatus interacts with everything else gravitationally. Thus, a large measuring apparatus might have a huge gravitational interaction with the system it is measuring and thereby disturb the properties it is supposed to measure. The only way to remove that disturbance is to move the apparatus far away—infinitely far as the apparatus gets arbitrarily large.

The need to separate measuring apparatus and measured system suggests that precise observables in QG are only definable for spacetimes whose asymptotic regions have an infinite spatial extent. Indeed, string theory was originally defined as a scattering theory in an asymptotically flat spacetime, that is, a spacetime whose asymptotic region has no curvature. Unlike EFT, string theory has no precisely defined observables that are even approximately localizable. Lagrangians that are local in the bulk of spacetime enter into string theory only as approximate tools for summarizing the low-energy behavior of the “S-matrix” that characterizes scattering.

Observables in QG appear to be associated with a so-called holographic screen on the boundary of spacetime. ¹ The name derives from an analogy with real holograms in that one may view the bulk of spacetime as being exactly encodable, with no loss of information, onto a surface one dimension lower.

Theoretical investigations beginning in the mid-1990s have confirmed the holographic idea in some detail. ² The string-duality revolution established a nonperturbative confirmation that S-matrices exist for a large class of string models of gravity in asymptotically flat spacetimes. (For more on strings and duality see the story in Physics Today, August 1998, page 20 , and two Physics Today articles by Edward Witten: April 1996, page 24 , and May 1997, page 28 .) That confirmation was the first conclusive evidence for models of quantum gravity that are consistent with all of the principles of quantum mechanics. A subset of those consistent models yielded a so-called matrix theory—a complete Hamiltonian formulation in terms of limits of quantum field theories with matrix-valued “coordinate” fields. ³ The fields do not live in spacetime, as do those of EFT. Rather, the space of fields is identified with the spacetime’s asymptotic boundary.

Holography and black holes

A particular class of spacetimes that is not asymptotically flat yields a clear connection between black holes and the cosmological constant. Those spacetimes asymptotically approach anti de Sitter (AdS) space, the most symmetric solution of Einstein’s equations with no matter, but with a negative cosmological constant. AdS space is not a realistic model of the universe we live in, but its mathematical description is much more tractable. In AdS space, quantum gravity can be described by a quantum field theory living on the spacetime boundary: As shown in figure 2, that boundary is the screen on which the hologram is projected.

Before discussing the black hole–CC connection, I briefly review some properties of black holes. The Schwarzschild radius is the radius beyond which an external observer cannot see into a black hole. It is always larger than the Planck length and grows with the mass or energy of the black hole. In QG, the high-energy, small-impact-parameter scattering meant to probe short distances will instead produce large black holes. Thus, there is no such thing as a short distance in a quantum theory of gravity. The connection between high energy and short distances, characteristic of EFT, fails when energies approach the Planck energy E _P and is replaced by a connection between high energies and large distances. And a quantum theory of gravity cannot have localizable observables. Nothing, when probed with an energy much greater than E _P, can be localized to within an accuracy better than a Schwarzschild radius. In QG, locality is an approximate notion, applicable at energies below the Planck energy.

The prediction that scattering meant to probe small distances will instead produce black holes that are only approximately localizable is based on classical general relativity. Why should one believe it is a true feature of a quantum theory of gravity? On dimensional grounds, the spacetime curvature outside the Schwarzschild radius of a black hole is inversely proportional to the square of the Schwarzschild radius. Thus, the curvature gets smaller—spacetime gets flatter—as the energy of the black hole gets larger. String theory confirms the expectations of EFT that, in such low curvature regions, general relativity is a good approximation to QG. That is not to say the entire scattering process is well described by relativity. However, the average, thermodynamic properties of the black hole creation and decay processes seem to be well described, in the limit of large black holes, by general relativity and EFT. Physicists using a rigorous mathematical model of QG in asymptotically AdS space have been able to verify many of the predictions Stephen Hawking made based on his semiclassical treatment of black holes.

In particular, theorists have identified, ⁴ for a class of black holes, the microscopic quantum states that are responsible for the black hole entropy formula guessed by Jacob Bekenstein and Hawking: Black hole entropy is one-quarter of the area (in the natural units in which Newton’s and Planck’s constants and the speed of light are all 1) of the black hole event horizon, the surface defined by the Schwarzschild radius. ⁵ The Bekenstein–Hawking formula suggests that black holes result from trying to pack the maximal amount of information into a given region of spacetime. More bits will cost more energy and take up more space.

Entropy and the cosmological constant

The Bekenstein–Hawking entropy formula and its rigorous verification (along with other black hole properties) in mathematically precise models of quantum gravity have allowed physicists to gain new insight into the cosmological constant problem. In asymptotically AdS spacetime, the formula predicts that large black holes are stable and that the density of black hole states ρ at large energy E is given (in four spacetime dimensions) by $$\rho \left(E\right)~exp\left[2^{2/3}\pi {\left(E/E_p\right)}^{2/3}{\left(L{E}_p\right)}^{4/3}\right],$$ where L is the radius of curvature of the asymptotic AdS space. The radius of curvature can be expressed in terms of the cosmological constant Λ via L = (8π/3)(–Λ/E_P ²)^{− $\frac{1}{2}$} .

The quantity L, moreover, is a discrete input parameter that partially characterizes a particular model of quantum gravity and determines the high-energy density of states of the model. That high energies are intimately connected with the large distance asymptotics of spacetime is easily understood in terms of the properties of black holes—the Schwarzschild radius of a black hole goes to infinity with its energy.

The meanings of the CC in effective field theory and in quantum gravity in an asymptotically AdS spacetime are in stark contrast. In EFT, the CC is a parameter in the long-wavelength theory. It appears to be a calculable quantity, whose value combines contributions from both low and high energies. The definition deriving from QG is a high-energy input; it cannot be affected by low-energy dynamics. Given that insight, it is no longer surprising that one cannot calculate the CC using the low-energy EFT approximation to QG. All one can do in EFT is to fine-tune the CC to reproduce the correct high-energy input. Note that the CC is a high-energy input parameter even though the corresponding energy scale is very small.

The necessity to fine tune in EFT seems mysterious if one supposes that long distances correspond to low energy. But theorists have learned that such is not the case in QG.

When the CC vanishes, the entire structure of the QG theory changes, because the asymptotic boundary of the corresponding curvature-free “Minkowski space” is very different from AdS space. Observables of asymptotically flat space are encoded in an S-matrix, rather than in field-theory correlation functions. The logarithm of the density of black hole states in asymptotically flat spacetime grows like a power of the energy greater than (rather than less than) one. That means the specific heat of the ensemble of black hole states is negative—the black hole ensemble is unstable, and decays by Hawking radiation (see figure 3).

From the point of view of EFT, the difference between asymptotically flat and AdS spaces appears minor. EFT physics is quasilocal and all effects of the curvature of AdS space on local physics vanish like a power of the curvature. EFT intuition leads one to expect that the heart of a theory does not depend on the boundary conditions, which are viewed as perturbations of a small number of degrees of freedom of the theory. Black hole physics and the rigorous results of string theory show that intuition to be wrong.

Positive cosmological constant

A cynic would be happy to note that, despite the progress I have described, physicists have no rigorous knowledge of quantum gravity for the case of a positive cosmological constant—the case that appears to be realized in our world. However, experience with AdS space has taught theorists to trust the clues that semiclassical black hole physics gives about the nature of the quantum theory. Those clues, in particular, may be applied to the most symmetric solution of Einstein’s equations with positive cosmological constant and no matter—de Sitter space.

The d-dimensional dS space is the surface (x ⁰)² – x ² = −R ² in d + 1 dimensional Minkowski space. If one looks at the spatial sections of dS space obtained by specifying fixed values of time, then one sees spheres that contract from an infinite radius to a minimum radius R and then reexpand as time runs from −∞ to 0 to +∞. The minimum radius may be simply expressed in terms of the cosmological constant: R = (Λ/E _P ²)^{− $\frac{1}{2}$} .

No observer can see all of dS space. The region on which any observer can make measurements, called the causal patch, has a finite spatial extent of radius R. As with the region outside of a black hole, one can describe the causal patch by a time-independent metric. The analogy between dS space and a spacetime containing a black hole goes further—a causal-patch observer sees a temperature and entropy that can be attributed to an interaction with states that have fallen into the “cosmological horizon” at radius R. Moreover, like an observer outside a black hole’s event horizon, a causal-patch observer in dS space never sees anything move beyond the horizon. When one formulates EFT in a causal patch, one finds a conserved Hamiltonian and sees that the system is in the canonical thermal state of the static Hamiltonian. The states of the system are divided into those that are localizable within a given causal patch and those that are not. EFT describes only localizable states; the thermal state arises because of the interactions between localizable and nonlocalizable states. From an observer’s point of view, the nonlocalizable states appear to reside on the horizon.

The finite area of that horizon provides another clue about the nature of quantum gravity in de Sitter space. Gary Gibbons and Hawking suggested that one-quarter of the horizon area is the entropy of empty dS space. ⁶ When a black hole is present in dS space, there are two horizons. One is associated with the black hole, and the other is a version of the cosmological horizon, distorted by the presence of the black hole. The sum of their areas is always less than the area of the cosmological horizon of empty dS space. Note that because a black hole’s area grows with its mass, the sum rule implies a maximum mass for dS black holes, as is indeed confirmed by rigorous analysis.

The thermodynamic interpretation of the clues provided by horizon areas is that the entropy of empty dS space is essentially the logarithm of the number of the very-low-energy, nonlocalizable states. Large localized objects in dS space reduce the entropy compared to that of empty space. To make a large black hole in the bulk, the degrees of freedom on the horizon must be frozen into a very special state. Because the energy spectrum is bounded from above and the thermal entropy is finite, the quantum system representing dS space is finite dimensional. Willy Fischler (in his 2000 talk “Taking de Sitter Seriously,” given at the symposium celebrating Geoffrey West’s 60th birthday) and I independently suggested that the cosmological constant is determined by that finite number of states; in particular, the constant is once again an input parameter of the theory. ⁷ (For further discussion of finite quantum systems, see the box on page 50.)

A positive cosmological constant, like a negative one, is a discrete variable, and different values of that variable describe different quantum systems. In the positive case, different CCs correspond to systems with different numbers of states.

The landscape of string theory

Asymptotically de Sitter space does not fit into the framework of string theory as it is currently conceived. String theory is a mathematically precise algorithm for computing S-matrix elements (or other kinds of boundary observables) in infinite spacetimes. In my view, the connection between the theory of the real, asymptotically dS world and string theory is obtained by taking the mathematical limit in which the adjustable cosmological constant approaches zero. That limiting model will fit into the framework of string theory. The small empirical value of the cosmological constant suggests that purely string-theoretic calculations might give good approximations to some quantities in the real world.

An alternate string-theory approach to explaining the accelerating universe adheres much more closely to the EFT philosophy. One takes the low-energy EFT describing the scattering of the massless fields in string theory formulated in 10-dimensional asymptotically flat spacetime. Then one searches for solutions of the field equations that have only four noncompact spacetime dimensions. Those solutions correspond to minima of an effective potential that is a function of the fields in the EFT as sketched in figure 4. Solutions corresponding to 4-dimensional AdS or dS space exist, as do solutions corresponding to a 4D, homogeneous, and isotropic cosmology with energy density coming from the variation of the inverse string coupling and the radii of extra dimensions as they go to infinity. I call those the Dine–Seiberg cosmologies, in honor of the two physicists who pointed out that the EFT derived from string theory was likely to have solutions of this type. ⁸ Like almost all cosmological solutions, the Dine–Seiberg cosmologies have a Big Bang singularity, in which the equations break down.

The proponents of the EFT-inspired approach argue that the values of the inverse string coupling and the radii of the compact dimensions are large at many of the effective-potential minima that correspond to dS space, and so those spacetimes should be good approximations to true quantum models of gravity. Those physicists posit that the real world corresponds to one of those metastable dS minima and will, on a time scale much longer than 10 billion years, decay into the Dine–Seiberg cosmology. Because the latter spacetime has infinite spatial extent, it should have some version of an S-matrix, a mathematically precise set of observables.

The EFT effective potential has a huge number of metastable dS minima—more than e ¹⁰⁰ of them. Leonard Susskind has dubbed that collections of solutions “the landscape of string theory.” ⁹ The richness of the landscape is supposed to resolve the problem that the CC in most of the dS minima is much too large to fit the data of our universe.

The anthropic principle

There is a perfectly sensible way to state the anthropic principle. Suppose one has convinced oneself that mathematical consistency alone cannot rule out any of a collection of candidate physical theories. Or suppose, as in the context of the string-theory landscape, that a great number of solutions are mathematically consistent. As physicists, we are instructed to supply data from the real world to pin down the right theory (or solution). Might it not be that the single complicated piece of data—that there exists a carbon-based life form with intelligence—could constrain our search through the high-dimensional solution space much more efficiently than a huge set of experiments?

Consider, for example, the cosmological constant. John Barrow and Frank Tipler long ago suggested that the unnaturally small value of the CC might be explained by anthropic arguments. Andre Linde introduced the anthropic point of view into modern inflationary theories in 1982. Steven Weinberg did the crucial calculation showing that if all other parameters have their observed values, then galaxies cannot form unless the CC is bounded by something close to its observed value. ¹⁰ A large enough positive CC makes the universe expand too rapidly for matter to clump into galaxies. So, because galaxies are necessary if there are to be stars and life, the existence of life chooses those models, or states of a given model, that have a CC satisfying Weinberg’s bound. Further refinements of the bound give a CC close to that which would account for cosmological data.

A number of other facts about parameters in current models of nature can be explained by assuming that those parameters are random numbers subject to the constraint that life can exist. The anthropic principle, however, is not nearly strong enough to fix everything close to its real value. For example, the properties of stars and galaxies depend primarily on energy scales up to the MeV scale of nuclear physics. The standard model of elementary particles has been tested to hundreds of thousands of times higher than that scale. Many variations of the standard model would give rise to nuclear physics essentially identical to what we see, but they are ruled out by experiment. It is a daunting task to examine all of the states in the landscape to see whether most of those allowed by anthropic reasoning actually have the right physics in the 100–1000 GeV regime characteristic of the weak interactions. There is no reason to believe that will be the case.

Many researchers in string theory are enthusiastically working on the consequences of the landscape. My own point of view is more pessimistic. I think more work has to be done to establish that the states described by the landscape really correspond to a well-defined model of quantum gravity. I also think that even if they do, they are likely to lead to many predictions that disagree with experiment.

By contrast, in the hypothetical theory of a stable dS space, everything appears to be determined by a single parameter, the CC. The anthropic determination of that parameter by Weinberg and others does agree with observation. The anthropic determination of the CC does not resolve the problem that, in EFT models, the scale of supersymmetry breaking is related to the CC in a manner incompatible with observation. I have proposed that large quantum fluctuations to the relation, analogous to the fluctuation corrections to critical exponents in second-order phase transitions, could predict supersymmetric partner particles at 1000 GeV for the observed value of the CC. The arguments though, are both technical and speculative.

The questions of the existence of viable landscape states on the one hand and stable dS space with a finite number of quantum states on the other are logically independent. As far as physicists currently understand, either or both of them could be mathematically valid. My own bet is that stable asymptotically dS space exists and it is more likely than landscape states to lead to predictions that agree with experiment.

The landscape of string theory and stable dS space are only two of the current ideas addressing the CC problem. The majority view in the field is probably that neither those nor any other published ideas about the cosmological constant are correct. In that view, explaining the value of the cosmological constant remains the outstanding problem of theoretical physics.