Saul Perlmutter announced the Supernova Cosmology Project's evidence for a cosmological constant at the annual meeting of the American Astronomical Society in Washington, D.C., on January 8, 1998. On February 18 of that year, Gerson Goldhaber and Perlmutter discussed the SCP evidence at the UCLA conference on Dark Matter in Los Angeles, where Alexei Fillipenko announced similar results from the High-Z Supernova Search Team.

What they had observed was the accelerating expansion of the universe, presumably caused by Einstein's cosmological constant (lambda). Initially a purely mathematical term in the equations of General Relativity — which Einstein later dropped — theorists by the end of the 20th century had come to regard the cosmological constant as a manifestation of the vacuum energy described by quantum mechanics.

Yet a straightforward formulation indicates that the vacuum energy is hundreds of orders of magnitude too powerful to account for observed cosmic acceleration. Thus acceleration was, as theorist Frank Wilczek, then at the Institute for Advanced Study in Princeton, put it, "maybe the most fundamentally mysterious thing in basic science."

At a meeting on the accelerating universe held May 1-3, 1998, at Fermilab, where the Supernova Cosmology Project and the High-Z Supernova Search Team presented their results and, according to cosmologist Michael Turner, "performed brilliantly," Turner propose a more inclusive term than the cosmological constant and showed the slide on the right. Nobody liked it, but he soon came up with one that stuck.

Within weeks a raft of theoretical alternatives to the cosmological constant had appeared, some invoking a contra-gravitational energy that changes over time, others phenomena like tears in the fabric of spacetime, and some even questioning the validity of General Relativity itself.

Seeing the need for an inclusive term, cosmologist Michael Turner of the University of Chicago came up with "funny energy." But, he said, "focus groups" — his colleagues, that is — "didn't like the name." He tried again, and this time came up with the one that stuck: dark energy.

The SCP's Goldhaber first glimpsed evidence for acceleration when, during the summer of 1997, he plotted the brightness and redshift of over three dozen Type Ia supernovae discovered by the SCP. His graph indicated that the more distant supernovae were dimmer than their redshifts would suggest if expansion were slowing down — or even if expansion were uniform.

"Gerson goes to our data and looks at 38 SCP SNe," read the minutes of the SCP meeting held September 24, 1997, at Berkeley Lab. "He has a hubble diagram, and made a histogram of Omega_M in the flat universe. It comes out peaking near 0.2...." As a measure of the proportion of mass density in a flat universe, 0.2 is low. It implies that most of the density of the universe isn't mass at all.

The most direct explanation was a positive value for omega_lambda, the proportion of Einstein's cosmological constant. Sometime later that fall, Adam Riess of the High-Z Team, working nearby on the campus of UC Berkeley, did a similar calculation with a smaller number of supernovae, which pointed to a similar conclusion.

One caveat, however, as Goldhaber emphasized: "It should be noted that a flat universe was an assumption on our part at that point in time (1997) based on inflation theory and on early CMB data" — that is, data hinting at fluctuations in the cosmic microwave background. "Measurements of the fluctuations in the CMB clearly demonstrating a flat universe came later."

By the first months of 1998 both teams were confident that their supernova measurements established the probability of a positive omega_lambda to better than 99 percent — even without knowing whether the universe is flat. Supporting evidence that the universe really is flat was not long in coming.

Evidence from the CMB

The CMB, or cosmic microwave background, dates to a time less than 400,000 years after the big bang when the hot early universe — until then an opaque soup of subatomic particles — had cooled and abruptly become transparent. Suddenly electrons could join ("recombine") with nucleons to form hydrogen atoms, leaving photons to move freely through space. Over time the photon energy cooled — wavelengths stretched and frequencies fell — to today's chilly 2.73 degrees Kelvin, not much above absolute zero. This faint signal, detectable at millimeter wavelengths, is the cosmic microwave background.

The BOOMERANG flight that circled the South Pole in January, 1999 produced the most detailed map until that time of fluctuations in the cosmic microwave background. Within weeks MAXIMA produced similar results for the northern sky. Resolution of both maps was sufficient to show that the universe is flat.

At the moment of recombination, the liquid-like universe was vibrating like a bell from pressure variations traveling through it; these were subsequently preserved as minute fluctuations in the temperature of the CMB, which record details about the history and geometry of the universe. ("Wrinkles in time," Berkeley Lab astrophysicist George Smoot called them; in 2006 Smoot won the Nobel Prize for his pioneering studies of CMB anisotropy in the early 1990s, using the COBE satellite.) Theory said that if the average size of these hot and cold spots averaged about one degree across, it would indicate that the universe is flat.

In January 1999, a year after the SCP announced supernova evidence for a cosmological constant, a balloon-borne mission called BOOMERANG circled the South Pole and mapped a wide swath of sky. "From the dataset, the BOOMERANG team was able to make the most detailed map of the CMB's temperature fluctuations ever seen," says Berkeley Lab astrophysicist Julian Borrill, a leader in analyzing the BOOMERANG data.

The results were announced in the spring of 2000: "According to the BOOMERANG data, the universe is cosmologically flat." Combined with the supernova evidence, the data suggested that at most a third of the universe is matter (visible and dark). The rest is dark energy.

Within weeks of the BOOMERANG results, independent confirmation came from another balloon-borne CMB study called MAXIMA, which mapped a smaller strip of sky over North America. Much as the High-Z Team's measurements confirmed what the SCP had found for supernovae, MAXIMA confirmed BOOMERANG's conclusions that the CMB showed the universe to be flat.

Uncertainties, statistical and systematic

For the supernova observers, the most pressing concern was to reduce uncertainty about acceleration. They needed to collect more Type Ia supernovae for a larger statistical sample, and they needed to understand the physical factors that might cause brightness to vary.

In addition to variables like extinction, K-correction, Malmquist bias, and gravitational lensing, characteristics of Type Ia supernovae themselves such as their chemical composition, or "metallicity," might give rise to potential differences in brightness (see Part I of this series for details).

One possible source of uncertainty is asymmetry. Type Ia supernova are thought to result when a white dwarf star accretes matter from an orbiting companion until it reaches critical mass, igniting a thermonuclear explosion. A binary system with an accretion disk is inherently asymmetrical, and even uniformly bright explosions might look different depending on the viewing angle — if, for example, an accretion disk or companion star were to mask the explosion. 

The best way to detect asymmetry in a distant object is by measuring its polarization, a method pioneered at the University of Texas's McDonald Observatory in the mid-1990s by Lifan Wang, who joined the Supernova Cosmology Project in 2000. The light from a supernova is randomly polarized, and a spherical star or explosion shows no net polarization at all. But if the explosion is not spherical, there will be more vertically polarized light in the long dimension than in the short one. Thus an aspherical explosion has net polarization.

Wang found net polarization in every kind of supernova he looked at except Type Ia's — until finally, in 2001, he was able to use the European Southern Observatory's Very Large Telescope in Chile to detect net polarization in a nearby Type Ia supernova designated SN 2001el. Far from this asymmetry weakening the status of Type Ia supernovae as dependable standard candles, says Wang, "They may be even more uniform than we thought. If all Type Ia supernovae are like this, it would account for a lot of the dispersion in brightness measurements."

The first Type Ia supernova found to have net polarization was SN 2001el, observed with the European Southern Observatory's Very Large Telescope. SN 2001el's spectral and polarimetric data were closely reproduced by an asymmetric model in which a calcium-rich clump of material — perhaps a companion star — was positioned in front of the explosion.

The SCP's Peter Nugent and Dan Kasen, then a graduate student doing his research at Berkeley Lab, modeled SN 2001el on a supercomputer at DOE's National Energy Research Scientific Computing Center (NERSC). They came close to matching Wang's data by assuming a clump of calcium-rich material between the exploding white dwarf and the observer — a clump that could even be the dwarf's companion star itself. Says Kasen, "People tend to forget about this, but in a binary system an explosion is going to run into the companion in just a few minutes."

Searching near and far

Reducing uncertainties ultimately depends on more observations — both of "nearby" supernovae and the most distant ones. The international Nearby Supernova Factory (SNfactory), led by SCP astronomer Greg Aldering, became operational in 2002 and within a year had discovered 34 supernovae, the most impressive debut ever for a supernova search. The SNfactory's goal is to establish itself as the premier source for calibrating Type Ia brightness, and eliminating systematic and statistical uncertainties, by obtaining detailed spectra and light curves for more than 300 low-redshift Type Ia supernovae.  

Meanwhile the SCP had launched a search for supernovae at redshifts greater than z = 1. The goal was to look back to a time before the universe began accelerating, when matter was still so densely packed that its mutual gravitational attraction was stronger than dark energy. (Although expansion was decelerating then, continued expansion would eventually dilute the density of matter, and acceleration would take over).

In the era of deceleration unobscured Type Ia supernovae ought to appear brighter, not dimmer, than their redshifts would otherwise suggest. If this was indeed the case it would be a sure sign of subsequent acceleration, reassurance that variables like gray dust or evolution had not skewed the measurements on which the discovery of dark energy was based.

The farthest supernovae indeed confirmed acceleration: one early discovery was a supernova at z = 1.2, still the highest redshift at which a Type Ia supernova has been spectroscopically confirmed. Analysis by members of both the SCP and High-Z teams — using data found by serendipity in Hubble Space Telescope records — established that an even more distant supernova designated SN 1997ff was almost certainly a Type Ia at a redshift of z = 1.7; it could only have been found with a telescope in space. "The results from SN 1997ff are one of the best arguments for the SNAP satellite," Nugent says.

SNAP, the SuperNova/Acceleration Probe, is an orbiting telescope conceived in 1999 by members of the SCP and their colleagues; it is the leading representative of a wave of future dark energy studies.

SNAP inspires a dark energy mission

At its core, learning the nature of dark energy requires determining whether the pulling-apart now acting against gravity — what SCP theorist Eric Linder has called the "springiness" of the universe — has been constant over time or has varied, and if it has varied, when and how much.

To do this requires fine-scale measurement of differences, if any, in the expansion rate of the universe over time. Supernovae are one way to approach the problem. Measuring the history and magnitude of variations in the distribution of matter, both visible and dark, is another.

	The proposed SuperNova/Acceleration Probe (SNAP) satellite inspired DOE and NASA's Joint Dark Energy Mission. It will find and measure thousands of Type Ia supernovae and will measure the distribution of matter in the universe through weak gravitational lensing.

SNAP, an international project led by Saul Perlmutter and Michael Levi at Berkeley Lab, will orbit a 2-meter-class reflecting telescope fitted with a half-billion-pixel imager that incorporates revolutionary new red-sensitive CCDs developed at Berkeley Lab, plus highly efficient infrared sensors. A spectrometer system will record accurate and consistent spectra, from the near ultraviolet to the near infrared, of every new supernova. By repeatedly imaging several large patches of sky, SNAP will discover and analyze 2,000 type Ia supernovae in a single year, 20 times the number from a decade of ground-based search. Many of the new supernovae will be at distances and redshifts far greater than any yet found.

In May of 2006, the Dark Energy Task Force (DETF) of NASA, DOE, and the National Science Foundation reported that different techniques for measuring dark energy in combination "have substantially more statistical power, much more ability to discriminate among dark energy models, and more robustness to systematic errors than any single technique." DETF called the use of Type Ia supernovae "presently the most powerful and best proven technique," but pointed to a promising complementary technique, potentially even more powerful, called weak gravitational lensing.

The SuperNova/Acceleration Probe does not depend on supernova measurements alone; weak lensing had been incorporated into its design from the beginning. SNAP will make a high-resolution map of the sky covering an area 2,000,000 times larger than the Hubble Deep Field, sensitive to minute distortions in the shapes of distant galaxies as their light passes through uneven distributions of matter. The goal is to measure the distribution of dark matter in the universe and determine the effect of dark energy on the growth structure of the universe over time.

From the beginning, intensive research and development of the SNAP concept have been based at Berkeley Lab and supported by the Department of Energy's Office of Science. UC Berkeley's Space Sciences Laboratory has been a major partner as the SNAP collaboration swiftly grew to include representatives of numerous U.S. and foreign government agencies and academic institutions. NASA added its support to SNAP in 2003, when NASA joined with DOE to pursue a Joint Dark Energy Mission (JDEM).

NASA incorporated JDEM into its Beyond Einstein program in 2004, and in September, 2007, the National Research Council's Beyond Einstein Program Assessment Committee, acting at the request of NASA and DOE, recommended that JDEM be the first of the Beyond Einstein cosmology missions to be developed and launched.

Although SNAP inspired JDEM, competing proposals have emerged, including ADEPT, the Advanced Dark Energy Physics Telescope, led by Charles Bennett of Johns Hopkins University, and Destiny, the Dark Energy Space Telescope, led by Tod Lauer of the National Optical Astronomy Observatory. Like SNAP, these missions would use more than one measuring technique. NASA and DOE must now jointly choose among the contenders.

The most promising of the remaining techniques for measuring dark energy is the "baryon acoustic oscillations" method, actively being pursued by David Schlegel and Nikhil Padmanabhan of Berkeley Lab's Physics Division using the telescope of the Sloan Digital Sky Survey. Baryon acoustic oscillations ("baryon" is shorthand for ordinary matter) are variations in the large-scale distribution of the galaxies which descend directly from temperature variations in the CMB. Looking far back in time, these "oscillations" are seen to be regular, forming a natural ruler with which to measure dark energy's influence on the evolution of the universe.

All these techniques — supernovae, weak lensing, baryon oscillation, and others being developed — aim to measure the "springiness" of dark energy, its equation-of-state parameter, written w. If dark energy is the cosmological constant, the value of w is minus one, corresponding to "negative pressure" — while ordinary radiation or mass have zero or positive pressure.

In opposition to the mutual gravitational attraction of mass, dark energy is pulling space apart. Various theoretical ideas can be represented by scalar fields, like fields of springs covering every point in space. If dark energy is the cosmological constant, each spring would be the same length and motionless. If dark energy is quintessence or some other dynamic entity, each spring would be stretched to a different length.

But there are many competing ideas for what dark energy is or isn't. "Quintessence" posits an unknown kind of matter that, fluid-like, fills the universe and has negative gravitational mass. Or there could be a time-varying form of dark energy that only temporarily mimics lambda; in the future it might bring acceleration to a halt or even superaccelerate it. "Topological defect" models attribute dark energy to defects created as the early universe cooled. Others picture our universe afloat in a higher-dimensional brane-world, with gravity free to interact and vary among the dimensions.

Unlike the cosmological constant, in competing models of dark energy w would not equal minus one — "springiness" would not scale uniformly as the universe expands but vary with the passage of time.

"One of the goals of the SuperNova/Acceleration Probe satellite is to determine whether w may be changing with time," says Perlmutter. "This will help us narrow the possibilities for the nature of dark energy. That's an exciting prospect for physicists, because understanding dark energy will be crucial to finding a final, unified picture of physics."

As with its discovery, what happens next in the pursuit of dark energy depends on many factors — social and political as well as scientific. The most challenging cosmological mystery since Hubble discovered that the universe is expanding will not likely be resolved quickly.

Additional information

• "Dark energy: the decade ahead," by Saul Perlmutter and Eric Linder, a preview of research made possible by new observing techniques, appears in the December, 2007 issue of Physics World.
• More about CMB results from BOOMERANG and analyzing the BOOMERANG data at NERSC
• More about MAXIMA
• More about measuring supernova asymmetry
• More about the Supernova Cosmology Project
• More about the Nearby Supernova Factory and the SNfactory's rookie year
• More about SNAP, the SuperNova/Acceleration Probe
• More about SN 1997ff, a supernova from the era of deceleration
• More about alternative theories of dark energy
• Learn more about the NASA/DOE/NSF Dark Energy Task Force (DETF)
  and read the DETF report
• Learn more about the National Research Council's Beyond Einstein Program Assessment Committee (BEPAC) and read the BEPAC report
• More about weak gravitational lensing, from Wikipedia
• More about baryon acoustic oscillations and the Sloan Digital Sky Survey
• Read Dark Energy's 10th Anniversary, Part I and Part II.