To celebrate the 10th anniversary of the discovery of dark energy, Science@Berkeley Lab presents a capsule history, in two parts, of the Supernova Cosmology Project's pioneering efforts to measure the expansion rate of the universe using Type Ia supernovae as standard candles. A subsequent issue will look at new proposals for studying the nature of dark energy.

Saul Perlmutter, leader of the international Supernova Cosmology Project (SCP) based at Berkeley Lab, made the first public announcement of evidence for the accelerating expansion of the universe on January 8, 1998, at a press conference sponsored by the American Astronomical Society (AAS), during the society's annual meeting in Washington, D.C.

This Supernova Cosmology Project data was presented to the scientific community and the public in January, 1998, indicating the existence of a cosmological constant and a positive value for lambda (Λ) sufficient to cause the accelerating expansion of the universe.

Perlmutter presented the SCP's hard-won data pointing to a "cosmological constant" — a term introduced into the equations of general relativity by Albert Einstein but later rejected by him as a mistake. Over the course of the decade 1988 to 1998, using novel methods in the face of intense initial skepticism, the SCP had gathered evidence that showed, to the surprise of most scientists including themselves, that there was something else in the universe besides mass and ordinary energy, and that Einstein had not been mistaken after all.

This mysterious something would soon be given the more inclusive name dark energy. But in January 1998, coming as it did in the last years of a century that had been filled with unsettling discoveries — including the existence of innumerable "island universes" (galaxies) besides the Milky Way, a universe undergoing rapid expansion, and its origin in a big bang — evidence for a cosmological constant was as profound and startling as anything that had gone before.

Science writer Charles Petit was at the press conference and was first to report the news, in a front-page article in the San Francisco Chronicle the next morning, January 9:

Call it the runaway universe, and we are on board.
     A new, detailed study of exploding stars and immense galaxy clusters billions of light years away seems to indicate that the universe not only will expand forever, contrary to some theories, but that its outward expansion is starting to speed up....
     "If these data hold up, then we can know we are in a universe that will exist forever," said Saul Perlmutter. "And we seem to be seeing, for the first time, direct evidence that the cosmological constant is real."

Three days later Petit reported on the AAS meeting for the weekly newsmagazine US News & World Report:

Strong signs that the expanding universe is actually starting to speed up come from astronomers using ground-based and orbiting telescopes to chart supernova exploding stars in galaxies, a gauge of how fast the universe has expanded so far. Says Saul Perlmutter of the Lawrence Berkeley Laboratory, a leader of the effort: "No big crunch. The galaxies out there seem to be accelerating outward. Not much, but they are not turning around, either."

James Glanz reported on the announcement in the January 30 issue of Science:

At the meeting of the American Astronomical Society in Washington, D.C., earlier this month, Saul Perlmutter of Lawrence Berkeley National Laboratory in Berkeley, California, announced that he and an international team of observers have now studied a total of 40 far-off supernovae, using them as beacons to judge how the cosmic expansion rate has changed over time. Not only did the results support the earlier evidence that the expansion rate has slowed too little for gravity ever to bring it to a stop; they also hinted that something is nudging the expansion along. If they hold up, says Perlmutter, "that would introduce important evidence that there is a cosmological constant."

Perlmutter's presentation at the January 8 press conference was deliberately conservative, aimed as much at scientists as at reporters, and although Princeton theoretical astrophysicist David Spergel was on hand to give a follow-up explanation of what the cosmological constant is and why the results were important, not all reporters were quick to grasp the implications.

"For the physicists, this notion of a cosmological constant was a shocking result," Perlmutter said recently, "and we felt it was crucial that scientists would understand every caveat in our announcement — since we thought they would have only our data to depend on in reaching their own conclusions about whether there was a cosmological constant. We wanted the results to be presented in the press as thoroughly considered, sober-minded, which was perhaps not the best way to capture the press's attention. Nevertheless, numerous queries immediately began coming in from around the world."

A month later attention intensified when other reporters did catch the drift. In February, at a UCLA symposium on dark matter held at Marina Del Rey in California, the Supernova Cosmology Project presented their results again, and the High-Z Supernova Search Team (HZT) — formed late in 1994 to compete with the SCP's use of Type Ia supernovae for cosmological measurements — followed with the first presentation of their own results. It was clear that both groups had reached the same conclusion: the expansion of the universe is accelerating.

Early in May, Fermilab convened a major scientific workshop to examine the startling findings. Both teams again presented their data and described the cross-checks they'd performed. A panel searched for possible holes in the evidence, followed by a straw pole in which, as John Noble Wilford reported in the New York Times, "a show of hands indicated that most scientists now agree, by a vote of two to one, that the supernova astronomers have made a strong case for an accelerating universe and thus for something like the missing energy."

As the 10-year anniversary of the first public announcement of evidence for what is now called dark energy approaches, some individuals have applied a narrow definition of the word "public" to dispute who was first with the news of cosmic acceleration. Science writer Charles Petit, who was at the AAS press conference and wrote the first newspaper article, has no doubt it was Perlmutter: "It was in the SF Chronicle the next morning, on the front page. If that's not public, what is?"

Prelude to discovery

When the 20th century began, few people imagined that the universe was anything but a small, unchanging place. That's why, when Albert Einstein introduced his general theory of relativity in 1917, he added a term to the equations to make sure they described a universe that was just barely static. He called it the cosmological constant and used Λ, the Greek letter lambda, to symbolize it.

	On a visit to the Mount Wilson Observatory near Pasadena in 1931, Albert Einstein demonstrated his calculation of the density of the Milky Way. By using Mount Wilson's 100-inch telescope Edwin Hubble had recently shown that the universe is expanding.

By the 1930s a completely different picture had emerged from observations with big telescopes. The universe was vast, and innumerable galaxies were moving outward in every direction; many believed that space and time and all they contained had originated in an incredibly hot, dense instant, later called the big bang. Faced with this evidence of a dynamic universe, Einstein berated himself for introducing a term that now seemed unnecessary, and cheerfully abandoned his cosmological constant.

For the next 50 years most scientists assumed that the mutual gravitational attraction of all the stars and galaxies and other matter in the universe would continually slow its expansion, and perhaps eventually bring it to a halt; perhaps expansion might even reverse, and the universe would begin to contract.

How could one begin to investigate such a question as the ultimate fate of the universe? Nobody even knew the universe's shape or dimensions. For decades astronomers had been hoping to find a way to use brightness to measure distance. If a regular relationship between a particular kind of astronomical object's brightness and its distance could be found, such objects could be used as "standard candles."

If the distance to a standard candle can be determined from its brightness, how much the universe expanded in the time it took light to cover that distance can be determined from the object's redshift. Redshift (symbolized by z) refers to the shift toward lower frequencies of lines in an object's spectrum: stretching light lowers the frequencies of its spectral features, a direct measure of the stretching of the space through which it travels.

Cepheid variable stars were one kind of standard candle, useful for measuring distances to relatively nearby galaxies. But no standard candles were known that were bright enough to measure the expansion history of the universe. They couldn't be measured far enough away — far enough back in time — to determine how much the universe's expansion was (presumably) slowing down.

There was one candidate for a bright standard candle, however. German astronomer Walter Baade was working at Mount Wilson Observatory in the 1930s with Fritz Zwicky when he first used the term supernova (SN) to describe the brightest of exploding stars. Baade proposed that supernovae (SNe) could be used as astronomical standard candles to make measurements at greater distances and higher redshifts than were possible with standard candles like Cepheid variables — far enough away to allow the study of changes in the rate at which the universe was expanding.

The more astronomers studied supernovae, however, the more complicated and various they appeared. As a group, supernovae weren't very standard after all. It wasn't until the mid-1980s, 50 years after supernovae were named, that astronomers once again saw the possibility of measuring the universe's expansion history by means of supernovae.

A standard among variables

First came the recognition that not all supernovae were physically the same. As early as 1941 supernovae had been divided into two types, rather unimaginatively called Type I and Type II: hydrogen emission lines were lacking in the spectra of Type I but were present in the spectra of Type II. Neither type made very good standard candles.

By 1985, however, it had become clear that Type I supernovae included a mixture of distinguishable subtypes, designated Ia, Ib, and Ic. Unlike the others, Type Ia supernovae (recognized by a silicon absorption feature) were remarkably similar: all were very bright and had similar spectra and light curves. Soon Type Ia supernovae would prove to be not only the brightest but the most dependable standard candles ever discovered.

Another major step forward in the mid-1980s came when physicist Richard Muller and his group at Berkeley Lab and UC Berkeley built and operated the first successful, completely automated supernova search. Supernovae are relatively rare events; they explode only a few times per century in a typical galaxy (and, in a typical galaxy, Type Ia supernovae explode only a few times in a thousand years). Thus the longstanding interest in automating the supernova search.

Astrophysicist Stirling Colgate had attempted a robotic supernova search in the late 1970s, but the technology available at the time had not been up to the task. Now, using a robotic telescope equipped with then-new CCD detectors instead of photographic plates, plus the latest computers, Muller's group discovered 20 supernovae, thus demonstrating that automatic image processing could identify new supernova explosions in images of galaxies.

Two different groups saw the time as ripe for the development of new searches for Type Ia supernovae at greater distances than ever before. A Danish team led by Hans Ulrik Norgaard-Nielsen started an intensive effort in 1986, working at the La Silla Observatory in Chile. After two years of searching they were able to find just one Type Ia supernova and one likely Type II. Unfortunately, their Type Ia was found several weeks after it had already begun fading, and so could not be used for a distance measurement. Finding a single specimen after two years, with no useful measurements and no advance warning to schedule telescopes for follow-up studies, convinced many astronomers that Type Ia supernovae would be very difficult if not impossible to use as cosmological tools.

The spectra of Type Ia supernovae are distinguished by the lack of hydrogen emission lines and the presence of a silicon absorption feature. In a normal Type Ia spectrum, absorption lines of many elements and their ionic states appear, often in several positions. (After Dan Kasen)

In 1988 one other search for high-redshift supernovae was begun, by Carl Pennypacker and Saul Perlmutter, two leading members of Muller's robotic supernova search team. Their partnership marked the foundation of what would later be named the Supernova Cosmology Project.

How to weigh the universe

The Supernova Cosmology Project's goal was ambitious: to discover and study distant supernovae in sufficient detail to permit a measurement of the (presumed) slowing of the universe's expansion. Effectively this would amount to "weighing the universe" by determining the total mass density that gravitationally slows its expansion.

If the mass density was sufficient, the fate of the universe would be sealed: eventually expansion would slow to a halt and reverse, and the universe would collapse in a big crunch. Such a mass density would actually curve space enough for it to bend in on itself — so much that the universe would be spatially finite, not infinite. Thus the SCP team hoped to address some of the most fundamental physical and even philosophical questions imaginable — Is the universe infinite? What is the fate of the universe? — by tackling the practical experimental problems of measuring high-redshift Type Ia supernovae.

A basic physical insight guided their approach. Type Ia supernovae were different from all other potential standard candles for measuring cosmological distances in that each was a separate event that broadcast a detailed accounting of its physics, in the form of a light curve and spectral time series.

With a dataset of light curves and spectra for enough supernovae at sufficiently high redshift, cosmological parameters could be measured. No such dataset existed. Nobody had yet applied the kind of concerted, multifaceted effort that particle physicists — including several members of the SCP team — were accustomed to using to get the data they needed.

The first step was to find the supernovae:

• They had to design and build a camera with a wide enough field to observe thousands of galaxies, and find a world-class telescope that would host such a camera.

• They had to develop new image-analysis software to sift through tens of thousands of galaxies and differentiate the real supernovae from "background" transients (many likely to be of new kinds). No previous work had looked for transient objects so faint over such a wide area.

• They had to establish computer network links that could bring large data sets from remote telescope sites to powerful computer centers that could analyze the data quickly enough to schedule follow-up studies on any supernovae discovered.

Even if the SCP could find the supernovae in the first place, some astronomers doubted they would obtain useful data. A few of their concerns:

• Would the signal-to-noise ratio be good enough to recognize Type Ia spectra?

• Could the magnitude or flux (brightness) of supernovae at different redshifts be reliably compared to one another (known as K-correction)?

• Could the light from a supernova's host galaxy reliably be subtracted from the supernova itself? (The host galaxy couldn't be observed until months later, after the supernova had faded completely, and observing conditions would be different.)

• Would it be possible to measure and account for dimming of the supernova (which skews distance measurements) caused by intervening dust?

There were nontechnical problems as well. Time on the world's largest telescopes is valuable. No responsible allocation committee would schedule follow-up time, needed for obtaining spectra and light-curve photometry to confirm Type Ia supernovae, for supernovae that might or might not be discovered. And they wouldn't allocate time to look for supernovae in the first place, if no follow-up time was available to confirm any discoveries. They wanted proof in advance that any proposed search strategy would work, although such a technique must necessarily go unproven until demonstrated — a classic chicken-and-egg problem.

The Supernova Cosmology Project systematically went about tackling each and every one of these problems. Perlmutter and Pennypacker had taken the first step in 1988, by building a wide-field imager for the 3.9-meter Anglo-Australian Telescope at Siding Spring in Australia — an imager that remained that telescope's most popular instrument for many years to come. In return, the Observatory allowed them 12 nights to look for distant supernovae, spaced over many months. The weather was good for only two and a half of those nights, yet they found a promising Type Ia candidate. Without follow-up time, however, they couldn't prove it.

The SCP team finally found its first convincing high-redshift Type Ia supernova in 1992, designated SN 1992bi, while working at the Isaac Newton Telescope in La Palma, the Canary Islands. Their discovery depended on a number of now-refined tools and techniques:

• Software that could find supernovae in wide fields of high-redshift galaxies within a day after the observation, after eliminating spurious transients such as asteroids, cosmic rays, and active galactic nuclei (AGN).

• A K-correction technique not based on the customary method of extrapolating a candidate's spectrum taken with a single filter and comparing it to the then-inadequate database of Type Ia spectra, but rather on observing the same part of a candidate's spectrum through two different redshift-adjusted filters.

• A way to remove host-galaxy light from the supernova measurement, while estimating its contribution to systematic uncertainties.

By 1994 the SCP had proved a new observing approach that could guarantee "supernovae on demand" — scheduled discoveries of batches of high-redshift supernovae captured while still brightening, enabling photometric calibration at peak brightness. The technique involved making images of 50 to 100 patches of sky, each containing roughly a thousand distant galaxies, in the dark sky just after a new moon. Three weeks later the same patches of galaxies are imaged again. Somewhere in these images a dozen or more supernovae, on average, would show up as new points of light. The method guarantees that the still-brightening supernovae (most of them) will be available for study before and during the new moon, the best nights for observation.

Guaranteeing supernovae on demand involves taking images of 50 to 100 adjacent patches of sky, comprising tens of thousands of galaxies, just after a new moon. The same patches are reimaged just before the next new moon. New bright spots pinpoint new supernovae explosions, and the timing ensures they are found before or near their peak brightness. During the moonless nights that follow, large telescopes (including the Hubble Space Telescope) are prescheduled to obtain follow-up spectra.

Several techniques the SCP developed for pinpointing Type Ia's among their supernova discoveries later became standard. For example, as astronomer Alex Filippenko was to recall later, "When possible, the HZT's follow-up images were obtained through custom-made filters" in two bands, "thereby minimizing the K-corrections" in the manner previously outlined by Alex Kim, Ariel Goobar, and Perlmutter of the SCP team. Another technique that became standard was to use images from the Hubble Space Telescope (HST) to identify elliptical host galaxies; Type Ia's are the only kind of supernova that appear in elliptical galaxies.  

The SCP had managed to prove, for the first time, that high-redshift Type Ia supernovae occur often enough and in sufficient numbers to do cosmology. Once they had cobbled together enough telescope time to prove that they could discover batches of still-brightening supernovae on demand and in a predictable manner, major telescopes were no longer reluctant to grant time either for observation or follow-up.

For observation these included the 4-meter telescope at the Cerro Tololo Inter-American Observatory (CTIO) in Chile and the Hubble Space Telescope, which had to be scheduled months in advance (with target locations specified to within a degree on the sky), and for follow-up the Keck 10-meter telescope on Mauna Kea in Hawaii and the 3.6‑meter WIYN telescope on Kitt Peak in Arizona. After 1994, no distant supernova search was ever again conducted without follow-up time scheduled in advance.

Meanwhile a number of theorists, hoping to explain certain thorny issues in cosmology, had revived Einstein's cosmological constant in a new form. Quantum theory holds that space is permeated with vacuum energy; these theorists equated the vacuum energy with an accelerating cosmological constant.

By the end of 1994 Perlmutter and Goobar had showed that by studying Type Ia supernovae at a sufficiently large range of redshifts, it would be possible to extend measurements of cosmological parameters beyond the question of mass density — to an investigation of the cosmological constant as well.

Dark Energy's 10th Anniversary, Part II

Additional information

• "Scientists see cosmic growth spurt: Universe getting bigger and bigger, faster and faster—forever," by Charles Petit, appeared in the January 9, 1998 issue of the San Francisco Chronicle.
• "A few starry and universal truths: Astronomers tell of a bright past, dark future," by Charles Petit, appeared in the January 12, 1998 issue of US News & World Report.
• "Exploding stars point to a universal repulsive force," by James Glanz, appeared in the January 30, 1998 issue of Science, and is available online to subscribers.
• More about Type Ia supernovae as standard candles
• More about the Einstein's cosmological constant and the general theory of relativity, from Wikipedia
• The website of the Supernova Cosmology Project