In the second quarter of the 20th century, quantum theory faced some serious challenges, including unexplained details of atomic spectra and difficulties calculating basic properties of charged particles. In 1947 Willis Lamb and Robert Retherford of Columbia University discovered an unexpected detail in the hydrogen spectrum, later called the Lamb shift, that became an essential clue in solving both problems. The measurement agreed with new calculations and was the first indication that the theoretical approach called renormalization could resolve the mathematical infinities that had threatened to derail the progress of quantum mechanics.
By the 1940s, theorists understood a variety of phenomena that had small effects on the energies of atomic electrons, such as relativistic corrections and interactions between spin and orbital angular momentum. These effects showed themselves in the so-called fine structure of atomic spectra—the way that many spectral lines, corresponding to jumps between electron energy levels, are seen on close examination to split into groups of closely spaced lines.
Lamb and graduate student Retherford wanted to measure the hydrogen fine structure by investigating two specific electron states. One was a relatively long-lived -state, with a spherically symmetric orbital, and the other was a shorter-lived -state, with less symmetry. Standard theory predicted that the two states should have equal energy but that applying a magnetic field should influence the states in different ways and induce an energy difference between them.
The team sent a stream of electrons at right angles into a beam of hydrogen atoms, exciting a few of them into the -state and also deflecting them slightly from the main beam direction. The excited atoms passed through a region containing both microwave radiation and an adjustable magnetic field, and then hit a metal target. The excited atoms would then drop back to the ground state, emitting electrons that the team could detect as a current. The key to the experiment was that if the magnetic-field-induced energy difference between the two states was equal to the energy of the microwave photons, then the long-lived -state would absorb a photon and turn into to the short-lived -state. These atoms would drop back to their ground state before reaching the target, and the current in the detector would essentially vanish.
By plotting the critical magnetic field strength for a variety of microwave frequencies, Lamb and Retherford could determine the energy difference between the two states in the absence of a magnetic field. Contrary to expectation, the difference was not zero.
This departure from theory became known as the Lamb shift and was a prime topic for discussion at the Shelter Island Conference on Quantum Mechanics that took place in June 1947 at the far end of Long Island, New York. Many of the attending theorists argued that the Lamb shift was a result of the “self-energy” problem in quantum electrodynamics. The problem was that calculations of the interaction of an electron’s charge with its own field yielded apparently infinite values for the particle’s energy and mass and also threw off calculations of atomic spectra.
It was Hans Bethe, on the train ride home, who wrote a short paper giving a somewhat sketchy but fairly accurate calculation of the shift . The solution to the self-energy problem, proposed by others, was to think of the “bare” electron as having infinite energy that is mostly cancelled out by the infinitely negative energy of its interaction with its own electric field. This so-called renormalization approach leads to a correction to the classical energy that depends on distance. A -state electron spends a different amount of time close to the nucleus than an -state electron, so they require different corrections. Bethe’s estimate for the resulting Lamb shift fit the experimental result remarkably well and demonstrated that renormalization—which is at the core of today’s quantum mechanics—could be verified in experiments.
Accepting a share of the 1955 Nobel Prize in Physics for his discovery, Lamb remarked that some previous experiments  a decade before his and Retherford’s “indicated a discrepancy which should have been taken seriously.” Silvan Schweber of Brandeis University says, however, that although the previous findings had led to some theoretical work, the importance of the cleaner Lamb-Retherford experiment was that “after it was reported at Shelter Island, it became the point of departure for the renormalization program.”