Note: This is a cross post of an original article of mine that was posted on the site wattsupwiththat.com today. Thanks to Anthony Watts and his readers for their support.
The foundation of all observational science is data. This is true whether the data is temperature measurements from ground networks, satellites, or any other thing in nature that can be observed, quantified, and recorded. After data is recorded it must be archived so that future researchers who seek to extend or question conclusions drawn from that data can go back to the original source to replicate results. This is a fundamental premise of the scientific method, without it we can make no reliable statements about nature and call it science. This is true whether or not the subject is climate change, planetary motion, or any other scientific discipline. This missive is about the supremely important subject of data archival and how you the reader can support our lunar data archival project. First a historical digression.
The Importance of the Recording and Archival of Scientific Data
In the era before computers and the Internet, data archival was the responsibility of the scientist who obtained and recorded scientific observations. Johannes Kepler used Tycho Brahe’s archived records of meticulous observations of planetary motion to calculate the elliptical orbit of Mars and thus developed his laws of planetary motion. After the laws were published, anyone could check Kepler by going to the observatory and do their own calculations based on the archived data. The archived work of Brahe and Kepler underpinned Sir Isaac Newton’s formulation of his theory of gravity. Without archived data, Newton would have had no basis for his calculations. A scientist’s archives, stored at institutes of learning, has been the standard method of preserving data and results until the era of the computer.
Data Archiving in the Modern Age
In recent times a structural deficiency has emerged in the sciences related to the storage, archiving, and the availability of original data. Beginning in the world war two years and exploding afterward, scientific data in many fields of the physical sciences began to be obtained though electronic means. Strip charts, oscilloscopes, and waveforms from analog and digital sensors began to be fed into calculating programs, and results obtained. These results were and are used to develop and or confirm hypotheses. This exploded in the 1960’s and has continued to where today it is ubiquitous. However, there has been a decoupling in the scientific process regarding the recording and archiving of data and the ability to replicate results. The following example is just one of a legion of problems that exist in this realm.
In the 1960’s when data was obtained and fed into the computer, the data was often truncated due to memory limitations and computational speed of computers of the era. For example a paper was published by NASA as NASA TM X-55954 entitled:
The Radiation balance of the Earth-Atmosphere System Over Both Polar Regions Obtained From Radiation Measurements of the Nimbus II Meteorological Satellite;
This is probably the first definitive study of the radiation balance of the Earth-Atmosphere system published in the space era. Figure 1 is a figure from that paper:
Figure 1: Radiation-Balance of the Earth-Atmosphere System Derived from Nimbus II MRIR Instrument 1966
This is an important paper in climate studies as it was the first paper to quantify the radiation balance based on data from satellites. However, the question is, where is the original data was fed into the computers to come up with these results?
Recovering the Nimbus II HRIR Data
In the paper the primary data used to produce the temperature gradients was obtained from the Medium Resolution Infrared Radiometer (MRIR) that flew on the Nimbus I-III meteorological satellite, the first satellite to carry this high quality of sensor. Where is that data today? I actually don’t know much about the MRIR data but I do know quite a lot about the High Resolution Infrared Radiometer (HRIR) that was a companion experiment on the early Nimbus birds.
During the missions the data from the spacecraft was transmitted in analog form to ground stations where it was recorded and from there it was sent for processing at NASA Goddard Spaceflight Center in Greenbelt Maryland. Figure 2 shows the design of the HRIR instrument and the computerized method of processing of the data:
Figure 2a, 2b: HRIR Calibration and HRIR Data Processing
Looking at Figure 2a on the left you see that a laboratory calibration was done against a known blackbody target. An in flight calibration standard was measured at the same time and a reference calibration for the instrument obtained. The same in flight calibration reference blackbody (shown in the upper left) is scanned on each swath (a swath is a line of recording representing an 8.25 x 1100 km section of the Earth), providing a continuous means to maintain calibration of the instrument in flight. Figure 3 shows a trace of a swath of HRIR analog data:
Figure 3: Nimbus HRIR Swath Trace With and Without Calibration Stair Step
In 2009 my company, as a result of our work on the 1966 Lunar Orbiter data, was contracted by the National Snow and Ice Data Center (NSIDC) to take raw Nimbus HRIR data, correct errors, and translate it into a modern NetCDF-4 format so that it could be used in studies of pre 1979 Arctic and Antarctic ice extent. The HRIR data had been digitized by the diligent effort of NASA Goddard scientists who had retrieved the surviving tapes from the federal records center. Since no tape drives exist anymore that can read the tapes, a company was contracted to use an MRI type machine to read these low data density tapes. This worked remarkably well and the data from over 1700 of these tapes were provided to us. However, it turns out that the data tapes do not have the original analog data. It turns out that the original analog tapes no longer exist.
The digitized data that we used are, as best as we can tell, is an intermediate product derived from the IBM 1704 computer processing. The swaths no longer have the calibration stair step or sync pulses but each one does have a metadata file with geo-positioning data. We reprocessed the data and re-gridded it to comply with modern Net-CDF4 conventions. The HRIR images produced are then used by the NSIDC to find the edges of the polar ice. We took the files and translated them into .kml files for display on Google Earth with dramatic effect. Our work is described in an AGU Poster (IN41A-1108, 2009). Figure 4 is a .kml file mapped in Google Earth.
Figure 4: Google Earth .kml File of the Nimbus II HRIR Data, August 23, 1966
This image is centered near Indonesia. Bluer temperatures are colder and clearly show the Monsoon clouds. The contrast between the ocean and Australia is clearly evident. Colder temps in the Himalayas are seen as is the heat of the Persian gulf and the deep cool temperatures of the clouds in the upper right from typhoon Helen and Ida. The HRIR data can be used for many purposes but due to the loss of calibration, only a relative comparison with modern IR data can be obtained. This also renders replication of the findings of the radiation balance paper nearly impossible. So, what the heck does all of this have to do with Lunar images?
The Lunar Orbiter Image Recovery Project (LOIRP)
In 1966-67 NASA sent five spacecraft to orbit the Moon as a photoreconnaissance mission to scout landing sites for the Apollo landings. Today’s reader must remember that prior to these missions mankind had never seen the Moon up close. The first three Lunar Orbiters were in a near equatorial orbit and the last two in polar orbits for general mapping. Each carried two visible light cameras, a 24” focal length instrument obtaining images at about 1 meter resolution, and an 8” focal length instrument at about 5-7 meters resolution on the on the lunar near side. The images were recorded on 70mm SO-243 photographic film which was processed on board. This film was then scanned with a 5 micron spot beam that modulated an analog signal that was transmitted to the Earth. This is shown in figure 4:
Figure 5: Lunar Orbiter Image Capture, Scan, Transmit, Storage and Print Process
The images were captured on the Earth via two dissimilar processes. At the lower left, of the most interest to our project, was the recording of the pre-demodulated combined raw analog and digital data on a 2” Ampex FR-900 Instrumentation tape drive. The second process demodulated the signal to produce a video signal that was sent to a long persistence phosphor called a kinescope. The resulting image was photographed by a 35mm film camera. The 35mm film strip positives were then assembled into a larger sub-image that was filmed again to create a 35mm large negative that was processed to create a 35mm print that was used by the photo analysts to look for landing sites. However, as one might suspect, there was degradation of the quality of the images in going through this many steps.
I was aware of this quality reduction as I had worked with the film records in the late 1980’s at the University of Alabama Huntsville. At that time I had researched the tapes but was informed that the tapes were unavailable, though rumors were that someone was digitizing them. However, this never happened and all the archived images, such as the excellent repositories at the USGS in Flagstaff Arizona and at the Lunar and Planetary Laboratory (LPI) in Houston were derived from the films and were the only high resolution images of the Moon available.
In 2007 quite by accident I read a newsgroup posting that Nancy Evans, a retired JPL researcher, was retiring from her second career as a veterinarian and had a four FR-900 tape drives that she wanted to give away. I later found that she was the responsible official at NASA JPL in the 1980’s that had saved the original Lunar Orbiter analog tapes and that they were still in storage at JPL. I contacted Nancy and JPL and she was willing to donate the tape drives and JPL was willing to loan the tapes to NASA Ames were we had donated facilities to attempt to restore the tape drives and read the tapes. I raised a bit of funding from NASA Watch editor Keith Cowing. We loaded two trucks with the 1478 tapes weighing over 28,000 lbs and the four tape drives weighing a thousand pounds each and drove to NASA Ames.
The reason that previous efforts by Nancy Evans and engineer Mark Nelson from Cal Tech had been unsuccessful was that NASA was not convinced of the value of the original data. I had known of the tapes before but we had to quantify the benefits to NASA before we could obtain funding. We found the money quote as we called it in an obscure NASA memo from 1966. This memo said in brief (figure 6):
Figure 6: NASA Memo Regarding Superiority of Mag Tape Lunar Images
This had originally been suggested by NASA contractor Bellcomm employee Charles Byrne as a means to improve the methods that would be used to analyze landing sites for the dangers from large boulders and to analyze the slope of the landing sites. If rocks were too big or the slope more than eleven degrees, it would be a bad day for the crews seeking to land. With this memo in hand NASA headquarters provided us with initial funding to get one tape drive out of the four operational and to see if we could produce one image. We had three questions to answer.
- Could we get a 40+ year old tape drive operational again?
- Even if the tape drive is operational, is there any data still on the tapes?
- Even if there is surviving data, is it of higher quality than the USGS and LPI archives of the film images?
Suffice to say we answered all three questions in the affirmative and in November of 2008 we unveiled to the world our first image, which just happened to be the famous “Earthrise” image of the Earth as seen from lunar orbit from August 23, 1966. The original image and our restored image is shown in figure 7:
Figure 7: Earthrise 1966 and Earthrise 2008!
The improvement in dynamic range we found from the documentation was a factor of four due to the reduced (250 to 1 on film vs 1000 to 1 on the tapes) dynamic range of the ground 35mm film. The raw data also preserves the sync pulses used to rectify each line of the data and when we used oversampling techniques (10x in frequency and bit depth) we can produce much larger images (the Earthrise image at full resolution is 60’ x 25’ at 300 dpi). With modern digitizing cards and inexpensive terabyte class drives this became a very manageable affair. For more information, this link is from a lunch presentation that I gave at Apple’s worldwide developer conference (WWDC) in 2009. Here is a link to an LPI paper.
Where We are in 2013
After our success NASA headquarters Exploration Systems Mission Directorate provided further funding. However, since ours was basically an unsolicited proposal that funding was limited. Each of the Lunar Orbiters (LO) acquired approximately 215 medium and high resolution images. The most important images are from Lunar Orbiter II, III, followed by LO-V, then I, then IV. The reason is that LO-II and III have the best high resolution images on the near side equatorial region. The digitized raw images best preserves the data in a form that can then be integrated into a multilayer dataset that best compares with today’s data which we have done on an experimental basis. In contrast to the Nimbus HRIR data the LO data fully preserves the calibration marks, which are on the tapes every 22 seconds. LO-I lost its image compensation sensor early in the mission resulting in blurred high resolution images. The medium resolution images are fine though they are less relevant for comparison purposes due to their lower resolution. LO-V has almost all of its high resolution images at 2 meters, thus being a good comparison to LRO. The lowest priority are the LO-IV images, which were obtained from a much higher altitude than the other missions and are thus of mostly historical value.
Our project has successfully digitized 98% of the LO-III images, with only six images lost to tape related causes (erased tapes), while we have found several images that are not in the existing USGS and LPI archives. We have so far digitized about 40% of the LO-II images, and about 10% of the LO-V, LO-IV, and LO-1 images.
We Need Your Help
We are today raising funds through the crowd funding site;
We are doing this as we do not expect further NASA funding and there is only a limited amount of time still available to digitize these tapes. The FR-900 tape drives use a head with four iron tips that rotate at 15,000 rpm. These heads are in direct contact with the tapes that are moving by at 12.5 inches per second, creating a sandpaper effect that quickly wears the heads down. Here is a video from a couple of years ago with a tour of the lab, which by is in an old MacDonald’s at the old Navy Base at Moffett field CA. Only a few dozen tapes can be played before the heads wear out, necessitating a refurbishment that costs well over $7000 each time.
We also have to pay our engineer to maintain the drive, our students to manage, assemble, and quality check the images as well as myself to manage the project, operate the tape drives (I worked in video production for years and thus do the operations and real time quality control during image capture). We are also preparing this data for subsequent archiving at the National Space Science Data Center though we also have the images archived at the NASA Lunar Science Institute and at our www.moonviews.com site where anyone is welcome to download them. We also have a Lunar Orbiter Facebook page that you are welcome to join.
The images that we are producing and the raw data will be available to anyone for their own purposes. We have students who have been doing real science of comparing the LOIRP digitized images with the latest images from the NASA LRO mission. Why is this important? Since the Moon has no atmosphere, even the smallest meteors impact the surface and make a crater. With a resolution on both LO and LRO ~one meter we can examine the lunar surface in detail over thousands of square kilometers over a period of almost half a century. We can then see what the frequency of small impactors are on the Moon. Not only does this provide information for crew safety while out on the surface of the Moon, it provides a statistical representation of the asteroid risk in near Earth space. The bolide that exploded over Russia is thought to represent a risk of a one in one hundred year event. What if that risk is higher? Our images, coupled with the LRO LROC camera images can help to better bound this risk.
Our project has been honored by congress and our images were used in a presentation by NASA to the president in 2009 and were part of a package of NASA photos provided in the inaugural package this year. We have had extensive coverage of our efforts in what we have termed “techno-archeology” or literally the archeology of technology. Many of these links are at the end of this article. However, with all of that it is a very difficult funding environment and that is why we need your help.
What is on the Crowdfunding Site
We are offering a lot of stuff for your donation on the site. We have collectable and historical images that were printed back during the Apollo era for varying price ranges. We have models of the Lunar Orbiter with a stand, suitable for your desk. We have microfilm from the original photographs and if you cannot afford any of that, you can just make a donation!
This is what we call citizen science, the chance to have a part in an ongoing effort to archive data that can never been archived again. Our tapes are gradually degrading and the tape drives cannot function without heads. Our engineering team is comprised of retired engineers who won’t be around forever. NASA JPL in 2008 estimated that to recreate what we have would cost over $6 million dollars. We have done what we have done with a tenth of that amount of money and with your generous donation we will complete our task by the end of this September.
The Big Picture
Stories like ours regarding the actual and potential loss of valuable original data is not a rarity. Due to funding cuts to NASA on October 1, 1977 they turned off the Apollo lunar surface experiments that we spent billions putting there. The majority of the data that was obtained up until the experiments were turned off was in great danger of being lost. Retired scientists and interested parties at NASA recently put together a team that retrieved these records from as far away as Perth Australia and the NASA Lunar Science Institute has a focus group dedicated to this effort. Sadly some of this data is still in limbo and may indeed be lost forever due to poor record keeping and preservation of the original data.
For the reader of climate science related sites most of you are well aware of the issues associated with the adjustments of original data in the field of climate science. The integrity of science is preconditioned on the ability to replicate results and the archival of data and the preservation of that original data is one of the highest priorities in science. We are doing our small part here with the Lunar Orbiter images. One of our team members is Charles Byrne, who just happened to be the one who wrote the original memo that resulted in the purchase of the tape drives. In talking with Charlie he never in a million years thought that a generation later he would be able to work with the original data. He has developed several algorithms that we are currently using to remove instrument related artifacts from our images. Charlie is still doing original science with Lunar Orbiter images and is the author of the near side mega-basin theory.
One of the reasons that I started thinking about original data was that at the same time I was working with the forth generation lunar orbiter film in the late 1980’s Dr. John Christy was working just down the hall from me at UAH recovering satellite data from the 1970’s that for all practical purposes was the genesis of the era of the climate skeptic. Did he think that his work would have had such a long lasting effect? Just think, did Brahe in his wildest dreams think that his meticulous work would lead to the theory of gravitation? We don’t know what may come in the future from the raw data that we are preserving but we do know that having an original record from 1966-67 could not be replicated at any price and with your support we will preserve this record for posterity.
A selection of published Articles About Our Project
Apple Worldwide Developer Conference Slide Show