Reading the "ClimateGate" EmailsUpdate: I wasn't competely satisfied with this post when I wrote it, but since I had been promising it to Charles Hugh Smith for days, I let it go out as is. Since it's getting thousands of hits now, I have revised it a bit. There is a bit more material and the order of sections is different. The newer version is here. On November 19, 2009, a 61 megabyte file called "FOI2009.zip" started to circulate on the internet. This contained a directory of over 1000 emails sent to and from people at the Climate Research Unit, University of East Anglia, UK, plus supporting documents and software. Check a BitTorrent site for "FOI2009.zip" if you want a copy, or you can find a searchable index of the email at http://www.climate-gate.org/. CRU is one of the major climate research centers, and the people sending the messages are the biggest names in climate science. The revealed messages have been fairly embarrassing to the people involved. Phil Jones, the director of the CRU, has temporarily stepped down while it's being investigated. Right-wing global warming "deniers" are having a field day with it. Mainstream press and the lefty blogs are taking a "move along, nothing to see here" attitude. Many commenters don't seem to have read the emails themselves. They are just repeating the same fragments over and over in the usual echo chamber. I've been interested in climate issues for awhile now, and have written a couple of long pieces on where I think things stand. If you had to describe my position in a single sentence, it would be "the current data, models and theories of climate aren't solid enough to say one way or the other if humans are warming the planet." You can find my previous items here:
- Global Warming: Our Story So far What's in the MailThe earliest email is March 7, 1996, and the latest, November 12, 2009. Most of them are to, from or CC'd to Phil Jones, Director of CRU, but many different authors are represented, and some of the emails do not even CC Mr. Jones. Also, despite the juicy comments that have been reprinted, the vast majority of the mail is day to day details on preparing research papers or chapters for the IPCC documents. It's dry as dust. See Email 1139932579. An interesting question is where this all came from. It's usually described as the result of hacking the Climate Research Unit (CRU). I doubt this is right. These are a very carefully selected set of emails, with none of the personal messages or administrative junk you'd normally find in an archive. It also just doesn't have the "nyah, nyah" shock value and sloppiness of most hacker efforts. I think this is more likely to be an inside job. Despite the name, and some speculation along those lines, I don't think this set of messages was selected as part of an official Freedom Of Information Act reply. No administrator, lawyer or CRU member would include emails that insult other members of the community, especially when there's no actual climate science mentioned. On top of that, the emails themselves show that CRU had no intention of answering the FOI requests. Since there's no single person listed as sender, recipient or CC on all of the messages, it's hard to see who could have collected all of these. Phil Jones himself comes closest, so perhaps this is extracted from his personal archive. It's hard to believe he'd have kept some of these though. It will be interesting to see what develops. The AccusationsYou can find lots of details out on the net, or look at the emails yourself. Let me give a summary of what CRU has been accused of, and my take from reading the email.CRU is Evading Freedom Of Information Act RequestsIt's a matter of record that Steve McIntyre and others made FOI requests for the data held by CRU. These requests were refused. The emails make it clear that CRU had no intention of answering the requests. They found excuses to refuse the requests and that was the end of it as far as they were concerned. There's also a single email from Phil Jones to Michael Mann asking him to delete emails concerning the preparation of the IPCC Fourth Assessment Report (AR4). If they get into any legal trouble, it will be because of this issue. The more interesting question is why they refused. Possibilities include:
There are two other reasons supported by the emails.
Email 1228330629 (farther down in the reply chain): At 01:17 03/12/2008, Ben Santer wrote to Tom Wigley:Note though that although Steven McIntyre is an outsider and not a climatologist, the other people named (Christy, Douglass and Singer) are authors of peer-reviewed papers published in major climatology journals.
At 16:50 02/10/2009, Malcolm Hughes wrote to Keith Briffa: It's not clear to me from reading the emails when the tree cores were done. Some of the messages make it sound like it was done in the 80s and 90s, which would be a long time to hold on to data. The email above makes it sound more recent. Of course, this would all be acceptable if climate science were still just an academic exercise. It's not so defensible when trillions of dollars and a whole new energy infrastructure are on the line. You see that with a lot of the emails. They want to do science the traditional way, and aren't prepared to deal with investigations, FOI requests, making data and software public, and answering blogs. They don't seem to understand this strange urgency their critics have... CRU is Corrupting the Peer Review ProcessThe CRU group has been accused of throwing their weight around to get papers by their critics rejected by scientific journals. Then they sneer at skeptics saying they have no published peer-reviewed papers. There's some evidence in the emails that they tried, and some of the blogs make it sound like they succeeded to some extent. It's hard to decide exactly what they were thinking though. There are a range of explanations.
Reading the email, I think the biggest problem is that they have no "loyal opposition" internally, or even in the field -- people who would take warming seriously, but force them to validate their results more carefully. None of the CRU emails that I've read come across as "Devil's Advocate" sort of messages. It's mostly true believers there, and they are a bit defensive. Trying to get reviewers fired, etc., is very bad form, but it's not clear to me that it has made any difference. No one seems to take any challenges to the basic idea of warming seriously at this point. CRU is Unable to Reproduce its Own ResultsIn addition to the email, there is a "documents" directory, and it contains a file named "HARRY_READ_ME.txt". Speculation is that this is by Ian "Harry" Harris, a CRU member. It documents an attempt to get some of the CRU software and datasets in usable condition, after the original authors have left, or perhaps just after years of neglect. The comment "Argh!" can be read at many, many points. You can go to open-source guru Eric S. Raymond's blog for scathing analysis of this code, ClimateGate issues, and global warming in general. Many sarcastic comments by programmers. I'm not going into this in detail because this item is too long already and I want to make some more important points below. Here's what I take away from the discussion:
CRU is Altering the Raw DataA lot has been made of CRU "hiding the raw data" or "altering the raw data", or throwing it away after they've produced their "value-added" data. The fact is, there is no single source of pure "raw" data, and the raw data isn't usable anyway. Datasets come in from many different sources. CRU "standardizes" them to put them all on the same temperature scale, so they can all be compared to one another, or fed into computer models that generate climate predictions. It's this "standardized" data that CRU keeps, and the original, unstandardized records that it has thrown away. For example, here's one of the emails on Sea Surface Temperature (SST) data. This is a writeup for publication (not sure by who), forwarded from Gavin Schmidt to Phil Jones: ... Like almost all historical climate data, ship-board sea surface temperatures (SST) were not collected with long term climate trends in mind. Thus practices varied enormously among ships and fleets and over time. In the 19th Century, simple wooden buckets would be thrown over the side to collect the water (a non-trivial exercise when a ship is moving, as many novice ocean-going researchers will painfully recall). Later on, special canvas buckets were used, and after WWII, insulated 'buckets' became more standard - though these aren't really buckets in the colloquial sense of the word as the photo shows (pay attention to this because it comes up later). From this, you see what we're talking about. Maritime records back to 1880 or before (handwritten!) would have been digitized. Then each record would have been coded for the type of bucket used. Then each measured temperature would have been "standardized" to adjust the temperature record based on type of bucket, to make them all uniform. That final temperature would have been kept by CRU as "value-added" data. This all matters because if you adjust a bucket temperature by 1 degree, you are adjusting it by more than the warming signal you are looking for! If you get that wrong (systematically wrong, because of your software or assumptions) you can bias the data upwards. There's even an email about this: On Oct 16, 2008, at 6:52 AM, Phil Jones wrote: There are these same issues across all of the data. Satellite data is adjusted. Land and sea measurements are adjusted. Tree-rings and other "proxies" that go far back in time are very adjusted. They should have kept original sources (hand-written logs, tree cores, etc.) and a description of what was done to each reading to standardize it. For tree rings, even the selection of which trees to core affects the "raw data". For ground temperatures, the siting of each thermometer (whether it's in a city, or next to an air-conditioning unit) is "raw data." It should all be online, with each number linked to software and a description and images. Dream on! The original data is probably still out there somewhere (and it's not clear how much CRU has actually thrown away.) A team that wanted to replicate everything CRU has done would digitize it, code it for source, adjust it (with their own software) and produce a new temperature record. That would be a lot of work. So did they really "alter" the data in the way that skeptics are claiming? I have two issues from reading the emails. First, the use of tree-ring data seems very problematic. As I'll discuss in the Climate Data section below, there are a list of questions about whether tree rings can be used to say anything about global climate. And as far as I can tell, trying to use this data is the source of all the "artificial" corrections in their software. Second, there's this email that seems odd: At 23:25 on 09/27/2009, Tom Wigley wrote to Phil Jones: It's hard to say what they are getting at here. Since he mentions the 1910-1040 warming (which is nearly as large as the 1980-2000 one), I doubt this is what he means by "1940s blip." Still, it does sound like they are trying to impose an artificial correction to the sea temperature data so that a smooth warming curve, without any inconvenient "blip" is shown on the result. MAGICC is their climate model, used to predict future warming. This is the opposite of starting with the data and looking for trends. This sounds like imposing a trend (or smoothing it, at least) on existing data. But without context, it's hard to be sure. The Bottom LineI have a lot of reservations about climate science and the quality of the data. The emails confirm some of those reservations. More on that below. But does the ClimateGate archive really prove anything about the science? The archive (probably leaked) is damaging to the people involved. It makes them look nasty, petty and defensive. It raises many suspicions about whether, as Global Warming True Believers, they have maybe found what they are looking for. For example, when the tree ring data comes out "wrong", you could just keep processing it until it looks "right" -- that is, agrees with all the other data you have. Or you could take the portions of the record you think bolster your case, and ignore the modern parts that don't. If you Truly Believe, and have put a lot of work into gathering tree ring data, you are less likely to say "tree rings are worthless as records of climate" or "we just don't understand why these numbers diverge from other records." It's not clear to me that CRU has really distorted the process much, or made a mess of climate research. I think the FOI and peer review issues are both overblown. They are very bad form, and perhaps legal trouble for some of the people involved, but they don't say much about the science of Global Warming. Going forwards, I don't think this release will help the science much. Activists on both sides will dig in their heels and refuse to admit problems. No climate scientist will write an email saying what he honestly thinks of a paper or a person for awhile. In fact, I expect a lot of email has been deleted around the world this week! Perhaps there will be more openness. I wouldn't be surprised if some scientific journals bend over backwards to publish some skeptical papers. But the bigger problem is that the science is "settled" in the minds of so many researchers. That is premature, in my opinion. Let me finish with an analogy. You've probably seen these Discovery Channel documentaries on dinosaurs. In one, a guy with a silly hat and wild hair will get on and declare that T-Rex was a predator, no doubt about it! And he'll point to bones and teeth, and then say something like "anyone who says otherwise just isn't doing good science!" Then the next guy with wild hair will come on and say that T-Rex was clearly a scavenger, and point to the absence of any fossil bite marks or trackways. He'll argue that T-Rex would have fallen over trying to chase something. And just look at the silly little arms! "Anyone who thinks T-Rex was a predator has watched too many movies." This is all very amusing (I keep expecting it to turn into a Monty Python sketch), and matters to almost no one outside the perhaps 100 people in the world who have ever measured a dinosaur bone. Now imagine that activists descended on these people, saying they were going to spend a few trillion dollars cloning a T-Rex, and absolutely had to know, right now if T-Rex was a predator or scavenger! Most of the scientists would freeze like deer in headlights. Some of them would mumble about uncertainties and lack of data, but the activists and politicians don't want to hear it. Instead, the few True Believers would step forward and say "T-Rex has been proven by science to be a predator (or a scavenger.) I'll stake my scientific reputation on it!" Policy makers would pick the side that was politically popular. They would be influenced by the public, who think "of course, it's a predator -- I saw Jurassic Park!", and whatever industry concerns there were. After a winner was picked by politicians, funding would dry up for the opposition. Without grants, they can't do their own fieldwork to get raw data. Without data, they can't write papers. Without publications, they can't get grad students. Without money or students, they can't get more data. The science would be declared "settled" and skeptical work would be shut down. That, unfortunately, is what I think has happened to climate science. These groups are still acting like academics -- sniping at rivals, keeping minimal records and writing amateur software as if this were a purely academic issue. But the politics have taken over and the stakes are too high to continue business as usual. Climate DataIf we see warming today, and want to blame it on human activity, it would help if we could answer two questions. 1) Is this warming unique, or has it happened before? If it has, then it seems unlikely to be just due to humans. And 2) If we are warming, when did it start? If it starts in 1950, industrialization could be a cause. If it starts in 1850 or earlier, industry is unlikely to be the culprit. To answer either of those questions, we need historical climate data. Knowing current temperatures and following the recent trend isn't enough. So let me do a quick review of the historical climate data, with some comments from the emails. As mentioned above, the data comes from several types of source, and is all standardized before it is used. Satellite DataWidely regarded as the best data there is, because it covers the entire planet, and seems independent of data gathering issues or "urban heat island" effects. I thought this was the gold standard, but there's an email that makes me doubt it: Email 1079108576: (fourth para, last msg) Even though much of the differences [between satellite and ground temps] may now be apparently explained, it's still a terribly messy job. The satellite system wasn't designed to measure tropospheric temperatures, the calibration and orbital decay and retrieval algorithm and all the other technical issues are ugly, and nobody knows how much the lower stratospheric cooling ought to have infected the upper troposphere, among other points one might make. Good or not, the real problem with satellite data is that it only goes back to the 1980s. This means we don't have any satellite coverage of prior warm periods like the 1930s to use as comparison. Instead, we compare standardized ground temperatures with standardized satellite temperatures. Since the data was collected with completely different types of instrument, the comparison will only be right if your standardization techniques are right. Land Temperature StationsThese are the ground stations at airports and cities, used for weather reports and forecasting before satellites. Some go back to 1880 in the U.S. The rest of the world is very different. The farther you go back, the fewer there are. For earlier periods in the 20th century, the coverage is minimal outside North American and Western Europe. There are issues with this data as well. Stations have been moved, the time of day readings were taken sometimes changed, and there are breaks in the data. If a city grows up around the station, the "Urban Heat Island" effect may bias the readings upwards over time. A website, http://www.surfacestations.org/ claims that many sites they have checked do not meet the standard for accurate readings. How much to correct for this is an open question. How much to correct for Urban Heat Island effects is an open question as well. At 17:01 on 01/06/2009, Phil Jones wrote to Chris Folland, Sea Surface TemperaturesAs I mentioned above, there are recorded by sailors on ships using a variety of methods. More recently, floating automated sensors are being used. Each different source and method has to be calibrated differently. Proxies, including Tree RingsPrior to 1880, you have few temperature records from thermometers. So they try to reconstruct climate from "proxies" -- natural events, like the growth of trees (shown in the width of the rings), that are related to temperature. The ideal case would be something like a California Redwood tree, sitting in a forest undisturbed by people or nearby cities, for 2000 years. If the growth of the tree were affected strongly by temperature, and that location had a strong global warming signal, you'd be set. Unfortunately, that's not the case. Here are some issues with tree rings:
At 11:57 11/16/2006, Keith Briffa wrote:And this email from a critic, quoted in Email 1256760240: At 10:34 10/02/2009, Donald Keiller wrote to Kieth Briffa: Coverage of the DataSo we have three datasets -- satellite, thermometers used on land and sea, and proxies like tree rings. Satellites cover the entire world, but the historical data is much thinner. The proxies probably only cover a few hundred points on the globe.The map below compares the warming of the 1930s with the warming of the 1990s. The gray parts of the map are places where there is no temperature data in the 1930s, and so no comparison can be made. You can see that much of the world is a bit warmer now than in the 1930s, but some of it is colder. The temperature in the U.S. is about the same now as then. Data is missing in South America, Africa, the Middle East, Central Asia, the Arctic and Antarctic. The Year 1900 would look even worse, almost completely blank outside North America and Western Europe.
I haven't plotted the sea data from CRU because it's heavily interpolated to show the entire world. This is a problem, since there's no way they have data for the entire world. Instead, they must have temperatures collected by working sailors traveling shipping lanes (no global warming research vessels or automated buoys in 1930!). The modern shipping lanes look like this:
Anything far off these lanes won't be in the data. Again, there would be little at the poles, and not many points from the middle of the Pacific Ocean. What's the point of these maps? Any graph that shows global warming this century has to have used this data. All of the climate research centers have access to the same weather stations. There's no separate set that only CRU or only NASA has access to. Their climate graphs and models only differ in how they process the data.
The climate researchers can't create data where there is none. They can't create
a true average temperature for 1930 with a map this skimpy. A big part of their trends could
very well be determined by how they compute these global temperatures.
For more, see Free The Memes! Have a Comment? More Writings |
|