Thursday, April 30, 2015 ... /////

Geomagnetic 44-month cycle seen in the climate data

Roy W. Spencer, John R. Christy, and William D. Braswell – the team of pioneers of the satellite temperature measurements – have released

Version 6.0 [beta] of the UAH [AMSU] Temperature Dataset Released: New LT Trend = +0.11 C/decade [raw data]
which is more compatible with the RSS AMSU dataset. In fact, UAH now shows a smaller (by 1/4 or so) warming trend than RSS. The trend has been exactly zero in the last 18 years. In the past, I tended to slightly prefer RSS AMSU – partly because I wanted to avoid suggestions that Spencer et al. aren't impartial just because they're skeptics (I surely do think that they are impartial). I also preferred a more silent method with which RSS was fixing their small bugs.

But this new release convinced me to play with the datasets again – do all kinds of Fourier analysis, Fourier filters, predictions, and so on. And I just found something that I want to share with you because it seems pretty exciting.

First, I used Mathematica to import the monthly global temperature anomalies:
b = Import["http://vortex.nsstc.uah.edu/data/msu/v6.0beta/tlt/uahncdc_lt_6.0beta1.txt", "Table"][[2;;-13,3]];
This gives me those 436 months from December 1978 to March 2015. You may plot them by ListLinePlot[b].

I looked at the Fourier decomposition, Fourier[b], somewhat carefully, and there were some peaks but they didn't quite impress me (although they already contained the reason of the excitement described below). Instead, I decided to look at the autocorrelation of the data:
core = Table[Correlation[b[[1 ;; -1 - k]], b[[1 + k ;; -1]]], {k, 1, 400}];
BarChart[core]
This takes the list of monthly anomalies b, shifts it by "k" months, and computes the correlation coefficient with the original data. The correlation coefficient is +1 for k=0, of course. But I expected a sort of a random curve. But this is what I got:

Wow, I told myself. It doesn't look chaotic. The ups and downs are almost regular, equally spaced: it looks almost like a sine function combined with a much slower one.

You should understand the graph. The horizontal axis is the delay that goes from zero to 400 months. The vertical axis is the correlation coefficient that oscillates. In fact, there are 9 periods between the delay 0 and the delay 395 months or so (the number of maxima in the interval is 10 but that's because you count both peaks at the boundaries, I am sure you know how to count periodicities).

So the average spacing between the delays is 395/9 = 44 months or so. To a certain extent, one feels confident that the graph above allows us to determine this constant 44 months rather accurately – so that one may conclude it's much more likely to be 44 than 43 or 45 months, for example. Think about it. The point is that the temperature wiggles are much more similar to those 44 or 88 months ago than to those 22 or 66 months ago.

OK, I could see the periodicity 44 months (3 years and 8 months) in the Fourier decomposition, too. There was a peak around these frequencies. I am just not experienced enough to immediately appreciate that the peaks around this frequency are really high. The similarity between the graph of correlation coefficients above and a sine was something I couldn't overlook.

Here is what I did to "see" the value referred to as 44 months above more accurately. Just calculate the average of the correlation coefficients over all the allowed delays that are multiples of "k" months:
coreD = Table[ Total[core[[1 ;; -1 ;; i]]]/Length[core[[1 ;; -1 ;; i]]], {i, 1, 60}]
BarChart[coreD]
Here is the resulting picture:

If you count the bars, you will see that the two highest peaks correspond to the delays that are multiples of 44 and 45 months, and the 45-month peak is actually a bit higher. So the periodicity seen in the global temperature anomalies is about 44.6 months or something like that.

Good. We live in the era of search engines so I immediately searched for 44-month and 45-month periodicities that could explain it. First, I ran into an essay by Willis Eschenbach who have Fourier-analyzed the temperature data as well and found the 44-month periodicity, too. So as I could have expected, I had discovered the wheel and America (in my mother tongue, we say that "you discovered America" if you did something trivial that every child and even Christopher Columbus could do, too). In that article, he concludes that the climate data don't show the slightest trace of the 11-year solar cycles and I completely agree with him. The people who believe that these 11-year sunspot cycles have to be critical for the terrestrial climate are fooling themselves.

But he probably didn't do any other search for that 44/45-month periodicity in the literature. I did.

First, I found the following 1997 article. Look at it:
Time variations of geomagnetic activity indices Kp and Ap: an update (full PDF)
Geophysicists G. K. Rangarajan and T. Iyemori in Bombay have analyzed some Kp and Ap geomagnetic indices between 1932 and 1995. The analysis up to 1961 was done previously; 35 extra years were added by these authors and they seemed remarkably consistent with the lessons of the first 30 years.

In particular, the Ap index – which quantifies some daily geomagnetic activity – shows periodicities of 16, 21, and 44 months. The first two were attributed to the solar wind and IMF oscillations with analogous periodicity while
the 44 month variation is associated with a similar periodicity in recurrent high speed stream caused by sector boundary passage.
Needless to say, my training in geophysics ended somewhere at the basic school – in the college, geophysicists were known as the least smart physicists, on par with the meteorologists. But search for the "sector boundary passage" or, if you know these things, please comment on the meaning.

The same Rangarajan and Araki found the same 44-month cycle in a slightly different quantity, the equatorial Dst index.

The idea that the geomagnetic activity could drive the climate change is intriguing, isn't it? But it gets even more puzzling if you read e.g. the 1933 paper by Abbot and Bond. On page 364, you may read:
We are able to reproduce it as the sum of seven regular periodicities of
7, 8, 11, 21, 25, 45 and 68 months.
Yes, 45 months is there, is a rather limited list. On page 370, they compare the 25- and 45-month periodicity with some climate data, see also Figure 7 on page 369. I don't quite see the "solid science" by which they obtained those 7 golden frequencies but I will later spend some time in attempts to fit the climate into a combination of these seven sines.

On page 366, they point out that 45 months is one-third of the sunspot period of 135 months, and offer some other numerology for the other frequencies. If the Sun itself were the driver, why would one-third of its sunspot cycle (the third harmonic) matter much more than the full cycle? A triangle would probably have to be hidden in the Sun.

Funnily enough, this photograph is actually real.

Let me stop these solar debates and return to the Earth.

Can you imagine that the geomagnetic activity drives the climate? If it has similarly fast cycles as some solar cycles, is it possible that the geomagnetic activity has been synchronized – the frequency was adjusted to agree – with the solar cycles? (I suppose that the Sun doesn't give a damn about the Earth's magnetic fields.) If they're independent, can then cooperate? Can some important effects depend on an interplay between the solar and geomagnetic activity?

Except for the hypothetical shielding of the cosmic rays, I can't see how the geomagnetic field would drive the climate, either by itself or in some combination with the solar cycles. Or perhaps the wind is (somewhere) charged and moves according to the magnetic fields and it matters for the climate? But because links between the climate and many weird things have been proposed, it seems surprising to me that the possible geomagnetic influences on the climate are not discussed at all.

I am going to look at the geomagnetic processes, compare the periods of the changing Earth's magnet with some historical climate data, and so on. It's clear that if this periodic signal in the climate data were due to geomagnetic effects, there could be many more geomagnetic effects with different cycles that impact the climate, too.

P.S.: Strength of the 44.9-month cycle

You may want to know how big changes of the temperature the cycle is generating. I found the strongest (Pythagorean hypotenuse) effect for the periodicity 44.9 months. This is the code:
perio = 44.9;
averagetemp = Total[b]/436;
sines = Table[Sin[2*Pi*i/perio], {i, 1, 436}];
cosines = Table[Cos[2*Pi*i/perio], {i, 1, 436}];
fit = Normal[LinearModelFit[{Transpose[{sines, cosines}], b - averagetemp}]]
If $i$ represents the month between $i=1$ for December 1978 and $i=436$ for March 2015 and not the imaginary unit, the temperature anomaly (centered to zero) may be fitted as$\eq{ \frac{T_i}{{}^\circ{\rm C}} &= 0.1068 s - 0.0575 c\\ s&=\sin \frac{2\pi i}{44.9}\\ c &= \cos \frac{2\pi i}{44.9} }$ If you combine the sine and cosine to a shifted sine, the amplitude would be the (Pythagorean) 0.1215 °C or so. This is rather nontrivial. Every 3.7 years, the temperature goes up 0.12 °C from the baseline and down by –0.12 °C in the middle of the cycle. The latest maximum (warm peak) was the 18th month from the end which, if I can count, was 5 months before March 2014 i.e. October 2013. In August 2015 or so, there will be a minimum of this periodic function.

So most of the year 2015 seems to be in the coolest phases of this cycle. This negative contribution may reduce the warming effect of the El Niño that was recently reborn (and that is predicted to become very strong) on the global mean temperature.

The (warm) maxima of the sine-combined-with-cosine (i.e. shifted sine) occurred in February 1980, November 1983, August 1987, April 1991, January 1995, October 1998, July 2002, April 2006, January 2010, October 2013. These cycles could have helped 1998 and 2010 to be among the warmest years and 2008 to be a cool one, and they may prevent 2015 from being the warmest satellite year despite the El Niño.

BTW I also calculated the Fourier transform of the GISS data since 1880 and the local peak periodicity was about 43 months even though it looks much less exceptional to me. I couldn't see any waves in the Correlation in the GISS data at all which strengthens the possibility that the 44-month periodic signal is a satellite artifact. Or maybe there's too much noise (inaccuracy of older measurements) in GISS.

Someone else mentions the 45-period in the data and suggests it is ENSO-related.

snail feedback (64) :

http://www.solarham.net/plane.htm

reader Luboš Motl said...

That's a pretty interesting picture and ideas. I don't realize that I have ever heard about the "interplanetary magnetic field" at all. Is it stronger than the Earth's magnetic field around the Earth?

reader Bernd Felsche said...

Just a note of caution:

If you see a pattern in the data, first make sure that it's not an artifact of how things are being measured and what is actually measured.

I'm not saying that that is the case here.

There are, as you say, many things that could hypothetically influence terrestial climate. In fact, one would have to be extremely narrow-minded not to see that there must be more than just a few factors; which probably inter-twine in the usual, natural, non-linear and sometimes not even monotonic manner.

reader Luboš Motl said...

Good point and yes, I had a worry that it's some instrumental cycle linked to the satellites. I don't think so, however.

First of all, it is seen both in RSS and UAH. Second, one may see it even in the weather station data although the amplitude is weaker (see Eschenbach).

The cycles they have to deal with are diurnal (daily). I am not aware of any 3.7-period cycle that could be source of systematic errors in the satellite measurements.

Another possibility that is ignored is perhaps processes in the sun and earth magnetic field are synched to something else outside the solar system. Suppose some other cosmological phenomena modulates certain cosmic rays which "catalyze" processes in the sun (say shifting the balance of UV, magnetic energy, etc produced).

reader Luboš Motl said...

A great extra possibility that shouldn't be overlooked. A correlation between A and B, if significant, may be due to A's influence on B, B's influence on A, or some C's influence on both A and B. ;-) Well, what is this C? Where is it? :-))

The Sun orbits the center of mass of the solar system, which moves into and out of the Sun's surface at regular intervals. It has been proposed that this introduces tidal mixing into the Sun's interior that might affect the rate of fusion. Here's a link to some older literature:

PS. No correlation with sunspot numbers, however.

Slightly off topic. Someone posted a link in this very blog sometime ago, to an interesting essay, "Earth’s Orbit and Contemporary Climate Change":

http://www.duncansteel.com/archives/996

where the author calculated the changes in insolation due to orbital precession, and provide an explanation for why sea ice is melting in the Arctic, but growing in the Antarctic. Despite the zero impact of this careful analysis, I think it's interesting to share here and have a look.

reader Per Strandberg said...

It the solar node cycle. The solar node cycle has a preciosity of 18.612958 years.
If you take (18.612958*12)/44.6 = 5.0079. It’s the Moon, not the Sun!
I’ve cracked the ENSO code which is almost exclusively driven by the tidal gravitational variations and from electromagnetic variations of the Sun. I have explosive data about this which I’m about to publish. The effect then on the temperature is through ENSO which is driven by extraterrestrial forcing by the Moon and the Sun, nothing to do with man.

reader S Graham said...

Interesting work, thanks.

re: 'it seems surprising to me that the possible geomagnetic influences on the climate are not discussed at all'

This is not quite correct. Google search this:
'site:tallbloke.wordpress.com geomagnetic'

Many solar cycles peak about a third of the way through i.e. close to 44-45 months.

"..monthly sunspot number for 1749-2009 years exhibit strong periodicity with a period approximately equal to 3.7 years."
http://arxiv.org/pdf/1004.4639v2.pdf

"It should be noted that the same period ∼ 3.7 years was recently found for the so-called flip-flop phenomenon of the active longitudes in solar activity"

"..supports the idea of the one-third subharmonic resonance as a background of the 11-years cycle of solar activity."

Now it seems that the unexplained period of 3.7 years moves everything:)

reader Uncle Al said...

"you discovered America" Discovering a wheel is no biggie. Trees are stacked wheels. Putting two wheels on an axle is a biggie.

reader Gene Day said...

Nothing physical happens when the center of mass happens to pass through the sun’s surface, Bob. This is pure science fiction.

reader Gene Day said...

That last paragraph is actually insulting, Lubos. You should be offended. In fact, the entire “note” is a bit pompous in tone.

Many things can influence the results of climate science. One of which is cooking the data.
There may be many factors but I won't consider them when it looks like fraud explains most global warming pronouncements.

http://www.telegraph.co.uk/comment/11561629/Top-scientists-start-to-examine-fiddled-global-warming-figures.html

could you please elaborate on your calculation? why 12? and why would the lunar orbit precession affect the earth's magnetic field when the moon crosses the ecliptic once a month anyway?

reader John Archer said...

"Nothing physical happens when the center of mass happens to pass through the sun’s surface, Bob. This is pure science fiction."

If the sun were a rigid object I'd have no hesitation in agreeing with you, but seeing as it's a fluid I don't find that obvious at all, especially with lots of charged particles and resulting magnetic fields whizzing around. Messy! :)

I see that the (chaotic looking) orbital period of the sun about the solar system's barycentre is very slow (order of a decade) compared to its (equatorial) rotational period (less than a month) — say a couple of orders of magnitude in the difference between the two. Is that enough for any effect to be negligible? But even if it is, it's not obvious that it can be ignored in principle, which seems to be what you are saying if I'm not mistaken.

By the way, I'm not so much concerned about it going through the surface as such but the fact that it the sun's centre moves with respect to it at all.

Dunno, but then I'm no physicist.

Obviously you would know this better than I would, but since the upper atmosphere is full of ions and polar molecules, couldn't changes in geomagnetic activity (hypothetically) effect their behavior?

Have you asked Dr. Spencer what he thinks of this? I sense a guest-blog...

What if we are in on the border of some CMB temperature variation bubble?

Ah, fluctuations are of the order of one part in a million so forget it.

reader Bernd Felsche said...

Not "unpredictable", Luboš. Only "mysterious" until it's identified.

The cycle which you're seeing could be e.g. a beat frequency of two factors. Or it could be one. Until you identify the physical source; it's speculative.

But it is still useful because it is a predictable pattern; regardless of its physical cause(s).

reader Bernd Felsche said...

I'm offended, Gene. Insulted! You appear to be more sensitive than climate models are to CO2. :-P

I greatly respect Luboš intellectual capacity and capabilities as a physicist. I also regard him as a nice bloke, having travelled half way around the world to have lunch with him.

If what I write appears "pompous", then it's because I'm not here to give back-rubs. Intellectual stimulation is by "unsettling thoughts". They are, by their nature; "uncomfortable".

reader Luboš Motl said...

It would surely be amazing if it were not the kind of fairy-tale that it sounds like. ;-) Good luck.

I had to look in my climate related Library and found this paper from 2010 : http://cdn.intechopen.com/pdfs-wm/11434.pdf

It identifies 2 periodicities, 1 of which being 43+/- something. As it builds up on other papers, it seems these periodicities were known for a long time.
What is striking in your graph is that the first 3 periods are not sin (perhaps f(t) + sin) and show only positive (strong) correlations before a negative correlation first shyly appears in the 4th period and the graph starts to look like a sin.
This made me think about critical shifts in chaotic oscillations.
As the strongest chaotic oscillation impacting the climate is ENSO, I would suspect that ENSO has something to do with it.
.
With your l33t skills with Mathematica it would certainly be a child's game for you to do what you did separately for the Northern and Southern hemisphere.
If this 45 month periode is seen more S than N then the suspicion about ENSO deepens.
Of course the easiest proof would be to remove ENSO but nobody knows how to do that correctly.

reader Per Strandberg said...

Thanks :=)

I just came by your site and saw your calculation, yesterday.

Because I know the cause of the drivers of ENSO and that
cycles connected to ENSO most likely are connected to lunar cycle, I made a
quick calculation. 5 may seem like an odd number. Well, 5 is an odd number mathematically.
But the fact that your cycle times 5 corresponds to the lunar node cycle make
full sense to me based on the mechanism I found.

I’ve used an ANN program
I created, which I used to play around with, with solar cycles, tidal effects
and sea current indexes.

I discovered the connection with tidal and solar forcing on
ENSO a while back, which I’ve presented to others.

I was preparing to write a paper on this, when, last month I
used a new approach.

The result I then got was extraordinary.

Look at this!

http://www.global-warming-and-the-climate.com/images/ENSO-solar-tidal-impact.jpg

Wow!

The training period is from 1979 until 2005, the testing
period is from 2005 until 2012. The rest is forecast. The data is based on the
MEI ENSO index.

The only inputs I use are from tidal gravitational anomalies
and from Kp, Ap and variations in solar wind parameters.

Most so called skeptics refer to the Svensmark’s effect when
it comes to the connection between global temperature variations on the Sun.

Critics from the alarmists’ camp often point that this
effect is too weak and I think they are right on that.

I can show with empirical evidence that one of the main
causes of recent temperature changes, directly can be attributed to
electromagnetic solar driven ENSO variations.

If one take into account this ENSO effect and the data fiddling
of the main surface temperature station records by NASA, NOAA and from CRU
hadley centre, then there is not much space left for any AGW influence on
recent temperature changes, irrespective if such effect exists or not.

I realize that what I have, is an atomic bomb set to explode
in the face of climate scientists, ENSO researchers and of course ultimately
this is going to be the (beginning) of the end of the CAGW hysteria.

I would also point out that the mechanism I found is chokingly
simple and should therefore be relative easy for others to replicate.

BTW Do you have any idea of I should go about publishing my
findings and results?

a tree on its side is two wheels connected by a fat axel, no?

reader John Archer said...

Ah! Literal lateral thinking!

Now with added alliteration! Buy one, get one free — while stocks last! :)

18.6 years x 12 months a year / 44.6 months. i haven't any clue for the other question.

maybe try explaining this magic "mechanism" you found before proceeding with the autofellatio?

reader Tom Trevor said...

Its what you do with a tree on its side that counts. If you just let sit there big deal. If you roll it and use it to help you move something, now you have done something.

Per, Please join the Azimuth Forum, http://forum.azimuthproject.org, where you will get a receptive response to your ideas. An ongoing project is to try to predict El Ninos.

Plotting it up as a time series, the cycle looks spurious. As I expected, before even plotting the series up, there was a peak in ca 1990 and a trough in ca 1993. Why did I expect that before plotting it? Because I was pretty dang certain it would turn out to be the case that some of the effect of the eruption of Pinatubo in 1991 was being attributed erroneously to the alleged cycle. Having previously estimated the effects as present in the land surface data:

Bah dang it how did I accidentally send that without finishing it. Anyway:
https://devoidofnulls.wordpress.com/2013/11/07/can-you-isolate-a-volcanic-temperature-signal-in-the-temperature-data/
I estimated the volcanic effect before. It looks to me like the response in the troposphere is even larger. I imagine that accounting for the volcanic effect would pretty much eliminate the apparent cycle.

Ah I ran a quick regression and actually it seems less spurious than I thought it would be? The cycle does seem to improve the fit without causing the volcanic effect to be smaller than expected-that is, about 1.2 times as large as it is at the surface. In fact, the coefficient was so close to what I expected I'm now nearly convinced the cycle is real.

reader anna v said...

Have you seen this article at Wattsup? http://wattsupwiththat.com/2015/04/30/how-plasma-connects-the-sun-to-the-climate/

reader Luboš Motl said...

Dear Werdna, how could one eruption produce the number 3.7 years, the length of the cycle?

The very fact that the autocorrelation contains many - eight - peaks shows that this can't depend on one random event, and not even 2 events separated by 3.7 years, for example, because in that case, only one peak would show up in the autocorrelation graph, not nine

Have you considered the opposite explanation that something cooling that is actually due to the 3.7-year cycle was erroneously attributed to the volcano?

reader Luboš Motl said...

Yup, the article is very short - sure, plasma is here and probably important for something - but I am missing more information and not sure what links I should go to. It is not a good summarizing article.

Lubos-I've already concluded that my initial supposition was erroneous, however I don't think it was an outlandish supposition. If the eruption happened to be timed in such a way that other random events, especially ENSO, aligned with it just right, it could have created a trough in just the right place to create a strong apparent cycle where none was actually present. I'd have to run tests to see how likely or unlikely that actually is, but that's probably not necessary since a quick regression analysis based on my previous estimates of the volcanic effect suggested that accounting for volcanoes would not significantly change the cycle's amplitude (I got a coefficient for that variable very close to 1).

To be honest when I see someone finding a pattern like this my instinct is to be pretty skeptical of it, so I suggested a way it could have arisen in the data by chance and then I decided to test to see whether I was right. Turns out I wasn't, so that's good for the cycle idea.

reader Luboš Motl said...

Sorry, Werdna, but you still seem to ignore my point.

One-time event is a very different thing than a periodic behavior. It doesn't have any characteristic scale. It just creates something different in all the graphs. It wouldn't create any periodic wiggles in the autocorrelation, with ENSO or without ENSO.

It would have to be a statistical fluke, essentially. In retrospect my instincts of the probability for such a fluke to occur were probably bad.

But that's essentially what a spurious correlation *is* after all.

reader anna v said...

Did you check the link at the bottom? https://drive.google.com/file/d/0B45V9V-NrtCIcXRYak8zT012ekk/view?pli=1 ? at the end there is a long list of references. I am a bit wary when I here "plasma" because ot the electric universe business , but the talk discusses earth-sun.

Well what I find shocking is that if you shift the data by (approximately) 2, 2+4 =6 and 6+4= 10 years you obtain a positive and very similar correlation (e.g the temperatures move in the same direction).
Then brutally when you shift it by 10+4 = 14 years you obtain a negative correlation (e.g the temperatures move in opposite direction) and then it continues like that for shifts of 18, 22 etc years.
So something pretty important must have happened around (10+14)/2 = 12 years shift to change the correlation sign.
That made me think of some much longer period (between 10 and 12 years) that modulates the shorter 4 years period.
And as this is the kind of thing that one (qualitatively) sees in ENSO indexes like SOI, it made me think of ENSO (pseudo period around 4) which would be modulated by something longer.
.
I was also thinking about another check - I don't know if it is easy to do with your data.
If you can eliminate the +30°, - 30° latitude band from the data and then redo what you did on the remainder of the sphere, you should see the signal weaken or disappear if ENSO which mainly happens in this band is the main culprit.

reader Luboš Motl said...

Dear Tom, there are surely other cycles apparently contributing, but a 12-year cycle is already too long so that one only gets 3+ periods in the satellite era, and it's too little to "safely" distinguish it from random wiggles.

Do you have Mathematica (or some other software)? You may want to buy it. Home editions are comparable to hundreds of euros. It could be easier to help you to get familiar with it than to rerun it to try all your experiments...

reader Luboš Motl said...

Dear Werdna, are we talking about the same thing? Have you read at least a part of the blog post above? Have you seen the graph of the autocorrelation?

It can't be described as a "spurious correlation". If it is spurious, then it is a "nine different spurious correlations carefully adjusted so that they are almost equally far from each other".

reader Luboš Motl said...

It looks like a very informative slide show - although a bit non-quantitative to my taste and not explaining any particular climate (or similarly relevant) terrestrial data...

reader W.A. Zajc said...

Lubos, I am again impressed by your ability to write very compact Mathematica expressions to perform such analyses. Like TomVonk, I too was puzzled by the fact that your BarChart[core] plot is positive-definite up to 150 months, then shows the oscillatory behavior. The first ~20 bins showing exponential decay make perfectly good sense: “this month will be a similar to last month”. But that decay constant of about 10 months is too steep to account for the completely positive correlation in the first two peaks, so some more complicated modulation will be needed.

The main point, the periodicity of about 44 months, is indeed striking. I used Mathematica’s Manipulate command

Manipulate[ListLinePlot[{b[[1 ;; -1 - k]], b[[1 + k ;; -1]]}], {k, 1, 400, 1}]

on the kernel of your correlation calculation to look by eye at various shifts of the temperature data. It’s quite easy to spot the maxima in the correlations (and anti-correlations). Here are two plots corresponding to shifts of 250 months (left) and 335 months (right):

reader Luboš Motl said...

Thanks for reminding me of Manipulate. ;-)

All the credit for the compact expressions must go to Stephen Wolfram and his folks. I can't be praised even as a very good user! ;-)

Just to be sure - your graphs show a very good anticorrelation because the lag is 5.5 or 7.5 cycles, right?

reader W.A. Zajc said...

Yes, exactly. By doing a "single-blind" analysis, I was able to vary the shift with Manipulate[ ], find by eye a shift value that seemed to maximize the anti-correlation, and then confirm that it corresponded to a minimum in your BarChart[core] plot.

These (anti)-correlations are so strong that I would be inclined to look for some instrumental effects. The link to the Willis Eschenbach post you provided is interesting, but I disagree with his statement that the 44-month signal seen in the UAH data is also visible in the HadCRUT4 data (looks consistent with noise to me). However, working against my prejudice of an instrumental effect, Fer137 provides a link to a not obviously wrong paper by Bershadskii that shows a strong signal at the 1/3 subharmonic ~ 3.7 years = 43 months of the sunspot cycle. So somewhat to my surprise (I too have been skeptical about the sunspot connection), there may be a relation when one properly accounts for this SUBharmonic of the cycle.

reader Luboš Motl said...

Dear Bill, minutes ago, before your new comment :-), I finally sent a mail to Spencer asking him about the instrumental effect that could explain it.

reader Luboš Motl said...

Roy Spencer answered almost immediately. He either doesn't know the answer or wants to treat me as a student. ;-) His reply is "apply the same thing to the ENSO index and you will see what you get".

I actually wanted to do it, too, but the ENSO bastards have removed the ENSO index from my favorite weekly ENSO PDF file so I don't immediately know where to find the ENSO indices. ;-)

reader W.A. Zajc said...

This is the first time I ever played with climate data, made possible by your posting the precise Mathematica commands to input and manipulate same (thank you!).

While it is fun to do so, I am glad to see you are going to do the homework, as I need to do my real homework, which is to write up solution sets for a homework assignment in my course ;-)

Hehe Lubos you already forgot !
I bought Mathematica on your advice when I was trying to solve the heat equation with radiation boundary conditions some 2 years ago.
We then had a mail exchange to only find out that Mathematica was not able to deal with PDEs with Neumann boundary conditions. I was quite disappointed and then decreased my use of it.
So I do have Mathematica, am at best an average user and use it not frequently enough to spend time improving my skills.
In the post above I was just thinking aloud (apparently isomorphically to what you reported later about R.Spencer) but had no intention to learn enough Mathematica to do the +30/-30 elimination I thought about :)

reader Luboš Motl said...

Dear Bill, I sent you a copy of some e-mails with Roy Spencer to an e-mail of yours. Sorry if it doesn't work or if you don't want it.

Roy resent it to Christy and Braswell and they will discuss it now.

HadCRUT4 since 1850 shows no significant wiggles - just the uniformly decreasing correlation. HadCRUT4 since 1979 shows some wiggles similar to UAH/RSS but they are vastly weaker and their regularity, while somewhat present, wouldn't catch my attention.The amplitude may be up to 5 times weaker, I think.

ONI ENSO index since 1950 or 1979 shows lots of wiggles of autocorrelation around zero but they're in no way regular or periodic or equally large.

reader Luboš Motl said...

Good luck with your metahomework. :-)

The importing of files in different formats may be messy but it's a deformation of a sort: I got rather efficient in importing arbitrarily formated numerical data into Mathematica. Take it as a table, find the right indices, table subfields, join them, flatten them, and so on, I can do it almost automatically regardless of the messy disagreement in the conventions.

Importing graphical data is much worse - like reverse-engineering colors in various density plots etc. But I've done numerous things of this kind, too.

If you wish, I can send you hundreds of megabytes of various Mathematica notebooks that import and play with various climate and other data.

reader Luboš Motl said...

I know you did buy it, but I couldn't know whether you un-bought it or something like that. ;-)

If you wanted, I can send you any Notebook I have...

Lubos-Yeah, sorry for wasting your time. In retrospect I was just being stupid.

reader Luboš Motl said...

No prob, Werdna, always appreciate when you look at things.

The problem with matching perodicities is that there are so many to choose from that there is a significant hazard of matching the wrong ones. Then, there is the question of correlation versus causality.

The two phenomena could be linked to a common causal factor. E.g., it might be expected that the solar and lunar generated tidal gyrations of the Earth would affect the internal dynamo, as well as heat dissipation from interacting ocean dynamics. This could produce roughly 5 year (could be 44 months) and 60 year temperature cycles.

Thanks Lubos. While I was at it I decided to examine v6.0 itself, and I'm a bit concerned because I'd previously identified what I thought was a problem with RSS during the transition period of NOAA-12. The problem is that UAH agrees with RSS over that period now-in fact it's worse:

If this is the only problem with v6.0, or at least the largest in magnitude (assuming it even is a problem, but I do think it is) it indicates v6.0 may be overestimating the long term trend a bit. On the other hand I'm also doubtful about UAH's new cooler drift since 1998, associated with the AMSU's. Again the corrections bring UAH closer to RSS's behavior, and I previously found UAH's arguments why UAH was probably better over both periods convincing so I'm a little worried UAH has updated their method but taken a step backwards in quality.

I've left a comment on Roy's blog to see if he can address my concerns.

On the plus side, the problems don't appear to connect to the cyclical pattern which appears to make it doubtful it's an artifact of the method of combining satellite data.

reader W.A. Zajc said...

Dear Lubos, thanks for the kind offer of your various Mathematica notebooks. There may be a time when I take you up on this offer, but for now I think it better that I concentrate on my day-job.

I did learn a lot about importing date by looking at how you imported the UAH .txt file; in particular the indexing structure [[2;;-13,3]] puzzled me greatly until I did my homework ;-)

reader Luboš Motl said...

Dear Bill, I got 70% convinced that this periodicity is an artifact due to the combination of the leap year drift - note that the cycle is close to 4 years - and a trace of their diurnal drifts.

BTW Roy Spencer told me to do the same with the ENSO data. Here's the funny more complex indexing protocol to get it quickly.

The data at

http://www.cpc.ncep.noaa.gov/products/analysis_monitoring/ensostuff/ensoyears.shtml

is messy, inside HTML tables inside a text, and so on. So I just copied-and-pasted whatever is left in the table to a TXT file and the real data from the TXT file was imported by

oniRAW = Import[
"C:\\Users\\Luboš\\Dropbox\\Lumo\\Mathematica\\2013\\Climate\\oni-from-noaa.txt", "Table"];

oniRAW2 = oniRAW[[29 ;; -1 ;; 2]];

oni = Flatten[Table[oniRAW2[[1 + i*13 ;; 12 + i*13]], {i, 0, 64}]]~Join~{0.6, 0.5}

Note that 29;;-1;;2 goes from the 29th to the 1st from the end by jumping over 2 - skipping each even entry. Then one has to omit the first column, by taking table of arrays, and flattening them, and manually adding the first two 3-month figures from 2015.

reader W.A. Zajc said...

Dear Lubos, the above indexing gymnastics argue strongly for a standard format for tabulating climate data!

Your leap-year hypothesis is very intriguing. I wish you the best of luck teasing this out of the noisy data. I did slightly extend your methodology in creating your "coreD" plot in this way:

coreD = Table[Total[core[[1 ;; -1 ;; i]]]/Length[core[[1 ;; -1 ;; i]]], {i, 1, 300}];BarChart[coreD,
Frame -> True, FrameTicks -> {{0, 50, 100, 150, 200, 250, 300}, {0.0, 0.1, 0.2, 0.3,
0.4, 0.5, 0.6}}, Epilog -> {Red, PointSize[0.01], Point[peaks]}]

where "peaks" is the output of this command:

peaks = FindPeaks[coreD, 6]

{{45, 0.339903}, {89, 0.444845}, {135, 0.445665}, {161, 0.355105}, {178, 0.495967}, {229, 0.505041}, {266, 0.583083}}

You see mostly multiples of 44-45 months; I leave it to you to determine if the peak at 161 months is also significant:

reader Steve R said...