News sentiment doesn't predict crypto returns. Two bugs in my own test almost said otherwise.

I build Cheruvo alone. It reads financial news, scores each headline from −1 to +1 from an investor's point of view, and puts the daily average next to the price. The whole thing runs on free tiers.

Six weeks ago I set up the test that could kill it: does today's sentiment tell you anything about tomorrow's return? I fixed the thresholds before looking at any result, because a threshold chosen afterwards is not a threshold.

The answer is no. The more interesting part is what I had to fix in the test before I could trust that answer, because one of those defects had been quietly manufacturing my best result.

The setup

Spearman rank correlation, not Pearson, because a single +18% day would otherwise carry the whole thing. Three horizons: next day, two days, seven days. Three horizons means three chances to get lucky, so the significance threshold drops from 0.05 to 0.017.

Two significance tests, and the difference between them matters later. A simple permutation shuffles the observations as if they were independent. A block bootstrap shuffles them in blocks of 2, 3, 5, 7 and 10 consecutive days, which preserves some of the time dependence. The p-value I report is the worst of the block lengths, never the best. Reporting the best would mean picking the block size that wins, which is the thing this entire exercise exists to avoid.

Days with fewer than 5 distinct stories are dropped: an average over two articles is an anecdote. Syndicated reprints of the same story count once. All news comes from The GDELT Project, whose terms permit unrestricted commercial use and redistribution with citation.

Bug one: the control that was correlating against zero

Alongside the forecast test there is a sanity check: does today's sentiment track today's price move? A positive answer there proves nothing interesting, but it confirms the pipeline works at all. The press describing what just happened is the easy case.

It kept printing rho = +0.000 on 38 days. Exactly zero, three decimals.

A number that round is not a measurement, it is a division that degenerated. The same-day case computed the return from the close of day T to the close of day T, so the two prices were the same price, and every return in the series was zero. I was correlating sentiment against a column of zeroes and printing the result as if it meant something.

That control is the instrument check for the whole analysis. It was supposed to show a clear positive relationship, and by showing zero it appeared to contradict the conclusion the program was about to reach. Fixed, the same-day correlation on Bitcoin is +0.338 over 44 days. The thermometer works.

Bug two: one direction was being graded easier than the other

The program always tests both directions, including the uncomfortable one: instead of asking whether sentiment leads price, it asks whether today's sentiment tracks the move that already happened over the preceding days.

That second direction was producing my strongest numbers. On Bitcoin, rho = +0.605 with p = 0.0001. On Ethereum, rho = +0.581 with p = 0.0004. Two independent assets, same direction, p-values far below threshold. It looked like a replication.

Then I looked at how those p-values were computed. Only the simple permutation. The forward direction had to survive the block bootstrap and the worst-block rule; the backward direction got the easy test, the one my own program prints with the label optimistic: it breaks the time dependence.

And it was the direction where that mattered most. The seven-day backward window compares today's sentiment with the move over the previous week, so two consecutive observations share six days out of seven. Under that much autocorrelation, a test that shuffles as if observations were independent returns p-values far lower than they should be.

So the strongest result in the project was computed with the only method the project itself declares unreliable. I put the backward direction through the same sieve as the forward one and ran everything again.

What survived

Bitcoin, 44 usable days, 5,285 stories.

The first row is not a third result: sentiment on day T against the move over the previous day is the same pairing as the same-day control above, so +0.338 appears twice by construction rather than by coincidence. It is worth stating, because a number that shows up in two tables usually means someone pasted it there.

There is a shape in there worth noticing. The correlation climbs from 0.34 at one day to 0.61 at two, then stops. It is not today's move that the news reflects, it is the move accumulated over the last two or three days. That is more specific than "sentiment follows price", and it is the part I did not expect.

What did not survive, and why that is not the same as a failure

Ethereum was supposed to be the replication. Under the block bootstrap its seven-day backward correlation goes from p = 0.0001 to p = 0.083, five times above threshold, with the block-length warning attached. It does not hold.

Before calling that a failed replication, look at the volume. Bitcoin: 5,285 stories over 44 days, about 120 a day. Ethereum: 926, about 21 a day. Ethereum's daily sentiment is estimated from a fifth as many articles, so it is far noisier, and noise in the variable attenuates any correlation that exists. Its same-day control is +0.122 against Bitcoin's +0.338, which is consistent with exactly that.

So it is not a failed replication. It is an underpowered one. The distinction matters, and collapsing it would be its own small dishonesty.

The limits, plainly

• 44 days. Short. Only large effects are visible at that length.
• The positive finding rests on one asset.
• The backward windows overlap heavily, which is the whole reason the block bootstrap is not optional here.
• Equities are still pending: stocks do not trade at weekends, so 44 calendar days become 24 aligned observations, and the test refuses to speak below 30.
• None of this says that news sentiment can never predict returns. It says that on this dataset, this universe, this period and this scoring model, it does not, and that I have no grounds to claim otherwise.

What I did with the answer

The honest sentence was already on the homepage before the test finished: a thermometer, not a barometer. What changed is that it is now measured rather than asserted.

It also cost me the one sentence that would have made this easy to sell. That is fine. The alternative was to keep a claim that the first person to check would take apart, and at that point nothing else I got right would count for anything.

What I keep coming back to is that both defects pointed the same way. The broken control made the product look worse than it was; the lenient test made my favourite result look stronger than it was. Fixing them killed the number I wanted and kept the one I had not asked for. That is roughly what fixing your own statistics is supposed to feel like.

This article is for information and education only and is not financial advice. Past performance and historical correlations do not guarantee future results.