Why the ‘oil price’ isn’t always the oil price

The writer is a former banker and author of several technical books on derivatives as well as Traders, Guns & Money, Extreme Money and the upcoming The Everything Bubble.

In 1936, John Maynard Keynes was almost forced to take delivery of an entire month’s supply of Argentine wheat bought via forward contracts after whilst speculating on the commodity’s price movements as manager of his college’s endowment. Whoops.

The famous economist contemplated storing the enormous amount of grain in the crypt of the King’s College Chapel at Cambridge, but that proved too small. Instead, Keynes cleverly avoided the problem by objecting to taking delivery on quality grounds. The delay for mandatory cleaning and inspection allowed the wheat market to recover, and let Keynes offload the shipment without ever having to physically take delivery and store it.

The Iran war has now once again starkly highlighted the differences between physical and paper commodities, the distinction between availability and price, and the real-life complications that can often intrude when handling real stuff rather than purely financial securities.

With limited data about dealings in the spot market for immediate delivery of oil, coverage has focused on the near-month futures contract price. But while the futures price hovered around $100 per barrel during the most intense kinetic action, actual cargoes were changing hands at levels 80 to 100 per cent higher. Diesel and jet fuel prices were at a similar premium, reflecting widening refining spreads and shortages in key locations.

Politicians, eager to downplay rising energy costs, emphasised the lower futures prices rather than actual physical costs. They mouthed specious arguments that it meant that prices would shortly come down — ignoring the fact that futures prices are poor indicators of actual prices.

To be fair, the relationship between spot and futures prices in oil, as for most commodities, is actually quite complex.

Unlike financial assets, such as currencies, interest rates, and equities, oil is physical and not homogenous. There are different grades, which work differently for different refined products, like gasoline, diesel jet fuel, heating oil or bunker fuel. Producers, refiners, and consumers are interested in availability for delivery at specific locations.

There are storage, insurance and transport costs. Commodities are also affected by changes in the asset — transportation and storage seepage, for example, or change in quality. Production and consumption are driven by complex factors and subject to (often large) time lags. Price behaviour exhibits mean reversion and seasonality.

Importantly, the basic arbitrage-based futures pricing model doesn’t hold. As the futures price is agreed today to be paid at a deferred date, a forward seller could (if the future prices are higher than the current spot price) purchase the required quantity of the asset immediately, fund and store it, then deliver on the agreed forward date.

A forward purchaser could replicate the transaction by selling in the spot market and investing the proceeds until the agreed future date, when the maturing investment could be utilised to fund the purchase. The cost of the immediate purchase or sale, funding and storage until delivery determines the futures price.

In theory, this standard model — adjusted for quality differences, insurance and transport costs — could be used for oil. In practice, this is difficult because the institutional structure of oil markets impedes the process where the future price is forced to this equilibrium level through arbitrage.

One problem is that physical oil is difficult to short. Mechanisms common in financial assets, like borrowing the asset for short-selling, are not widely available. Only producers and some traders are able to sell cargoes that they do not own on a limited basis.

Another is that the participants in the physical and futures market differ. While they use futures to hedge prices, producers, refiners and consumers need the actual oil. Financial investors use the futures market to simply trade price movements. This avoids the need to hold and deliver the physical commodity, as the contract can simply be closed out and settled in cash before maturity without any obligation to deliver the goods.

But like Keynes and his shipment of Argentine wheat, they are loath to actually take delivery. In 2020, the price of West Texas Intermediate oil famously went to minus $37.63 a barrel as investors were desperate to avoid having to take and store oil amid a huge global glut and a storage shortage. In effect, purchasers were willing to pay sellers to avoid having to physically hold oil.

The ability to use futures to go short — in expectation of price falls — is also appealing to speculators. Another consideration is liquidity. Daily oil derivative trading volumes are up to 60 times physical oil trades.

These factors mean that actual futures prices for commodities don’t follow arbitrage models and are often lower than the spot price — or in “backwardation” in the industry argot. Keynes argued that the long-run average futures price for a commodity will, in general, be below the long-run average spot price, reflecting the fact that producers are prepared to pay a positive insurance premium (the convenience yield) to protect against unforeseen adverse price movements. The concept is equivalent to income or yield accruing to the owner of a physical commodity.

Keynes’ premise was that the participants in the supply chain have asymmetric risk aversion. Refiners, distributors and some consumers might not hedge, as they can pass on price fluctuations. In contrast, producers incur high upfront capital expenditures and bear greater financial risk from lower prices, encouraging hedging. This gives futures prices a downward bias, with producers willing to sell at a discount to the theoretical futures prices.

The level of backwardation can be exacerbated by shortages of stock available for immediate delivery from an adverse supply shock — as in the oil market as during the Iran war — or a positive demand shock. This reflects the ‘possession’ value, as refiners and consumers hoard available stocks as a security measure. But higher prices should choke off demand and lead consumers to seek substitutes, while producers should increase output.

This depends on the commodity’s price-volume elasticity — the ability of producers to adjust production — and time lags in adjusting demand and supply.

Backwardation therefore tends to be greatest in markets where commodity prices are volatile, producers are sensitive to commodity price fluctuations, inventory costs are high, and there’s limited surplus production capacity. The oil market has many of these characteristics, as we saw again this year.

Meanwhile, global energy intensity has fallen from 131 litres per $1,000 of GDP at 2025 prices in 1973, 116 litres in 1980 to 52 litres today. While this means that the average oil burden is 60 per cent lower than 50 years ago, oil usage is concentrated in critical areas without easy substitutes — like freight transport — which are less price sensitive and non-discretionary, meaning that the elasticity of demand is reduced.

Similarly, the recent lack of investment due to the transition from fossil fuels means the ability to increase production is limited.

Saudi Arabia, historically the main swing producers, is in a fraught neighbourhood. Russia has limited surplus capacity and is involved in the Ukraine War which exposes it to attacks on its oil infrastructure. The US, which has stepped up exports, is also constrained. Shale liquid producers have been reluctant to expand output. Scarred by past market crashes, they are worried that higher prices may not be durable. As a result they are focused on capital discipline and shareholder returns, rather than increasing output.

Regardless, any expansion in US drilling cannot compensate for the loss of 12mn barrels of oil a day from global supplies due to the closure of the Strait of Hormuz, if this happens again.

The level of the convenience yield is affected by traders looking to earn the convenience yield by buying distant and selling near-term futures contracts to lock in price differences. This is sometimes repackaged as a low-risk investment with high leverage to enhance returns. The need to roll these contracts to avoid physical delivery as they approach maturity can distort prices.

Backwardation isn’t a permanent state. Oil future prices are sometimes higher than the theoretical level — known as “contango”. This occurs especially when there’s a sudden excess of supply causes spot prices to collapse. That may occur when oil trapped in the Gulf reaches the market, but the phenomenon is unlikely to last long. In April 2020, “super contango” caused WTI oil prices to briefly go negative, as supply overwhelmed available US storage capacity, but a month later WTI was back at $33 a barrel.

A CME study found that, since 1985, the WTI crude oil market has been in contango and in backwardation around 42 per cent and 58 per cent respectively (using the price difference between the front-month contract which traded close to spot prices and contracts six months in the future).

In his 2023 book Virtual Barrels, Ilia Bouchouev argued that the spot price of oil is a function of the derivative not the other way around. While true under certain circumstances, in times of war or extreme uncertainty, the focus is on availability, not price — meaning high possession value or dealing with oversupply makes the futures price less relevant.

As Mohammad-Bagher Ghalibaf, the speaker of Iran’s parliament, tweeted in March 2026: