Compare S1 results between ECMWF and METplus

[wuxx66]

I am comparing WMO required MSLP S1 results between our legacy system and METplus. We uses ECMWF "Verify" package to calculate S1. METplus S1 values are consistently about 2–3 units larger than those from ECMWF verify, while RMSE, ME and ANOM_CORR are generally very consistent.

ECMWF "Verify" uses grid-space finite differences (WMO definition). I think MET computes projection-aware physical-space gradients. Is this correct? So the differences is because of different definitions of horizontal gradients?

I have attached a S1 comparison plot between ECMWF and METplus. There are 9 WMO standard verification domains in the plot. Almost all of them METplus has consistently larger MSLP S1 than ECMWF. The patterns are very similar.

Thank you

Xiaoxi

[j-opatz]

Hello Xiaoxi,

And thank you for your question. As you suspected, the culprit to these differences is in how the two methods calculate the gradient.

MET's approach is more real-world, calculating the actual spatial derivatives for the forecast values in the x and y directions. This involves including the physical distance between gridpoints, which can change by projection and latitude.

The ECMWF method ignores the physical aspects, instead using just the adjacent grid points. This aligns with the WMO definition, but does reduce the gradient calculation to essentially |f(x+1) - f(x) | + |f(y+1) - f(y)|; the gridpoints around the evaluation point are always 1 gridpoint away.

We want to consider how to handle this better, or at the very least update our documentation to reflect this difference in methodology. We will be discussing this with our resident statistical expert and get back to you with our intended path forward.

Again, thank you for bringing this to our attention; we are always interested in identifying and adapting to any user-found discrepancies!

[wuxx66]

Thank you so much for your reply @j-opatz

[j-opatz]

Hi Xiaoxi,

I wanted to let you know that after speaking to our in-house statistician, we've determined that the best course of action would be to create a new statistical calculation of S1 that follows the ECMWF method and would be WMO-compliant.

To that end, I've created issue 3390 to track progress on this. Please note that due to funding constraints we won't necessarily be able to implement this right away, but the issue will be available for our developers to begin work as soon as funding and time allows.

[wuxx66]

Thank you so much @j-opatz. This is a really good news