Inconsistent number of time steps returned for 2d field and point requests

[koldunovn]

What happened?

If I request 2d fields for one day from story line simulations:

request = {
    'class': 'd1',
    'dataset': 'climate-dt',
    'activity': 'story-nudging',
    'experiment': "cont",
    'expver': '0001',
    'model': 'IFS-FESOM',
    'generation': '1',
    'realization': '1',
    'resolution': 'high',
    'date': '20170101',
    'time': '0000/to/2300',
    'stream': 'clte',
    'type': 'fc',
    'levtype': 'sfc',
    'param': '167'
}
data = earthkit.data.from_source("polytope", "destination-earth", request, address="polytope.lumi.apps.dte.destination-earth.eu", stream=False)

I get back 4 time steps:

If I request time series for one point from the same dataset:

%%time
location = [[38.88655345978706, -9.119280931080349]]
request = {
    'class': 'd1',
    'dataset': 'climate-dt',
    'activity': 'story-nudging',
    'experiment': "cont",
    'expver': '0001',
    'model': 'IFS-FESOM',
    'generation': '1',
    'realization': '1',
    'resolution': 'high',
    'date': '20170101',
    'time': '0000/to/2300',
    'stream': 'clte',
    'type': 'fc',
    'levtype': 'sfc',
    'param': '167',
    "feature": {
        "type": "timeseries",
        "points": location,
        "time_axis": "date",
    },
}
data = earthkit.data.from_source("polytope", "destination-earth", request, address="polytope.lumi.apps.dte.destination-earth.eu", stream=False)

I get hourly data:

What are the steps to reproduce the bug?

In the main body

Version

0.10.1

Platform (OS and architecture)

Linux levante5.lvt.dkrz.de 4.18.0-513.24.1.el8_9.x86_64 #1 SMP Thu Mar 14 14:20:09 EDT 2024 x86_64 x86_64 x86_64 GNU/Linux

Relevant log output

Accompanying data

No response

Organisation

No response

[awarde96]

This is because the feature extraction and the global field extraction expand the requests in a different way. In FDB when using /to/ with time it only expands to 6 hour intervals eg 0000/to/2300 only returns 0000/0600/1200/1800, while in feature extraction we include all hour time steps as we know the digital twins are hourly so 0000/to/2300 will expand to include every hour from 0000 to 2300.

If you add /by/1 to the first request it will return the same data. We could harmonise these two but that would require some extra changes in other software if we want to make the expansion hourly.

[koldunovn]

I see... This is indeed totally confusing. Is this behaviour documented somewhere?
Are there any surprises when I do 'date': '20170101/to/20170301', - do I get only Sundays? :)