nsrdb-GOES-conus-v4-0-0 not working?

[williamhobbs]

PSM4 CONUS (GOES CONUS v4) appears to not be working. E.g.,

https://developer.nrel.gov/api/nsrdb/v2/solar/nsrdb-GOES-conus-v4-0-0-download.csv?wkt=POINT(-80.00%2040.00)&attributes=ghi&names=2018&utc=false&leap_day=true&email=YOUR_EMAIL&api_key=DEMO_KEY

(after changing email and key)

returns

{
  "inputs": {
    "body": {

    },
    "params": {

    },
    "query": {
      "wkt": "POINT(-80.00 40.00)",
      "attributes": "ghi",
      "names": "2018",
      "utc": "false",
      "leap_day": "true",
      "email": "myemail@email.com"
    }
  },
  "metadata": {
    "version": "2.0.0"
  },
  "status": 400,
  "errors": [
    "Data processing failure."
  ]
}

See pvlib/pvlib-python#2587.

[kandersolar]

cc @PjEdwards

[jessemilam]

I am experiencing the same issue. Even using the provided example from the documentation, the request fails with error "Data processing failure".

[PjEdwards]

OK I see the problem. The years 2018, 2019, and 2020 are giving an authorization error in the AWS side of the stack. I'm working with the team that manages our data in AWS to get a resolution.

FWIW 2021-2024 are working fine.

I'll report back when we get this resolved.

[williamhobbs]

Thanks, @PjEdwards. Also FWIW, AWS was returning unexpected rate limit errors last week for ECMWF’s S3 bucket. I wonder if the issues are in some way related.

blaylockbk/Herbie#487

[PjEdwards]

I will say it is really weird that this issue would just show up on a Thursday afternoon. AFAIK we don't ever manipulate the permissions on those S3 buckets except when we're setting up new datasets the first time.

[PjEdwards]

The evidence of that is in the fact that to get these fixed we have to contact the special NREL representative who works with AWS to manage these public data buckets! That's why the fix is taking a non-trivial amount of time even though setting an ACL on the bucket is the work of seconds.

[PjEdwards]

Tuesday morning update... the permissions issue was a red herring. The underlying cause is some corrupt data in S3. Unfortunately this means re-uploading the entire dataset which is a multi-day process. I would guess we should have these up my Monday, but that isn't a guarantee.

[PjEdwards]

Friday morning update... 2018 - 2021 are working now. 2022 - 2024 are still reloading.

[PjEdwards]

Final reloading is complete. I had to manually clear some caches and now it looks to me like everything is back to nominal. Please let me know if you continue to have access issues! Otherwise I'll close the ticket after a day or so.

[PjEdwards]

FYI we are seeing sporadic service outages this week due to some NREL internal networking issues. With the holiday weekend we may not get a final resolution until next week.