diff --git a/PW_OpenAPI.yaml b/PW_OpenAPI.yaml index daced11..6c15b31 100644 --- a/PW_OpenAPI.yaml +++ b/PW_OpenAPI.yaml @@ -1,6 +1,6 @@ openapi: 3.1.0 info: - version: "2.3.3" + version: "2.4.1" title: Pirate Weather API description: Pirate Weather provides an open, free, and documented source of government weather data. termsOfService: https://pirate-weather.apiable.io/terms @@ -775,7 +775,7 @@ components: version: type: string description: The version of Pirate Weather used to generate the forecast. - example: V2.3.3 + example: V2.4.1 sourceIDX: type: object description: The X, Y coordinate and the lat/long coordinate for each model used to generate the forecast. Only returned when version>2. diff --git a/README.md b/README.md index 135d65d..01bfad6 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,7 @@ # Pirate Weather Repo This is the primary repository for the Pirate Weather API, a free and open API designed to serve weather forecast data using the same syntax as Dark Sky. -To sign up, a free API key can be requested at [https://pirateweather.net/](https://pirateweather.net/en/latest/). API Documentation is at [https://pirateweather.net/en/latest/API/](https://pirateweather.net/en/latest/API/). A front end is available at [https://merrysky.net](https://merrysky.net). +To sign up, a free API key can be requested at [https://pirate-weather.apiable.io/](https://pirate-weather.apiable.io/). API Documentation is at [https://pirateweather.net/en/latest/API/](https://pirateweather.net/en/latest/API/), a front end is available at [https://merrysky.net](https://merrysky.net) and the status page is available at https://pirateweather.xitoring.io/ ## Why @@ -11,13 +11,10 @@ To sign up, a free API key can be requested at [https://pirateweather.net/](http ## What In this repository, I've included: - * The Docker image for processing: - * - * - * The processing scripts: - * - * API Documentation: - * +* API Documentation: + * +* The full API source code is available at + * https://github.com/Pirate-Weather/pirate-weather-code ## Support Keeping this free and running isn't free, so [donations to support this project](https://github.com/sponsors/alexander0042) are greatly appreciated! Plus, recurring monthly donations let me raise a API limit, allowing more frequent weather refreshes! diff --git a/docs/API.md b/docs/API.md index 7722f16..2785dbb 100644 --- a/docs/API.md +++ b/docs/API.md @@ -244,7 +244,7 @@ If `version=2` is included fields which were not part of the Dark Sky API will b }, "nearest-station": 0, "units": "ca", - "version": "V2.3.3" + "version": "V2.4.1" } } ``` diff --git a/docs/changelog.md b/docs/changelog.md index ab4ad86..6a8aab8 100644 --- a/docs/changelog.md +++ b/docs/changelog.md @@ -2,7 +2,19 @@ For a RSS feed of these changes, subscribe using this link: . -???+ note "Version 2.3" +???+ note "Version 2.4" + + * November 26, 2024, API Version 2.4.1 + * Fixed unit issues that occured when fixing the data point value issue as per [issue #360](https://github.com/Pirate-Weather/pirateweather/issues/360). + * November 25, 2024, API Version 2.4 + * First Official Open Source Release! Details in the new [Pirate Weather Code](https://github.com/Pirate-Weather/pirate-weather-code) repository, but starting today, you can see exactly how the data is processed, and even host your own instance of Pirate Weather! Contributions are welcome, so come check it out. Addresses the second oldest outstanding [issue #11](https://github.com/Pirate-Weather/pirateweather/issues/11). + * Fixed a datetime bug per [issue #330](https://github.com/Pirate-Weather/pirate-weather-ha/issues/330). + * Corrects the Apparent Temperature calculation per [issue #363](https://github.com/Pirate-Weather/pirateweather/issues/363). + * Changed the behaviour of new lines in NWS per [issue #367](https://github.com/Pirate-Weather/pirateweather/issues/367). + * Fixed issues where data points could return values outside of expected range as per [issue #360](https://github.com/Pirate-Weather/pirateweather/issues/360). + +??? note "Version 2.3" + * October 25, 2024, API Version 2.3.3 * Updated cache parameters per [issue #350](https://github.com/Pirate-Weather/pirateweather/issues/350). * Changed alert start time to use NWS effective time instead of onset time per [issue #353](https://github.com/Pirate-Weather/pirateweather/issues/353). diff --git a/docs/index.md b/docs/index.md index ca71b7f..6a29fb7 100644 --- a/docs/index.md +++ b/docs/index.md @@ -10,7 +10,8 @@ * To [**register for the API**](https://pirate-weather.apiable.io/) * [Get a weather forecast in the Dark Sky style](https://merrysky.net/) * [Home Assistant Integration](https://github.com/alexander0042/pirate-weather-hacs) -* [Processing code repo](https://github.com/alexander0042/pirateweather) +* [API repo](https://github.com/alexander0042/pirateweather) +* [Open Source code repo](https://github.com/Pirate-Weather/pirate-weather-code) * [Changelog](https://pirateweather.net/en/latest/changelog/) * [Status page](https://pirateweather.xitoring.io/) @@ -42,33 +43,21 @@ Alternatively, I also have a GitHub Sponsorship page set up on my [profile](http -## Recent Updates- Summer 2024 -Up to version 2.3! As always, details are available in the [changelog](https://pirateweather.net/en/latest/changelog/). - -* Major time machine (historic data) update! - * ERA-5 data now available from January 1940 to June 2024 via the excellent [NCAR archive](https://registry.opendata.aws/nsf-ncar-era5/)! - * Performance for these requests has been considerably improved (~10 s), since it is no longer querying against the Google data. - * Implemented using the excellent [Kerchunk library](https://fsspec.github.io/kerchunk) - * The June 2024 end date will be moved up as the ERA-5 data is updated. - * [Issue #130](https://github.com/Pirate-Weather/pirateweather/issues/130) - * [Issue #316](https://github.com/Pirate-Weather/pirateweather/issues/316) -* Historic model 1-hour forecast data is now available from June 2024 to present via the Pirate Weather Zarr archive. - * While technically forecast data, these forecasts are as close to observations as possible. - * Slower than ERA-5, since the full range of forecast models is used (~30 s). -* Historic data is now accessible from both the timemachine.pirateweather.net endpoint and the api.pirateweather.net endpoint. -* Documentation updates: - * [Issue #315](https://github.com/Pirate-Weather/pirateweather/issues/315) - * [Issue #320](https://github.com/Pirate-Weather/pirateweather/issues/320) -* Added the ability to provide the API key as a query parameter or header (as `apikey`) per [issue #314](https://github.com/Pirate-Weather/pirateweather/issues/314). -* Improved error handling for invalid locations per [issue #318](https://github.com/Pirate-Weather/pirateweather/issues/318) -* Fixed an unreported bug for max/min Apparent Temperature Times +## Recent Updates- Fall 2024 +Up to version 2.4! As always, details are available in the [changelog](https://pirateweather.net/en/latest/changelog/). + +* First Official Open Source Release! Details in the new [Pirate Weather Code](https://github.com/Pirate-Weather/pirate-weather-code) repository, but starting today, you can see exactly how the data is processed, and even host your own instance of Pirate Weather! Contributions are welcome, so come check it out. Addresses the second oldest outstanding [issue #11](https://github.com/Pirate-Weather/pirateweather/issues/11). +* Fixed a datetime bug per [issue #330](https://github.com/Pirate-Weather/pirate-weather-ha/issues/330). +* Corrects the Apparent Temperature calculation per [issue #363](https://github.com/Pirate-Weather/pirateweather/issues/363). +* Changed the behaviour of new lines in NWS per [issue #367](https://github.com/Pirate-Weather/pirateweather/issues/367). +* Fixed issues where data points could return values outside of expected range as per [issue #360](https://github.com/Pirate-Weather/pirateweather/issues/360). ## Background This project started from two points: as part of my [PhD](https://coastlines.engineering.queensu.ca/dunexrt), I had to become very familiar with working with NOAA forecast results (). Separately, an old tablet set up as a "Magic Mirror,” and was using a [weather module](https://github.com/jclarke0000/MMM-DarkSkyForecast) that relied on the Dark Sky API, as well as my [Home Assistant](https://www.home-assistant.io/) setup. So when I heard that it was [shutting down](https://blog.darksky.net/dark-sky-has-a-new-home/), I thought, "I wonder if I could do this.” Plus, I love learning new things (), and I had been looking for a project to learn Python on, so this seemed like the perfect opportunity! Spoiler alert, but it was way more difficult than I thought, but learned a lot throughout the process, and I think the end result turned out really well! ## Why? -This API is designed to be a drop in replacement/ alternative to the Dark Sky API, and as a tool for assessing GFS and HRRR forecasts via a JSON API. This solves two goals: +This API is designed to be a drop in replacement/ alternative to the Dark Sky API, and as a tool for assessing GFS, HRRR and NBM forecasts via a JSON API. This solves two goals: 1. It will also allow **legacy** applications to continue running after the Dark Sky shutdown, since as Home Assistant Integrations, Magic Mirror cards, and a whole host of other applications that have been developed over the years. 2. For anyone that is interested in knowing **exactly** how your weather forecasts are generated, this is the "show me the numbers" approach, since the data returned is directly from NOAA models, and every processing step I do is [documented](https://blog.pirateweather.net/). There are [lots](https://openweathermap.org/) [of](https://www.theweathernetwork.com) [existing](https://weather.com) [services](https://www.accuweather.com/) that provide custom forecasts using their own unique technologies, which can definitely improve accuracy, but I'm an engineer, so I wanted to be able to know what's going into the forecasts I'm using. If you're the sort of person who wants a [dense 34-page PowerPoint](http://rapidrefresh.noaa.gov/pdf/Alexander_AMS_NWP_2020.pdf) about why it rained when the forecast said it wouldn't, then this might be for you. diff --git a/docs/roadmap.md b/docs/roadmap.md index f5e26d4..d291390 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -1,12 +1,10 @@ # Roadmap -1. Open source API code and Docker containers to allow self-hosting. - * Longer blog post on V2 processing flow using Herbie -> wgrib2 -> xarray -> Dask -> Zarr arrays on LMDB for data. -2. Text summaries and translations +1. Text summaries and translations * Based on the [existing repository](https://github.com/alexander0042/translations). -3. Weather maps from new Zarr datafiles. -4. Add in alerts for Canada/ EU/ other. -5. Add in a Air Quality Index as suggested in [issue #92](https://github.com/Pirate-Weather/pirateweather/issues/92) -6. Add source data. +2. Weather maps from new Zarr datafiles. +3. Add in alerts for Canada/ EU/ other. +4. Add in a Air Quality Index as suggested in [issue #92](https://github.com/Pirate-Weather/pirateweather/issues/92) +5. Add source data. * Add in the Canadian models ([HRDPS](https://herbie.readthedocs.io/en/stable/gallery/eccc_models/hrdps.html), [GDPS](https://herbie.readthedocs.io/en/stable/gallery/eccc_models/gdps.html) and [RDPS](https://herbie.readthedocs.io/en/stable/gallery/eccc_models/rdps.html)) * [NBM-Alaska/ Hawaii/ Puerto Rico/ Guam](https://herbie.readthedocs.io/en/stable/gallery/noaa_models/nbm.html) * [RTMA/ URMA](https://herbie.readthedocs.io/en/stable/gallery/noaa_models/rtma-urma.html) @@ -17,7 +15,7 @@ * [DMI HARMONIE](https://opendatadocs.dmi.govcloud.dk/Data/Forecast_Data_Weather_Model_HARMONIE_DINI_IG)? * [RRFS](https://herbie.readthedocs.io/en/stable/gallery/noaa_models/rrfs.html) and [3D-RTMA/3D-URMA](https://vlab.noaa.gov/web/ufs-r2o/srw-cam) * Waiting for the full release to launch before integrating. -7. Add in a day/night forecast as suggested in [issue #49](https://github.com/Pirate-Weather/pirateweather/issues/49) -8. Add in water-related data as suggested in [issue #160](https://github.com/Pirate-Weather/pirateweather/issues/160) -9. Investigate using radar data/station data. +6. Add in a day/night forecast as suggested in [issue #49](https://github.com/Pirate-Weather/pirateweather/issues/49) +7. Add in water-related data as suggested in [issue #160](https://github.com/Pirate-Weather/pirateweather/issues/160) +8. Investigate using radar data/station data. * Investigate if it's feasible to use radar data and/or station data for the currently conditions. This is a suggestion in [issue #10](https://github.com/alexander0042/pirateweather/issues/10). diff --git a/scripts/gefs_combined-fargate.py b/scripts/gefs_combined-fargate.py deleted file mode 100644 index 1e78c28..0000000 --- a/scripts/gefs_combined-fargate.py +++ /dev/null @@ -1,269 +0,0 @@ -import json -import os -import shutil -import sys -import uuid -from datetime import datetime, timedelta -from urllib.parse import unquote_plus - -import boto3 -import numpy as np -import pywgrib2_s -from botocore import UNSIGNED -from botocore.client import Config -from netCDF4 import Dataset, MFDataset - -s3_client = boto3.client('s3', config=Config(signature_version=UNSIGNED)) -s3 = boto3.resource('s3') - - -# Set variables from ENV -bucket = os.environ['bucket'] -download_path = os.environ['download_path'] -time_in = os.environ['time'] -temp_path = os.environ['temp_path'] - -datetime_IN = datetime.strptime(time_in, "%Y-%m-%dT%H:%M:%S%z") -datetime_RUN = datetime_RUN = datetime_IN - timedelta(hours=7) - -fDate = datetime_RUN.strftime("%Y%m%d") -runTime = "t" + datetime_RUN.strftime("%H") + "z" - -print(time_in) -print(datetime_IN) -print(datetime_RUN) -print(fDate) -print(runTime) - -# Setup paths -prod_A = 'gefs2' # GEFS -prod_B = 'gefsa2' # GFS Averaged - -if not os.path.exists(download_path + '/' + prod_A): - os.makedirs(download_path + '/' + prod_A) -if not os.path.exists(download_path + '/' + prod_A + '/' + fDate): - os.makedirs(download_path + '/' + prod_A + '/' + fDate) -if not os.path.exists(download_path + '/' + prod_A + '/' + fDate + '/' + runTime): - os.makedirs(download_path + '/' + prod_A + '/' + fDate + '/' + runTime) - -if not os.path.exists(download_path + '/' + prod_B): - os.makedirs(download_path + '/' + prod_B) -if not os.path.exists(download_path + '/' + prod_B + '/' + fDate): - os.makedirs(download_path + '/' + prod_B + '/' + fDate) -if not os.path.exists(download_path + '/' + prod_B + '/' + fDate + '/' + runTime): - os.makedirs(download_path + '/' + prod_B + '/' + fDate + '/' + runTime) - - -# Merge paths -download_path_CK = download_path + '/' + prod_A + '/' + \ - fDate + '/' + runTime + '/' + 'out_' + prod_A + '_chunked.nc' -download_path_CK_AVG = download_path + '/' + prod_B + '/' + \ - fDate + '/' + runTime + '/' + 'out_' + prod_B + '_chunked.nc' - -download_path_NC_A = temp_path + '/' + prod_A + '_tmp.nc' -download_path_NC_B = temp_path + '/' + prod_B + '_tmp.nc' - -# Setup download file range -ncFileRange = range(3, 241, 3) -# ncFileRange = range(3,9,3) - -ncTimeCount = 0 - -for ncFileName in ncFileRange: - # Two loops for regular and averaged - grbTypes = ['gep', 'geavg'] - - for grbType in grbTypes: - if grbType == 'gep': - - # Loop through ensemble members - ensFileRange = range(1, 31) - download_path_GB_EN = temp_path + '/gep.f' + \ - str(ncFileName).zfill(3) + '.' + grbType + '.grb' + '.ens' - download_path_GB_ENP = temp_path + '/gep.f' + \ - str(ncFileName).zfill(3) + '.' + grbType + '.grb' + '.enp' - download_path_GB_ENP2 = temp_path + '/gep.f' + \ - str(ncFileName).zfill(3) + '.' + grbType + '.grb' + '.enp2' - - for ensFileName in ensFileRange: - # Define temp path - download_file_pathA = temp_path + '/gep' + \ - str(ensFileName).zfill(2) + '.f' + \ - str(ncFileName).zfill(3) + '.' + grbType + '.grb' - - # Download from S3 - s3_filename = 'gefs.' + fDate + '/' + runTime[1:3] + '/atmos/pgrb2sp25/gep' + str( - ensFileName).zfill(2) + '.' + runTime + '.pgrb2s.0p25.f' + str(ncFileName).zfill(3) - s3_client.download_file( - bucket, s3_filename, download_file_pathA) - - # Append precpn to ensemble file and remove member - pywgrib2_s.wgrib2( - [download_file_pathA, '-match', 'APCP', '-append', "-grib_out", download_path_GB_EN]) - pywgrib2_s.close(download_file_pathA) - os.remove(download_file_pathA) - - # Ens processing at each time step - # Ensemble processing and extract spread - pywgrib2_s.wgrib2( - [download_path_GB_EN, '-ens_processing', download_path_GB_ENP, '0']) - pywgrib2_s.close(download_path_GB_ENP) - matchString = ":(ens spread):" - pywgrib2_s.wgrib2([download_path_GB_ENP, '-match', matchString, - '-set_ext_name', '1', '-append', '-netcdf', download_path_NC_A]) - - # Process ensemble using rpn and extract mean - pywgrib2_s.wgrib2([download_path_GB_EN, '-rewind_init', download_path_GB_EN, - '-rpn', '1:>=', '-ens_processing', download_path_GB_ENP2, '0']) - pywgrib2_s.close(download_path_GB_ENP2) - matchString = ":(ens mean):" - pywgrib2_s.wgrib2([download_path_GB_ENP2, '-match', matchString, '-set_prob', '1', - '1', '1', '1', '1', '-set_ext_name', '1', '-append', '-netcdf', download_path_NC_A]) - - pywgrib2_s.close(download_path_GB_EN) - pywgrib2_s.close(download_path_GB_ENP) - pywgrib2_s.close(download_path_GB_ENP2) - - os.remove(download_path_GB_EN) - os.remove(download_path_GB_ENP) - os.remove(download_path_GB_ENP2) - - if grbType == 'geavg': - # Averaged - # Define temp paths - download_file_pathB = temp_path + '/gefs.f' + \ - str(ncFileName).zfill(3) + '.' + grbType + '.grb' - download_path_GB_B = download_file_pathB + '.earth' - # Download from S3 - s3_filename = 'gefs.' + fDate + '/' + \ - runTime[1:3] + '/atmos/pgrb2sp25/geavg.' + runTime + \ - '.pgrb2s.0p25.f' + str(ncFileName).zfill(3) - # print(bucket) - # print(s3_filename) - s3_client.download_file(bucket, s3_filename, download_file_pathB) - - matchString = (":(CRAIN|" - "CSNOW|CFRZR|" - "PRATE|CICEP):") - - pywgrib2_s.wgrib2([download_file_pathB, '-match', matchString, - '-append', "-grib_out", download_path_GB_B]) - a = pywgrib2_s.wgrib2([download_file_pathB, '-rewind_init', download_file_pathB, - '-match', 'APCP', '-append', '-grib', download_path_GB_B, '-quit']) - - ### Merge and Remove ### - pywgrib2_s.wgrib2([download_path_GB_B, '-append', - '-netcdf', download_path_NC_B]) - - pywgrib2_s.close(download_path_GB_B) - os.remove(download_path_GB_B) - os.remove(download_file_pathB) - - -# Remove Chunk file if exists -if os.path.isfile(download_path_CK): - os.remove(download_path_CK) -if os.path.isfile(download_path + '/' + prod_A + '/' + fDate + '/' + runTime + '/' + prod_A + '.done'): - os.remove(download_path + '/' + prod_A + '/' + fDate + - '/' + runTime + '/' + prod_A + '.done') - -if os.path.isfile(download_path_CK_AVG): - os.remove(download_path_CK_AVG) -if os.path.isfile(download_path + '/' + prod_B + '/' + fDate + '/' + runTime + '/' + prod_B + '.done'): - os.remove(download_path + '/' + prod_B + '/' + fDate + - '/' + runTime + '/' + prod_B + '.done') - -print('###Chunk') -chkA = Dataset(download_path_CK, "w") -chkB = Dataset(download_path_CK_AVG, "w") - -srcA = Dataset(download_path_NC_A, 'r', format="NETCDF3_CLASSIC") -srcB = Dataset(download_path_NC_B, 'r', format="NETCDF3_CLASSIC") - -# copy global attributes all at once via dictionary -for grbType in grbTypes: - if grbType == 'gep': - src = srcA - chk = chkA - chk.setncatts(srcA.__dict__) - prod = prod_A - elif grbType == 'geavg': - src = srcB - chk = chkB - chk.setncatts(srcB.__dict__) - prod = prod_B - - # Save Lat Lon - lats = src.variables['latitude'][:] - lons = src.variables['longitude'][:] - - np.save(download_path + '/' + prod + '/' + prod + '-lats.npy', lats.data) - np.save(download_path + '/' + prod + '/' + prod + '-lons.npy', lons.data) - - # copy dimensions for srcA and srcB - for name, dimension in src.dimensions.items(): - chk.createDimension( - name, (len(dimension) if not dimension.isunlimited() else None)) - # copy all file data except for the excluded - for name, variable in src.variables.items(): - print('##DIM##') - print(len(variable.dimensions)) - print(variable.shape) - if len(variable.dimensions) == 3: - if 'APCP_surface' in name: - x = chk.createVariable(name, variable.datatype, variable.dimensions, chunksizes=[ - 80, 7, 7], zlib=True, least_significant_digit=4, complevel=1) - elif len(variable.dimensions) == 3: - x = chk.createVariable(name, variable.datatype, variable.dimensions, chunksizes=[ - 80, 7, 7], zlib=True, least_significant_digit=2, complevel=1) - else: - x = chk.createVariable( - name, variable.datatype, variable.dimensions) - else: - x = chk.createVariable( - name, variable.datatype, variable.dimensions) - - chk[name][:] = src[name][:] - # copy variable attributes all at once via dictionary - for ncattr in src[name].ncattrs(): - if ncattr != '_FillValue': - chk[name].setncattr(ncattr, src[name].getncattr(ncattr)) - - src.close() - chk.close() - -# Remove runs greater than a week old -for runPath in range(0, 2): - if runPath == 0: - runDatesPath = download_path + '/' + prod_A - elif runPath == 1: - runDatesPath = download_path + '/' + prod_B - - runDates = [f for f in os.listdir(runDatesPath) if not os.path.isfile( - os.path.join(runDatesPath, f))] - runDatesInt = [int(i) for i in runDates] - - for g in runDatesInt: - # Cycle through dates - dataRuns_g = [f for f in os.listdir(runDatesPath + '/' + str( - g)) if not os.path.isfile(os.path.join(runDatesPath + '/' + str(g), f))] - dataRunsInt_g = [int(z[1:3]) for z in dataRuns_g] - - gfsRunDate = datetime(int(str(g)[0:4]), - int(str(g)[4:6]), - int(str(g)[6:8]), 00, 00, 00) - - if gfsRunDate < (datetime.now() - timedelta(days=7)): - shutil.rmtree(runDatesPath + '/' + str(g)) - print(runDatesPath + '/' + str(g)) - - -# Save completion files -f = open(download_path + '/' + prod_A + '/' + fDate + - '/' + runTime + '/' + prod_A + '.done', "w") -f.write(download_path_CK) -f.close() -f = open(download_path + '/' + prod_B + '/' + fDate + - '/' + runTime + '/' + prod_B + '.done', "w") -f.write(download_path_CK_AVG) -f.close() diff --git a/scripts/gfs_combined-fargate.py b/scripts/gfs_combined-fargate.py deleted file mode 100644 index a9ad544..0000000 --- a/scripts/gfs_combined-fargate.py +++ /dev/null @@ -1,210 +0,0 @@ -import json -import os -import shutil -import sys -import uuid -from datetime import datetime, timedelta -from urllib.parse import unquote_plus - -import boto3 -import numpy as np -import pywgrib2_s -from botocore import UNSIGNED -from botocore.client import Config -from netCDF4 import Dataset, MFDataset - -s3_client = boto3.client('s3', config=Config(signature_version=UNSIGNED)) -s3 = boto3.resource('s3') - - -# Set variables from ENV -bucket = os.environ['bucket'] -download_path = os.environ['download_path'] -time_in = os.environ['time'] -temp_path = os.environ['temp_path'] - -datetime_IN = datetime.strptime(time_in, "%Y-%m-%dT%H:%M:%S%z") -datetime_RUN = datetime_RUN = datetime_IN - timedelta(hours=5) - -fDate = datetime_RUN.strftime("%Y%m%d") -runTime = "t" + datetime_RUN.strftime("%H") + "z" - -print(time_in) -print(datetime_IN) -print(datetime_RUN) -print(fDate) -print(runTime) - -# Setup paths -prod_A = 'gfs2' # GFS - -if not os.path.exists(download_path + '/' + prod_A): - os.makedirs(download_path + '/' + prod_A) -if not os.path.exists(download_path + '/' + prod_A + '/' + fDate): - os.makedirs(download_path + '/' + prod_A + '/' + fDate) -if not os.path.exists(download_path + '/' + prod_A + '/' + fDate + '/' + runTime): - os.makedirs(download_path + '/' + prod_A + '/' + fDate + '/' + runTime) - -# Merge paths -download_path_CK = download_path + '/' + prod_A + '/' + \ - fDate + '/' + runTime + '/' + 'out_' + prod_A + '_chunked.nc' -download_path_NC = temp_path + '/' + prod_A + '_tmp.nc' - - -# Setup download file range -gfs_range1 = range(1, 121) -gfs_range2 = range(123, 241, 3) -ncFileRange = [*gfs_range1, *gfs_range2] -# ncFileRange = range(3,241,3) -# ncFileRange = range(1,3,1) - -ncTimeCount = 0 - -for ncFileName in ncFileRange: - # Two loops for primary and b files - grbTypes = ['pgrb2', 'pgrb2b'] - - for grbType in grbTypes: - # Define download destination path - # download_file_path = download_path + '/gfs2/' + fDate + '/' + runTime + '/download/' + 'gfs.f' + str(ncFileName).zfill(3) + '.' + grbType + '.grb' - download_file_path = temp_path + '/gfs.f' + \ - str(ncFileName).zfill(3) + '.' + grbType + '.grb' - - # Download from S3 - s3_filename = 'gfs.' + fDate + '/' + \ - runTime[1:3] + '/atmos/gfs.' + runTime + '.' + \ - grbType + '.0p25.f' + str(ncFileName).zfill(3) - s3_client.download_file(bucket, s3_filename, download_file_path) - - if grbType == 'pgrb2': - # Define temp path - download_file_pathA = temp_path + '/gfs.f' + \ - str(ncFileName).zfill(3) + '.' + grbType + '.grb' - - # download_path_GB_A = download_pathA + '/gfs2/' + fDate + '/' + runTime + '/gribs/' + 'gfs.f' + str(ncFileName).zfill(3) + '.' + grbType + '.earth.grb' - download_path_GB_A = download_file_pathA + '.earth' - - matchString = (":(DPT:2 m above ground|TMP:2 m above ground|CRAIN:surface:.*hour fcst|" - "CSNOW:surface:.*hour fcst|CFRZR:surface:.*hour fcst|" - "PRATE:surface:.*hour fcst|PRES:surface|TOZNE|" - "UGRD:10 m above ground:.*hour fcst|VGRD:10 m above ground:.*hour fcst|" - "VIS:surface|GUST:surface:.*hour fcst|" - "RH:2 m above ground:.*hour fcst|CICEP:surface:.*hour fcst|TOZNE):") - - pywgrib2_s.wgrib2([download_file_pathA, '-match', matchString, - '-append', "-grib_out", download_path_GB_A]) - pywgrib2_s.wgrib2([download_file_pathA, '-rewind_init', download_file_pathA, - '-match', 'APCP', '-append', '-grib', download_path_GB_A, '-quit']) - pywgrib2_s.wgrib2([download_file_pathA, '-rewind_init', download_file_pathA, '-match', - 'TCDC:entire atmosphere', '-append', '-grib', download_path_GB_A, '-quit']) - pywgrib2_s.close(download_file_pathA) - - if grbType == 'pgrb2b': - # Define temp path - download_file_pathB = temp_path + '/gfs.f' + \ - str(ncFileName).zfill(3) + '.' + grbType + '.grb' - - # download_path_GB_B = download_path + '/gfs2/' + fDate + '/' + runTime + '/gribs/' + 'gfs.f' + str(ncFileName).zfill(3) + '.' + grbType + '.earth.grb' - download_path_GB_B = download_file_pathB + '.earth' - - pywgrib2_s.wgrib2([download_file_pathB, '-match', '(:DUVB:surface:)', - '-append', '-grib_out', download_path_GB_B]) - pywgrib2_s.close(download_file_pathB) - - ### Merge ### - # print('##File 1##') - pywgrib2_s.wgrib2([download_path_GB_A, '-append', - '-netcdf', download_path_NC]) - - # print('##File 2##') - pywgrib2_s.wgrib2([download_path_GB_B, '-append', - '-netcdf', download_path_NC]) - - pywgrib2_s.close(download_path_GB_A) - os.remove(download_path_GB_A) - pywgrib2_s.close(download_path_GB_B) - os.remove(download_path_GB_B) - os.remove(download_file_pathA) - os.remove(download_file_pathB) - - -# Remove Chunk file if exists -if os.path.isfile(download_path_CK): - os.remove(download_path_CK) - -if os.path.isfile(download_path + '/' + prod_A + '/' + fDate + '/' + runTime + '/' + prod_A + '.done'): - os.remove(download_path + '/' + prod_A + '/' + fDate + - '/' + runTime + '/' + prod_A + '.done') - - -print('###Chunk') -chk = Dataset(download_path_CK, "w") -src = Dataset(download_path_NC, 'r', format="NETCDF3_CLASSIC") - -# Save Lat Lon -lats = src.variables['latitude'][:] -lons = src.variables['longitude'][:] - -np.save(download_path + '/' + prod_A + '/' + prod_A + '-lats.npy', lats.data) -np.save(download_path + '/' + prod_A + '/' + prod_A + '-lons.npy', lons.data) - - -# copy global attributes all at once via dictionary -chk.setncatts(src.__dict__) - -# copy dimensions -for name, dimension in src.dimensions.items(): - chk.createDimension( - name, (len(dimension) if not dimension.isunlimited() else None)) -# copy all file data except for the excluded -for name, variable in src.variables.items(): - print('##DIM##') - print(len(variable.dimensions)) - print(variable.shape) - if len(variable.dimensions) == 3: - if 'PRATE_surface' in name: - x = chk.createVariable(name, variable.datatype, variable.dimensions, chunksizes=[ - 160, 2, 4], zlib=True, least_significant_digit=4, complevel=1) - elif 'APCP_surface' in name: - x = chk.createVariable(name, variable.datatype, variable.dimensions, chunksizes=[ - 160, 2, 4], zlib=True, least_significant_digit=4, complevel=1) - else: - x = chk.createVariable(name, variable.datatype, variable.dimensions, chunksizes=[ - 160, 2, 4], zlib=True, least_significant_digit=1, complevel=1) - else: - x = chk.createVariable(name, variable.datatype, variable.dimensions) - chk[name][:] = src[name][:] - # copy variable attributes all at once via dictionary - for ncattr in src[name].ncattrs(): - if ncattr != '_FillValue': - chk[name].setncattr(ncattr, src[name].getncattr(ncattr)) - -src.close() -chk.close() - - -# Remove runs greater than a week old -runDatesPath = download_path + '/' + prod_A -runDates = [f for f in os.listdir(runDatesPath) if not os.path.isfile( - os.path.join(runDatesPath, f))] -runDatesInt = [int(i) for i in runDates] - -for g in runDatesInt: - # Cycle through dates - dataRuns_g = [f for f in os.listdir(runDatesPath + '/' + str( - g)) if not os.path.isfile(os.path.join(runDatesPath + '/' + str(g), f))] - dataRunsInt_g = [int(z[1:3]) for z in dataRuns_g] - - gfsRunDate = datetime(int(str(g)[0:4]), - int(str(g)[4:6]), - int(str(g)[6:8]), 00, 00, 00) - - if gfsRunDate < (datetime.now() - timedelta(days=7)): - shutil.rmtree(runDatesPath + '/' + str(g)) - print(runDatesPath + '/' + str(g)) - -# Save completion files -f = open(download_path + '/' + prod_A + '/' + fDate + - '/' + runTime + '/' + prod_A + '.done', "w") -f.write(download_path_CK) -f.close() diff --git a/scripts/hrrr6h_combined-fargate.py b/scripts/hrrr6h_combined-fargate.py deleted file mode 100644 index 123a329..0000000 --- a/scripts/hrrr6h_combined-fargate.py +++ /dev/null @@ -1,184 +0,0 @@ -# Ingest script for docker image for hourly HRRR products - -import json -import os -import shutil -import sys -import uuid -from datetime import datetime, timedelta -from urllib.parse import unquote_plus - -import boto3 -import numpy as np -import pywgrib2_s -from botocore import UNSIGNED -from botocore.client import Config -from netCDF4 import Dataset, MFDataset - -s3_client = boto3.client('s3', config=Config(signature_version=UNSIGNED)) -s3 = boto3.resource('s3') - - -# Set variables from ENV -bucket = os.environ['bucket'] -download_path = os.environ['download_path'] -time_in = os.environ['time'] -temp_path = os.environ['temp_path'] - -# 2 hour 30 minute delay -datetime_IN = datetime.strptime(time_in, "%Y-%m-%dT%H:%M:%S%z") -datetime_RUN = datetime_RUN = datetime_IN - timedelta(hours=2, minutes=30) - -fDate = datetime_RUN.strftime("%Y%m%d") -runTime = "t" + datetime_RUN.strftime("%H") + "z" - -print(time_in) -print(datetime_IN) -print(datetime_RUN) -print(fDate) -print(runTime) - -# Setup paths -prod_A = 'hrrr6h2' # Hourly HRRR + 18 -if not os.path.exists(download_path + '/' + prod_A): - os.makedirs(download_path + '/' + prod_A) -if not os.path.exists(download_path + '/' + prod_A + '/' + fDate): - os.makedirs(download_path + '/' + prod_A + '/' + fDate) -if not os.path.exists(download_path + '/' + prod_A + '/' + fDate + '/' + runTime): - os.makedirs(download_path + '/' + prod_A + '/' + fDate + '/' + runTime) - -# Merge paths -download_path_CK = download_path + '/' + prod_A + '/' + \ - fDate + '/' + runTime + '/' + 'out_' + prod_A + '_chunked.nc' - -download_path_NC_A = temp_path + '/' + prod_A + '_tmp.nc' - -# Setup download file range -ncFileRange_A = rncFileRange = range(19, 49) - -# Setup grid transformation -HRRR_grid1 = 'lambert:262.500000:38.500000:38.500000:38.500000' -HRRR_grid2 = '237.280472:1799:3000.000000' -HRRR_grid3 = '21.138123:1059:3000.000000' - - -# hrrr6h2 -# HRRRH -grbType = 'wrfsfc' -for ncFileName in ncFileRange_A: - download_file_pathA = temp_path + '/hrrrh.f' + \ - str(ncFileName).zfill(3) + '.' + grbType + '.grb' - download_path_GB_A = temp_path + '/hrrrh.f' + \ - str(ncFileName).zfill(3) + '.' + grbType + '.grb.earth' - - # Download from S3 - s3_filename = 'hrrr.' + fDate + '/conus/hrrr.' + runTime + \ - '.wrfsfcf' + str(ncFileName).zfill(2) + '.grib2' - print(s3_filename) - - s3_client.download_file(bucket, s3_filename, download_file_pathA) - - matchString = (":(TMP:2 m above ground|CRAIN:surface|CSNOW:surface|" - "CFRZR:surface|PRATE:surface|PRES:surface|CICEP:surface|" - "UGRD:10 m above ground:.*hour fcst|" - "VGRD:10 m above ground:.*hour fcst|" - "VIS:surface|DPT:2 m above ground|TCDC:entire atmosphere|GUST:surface|RH:2 m above ground):") - - pywgrib2_s.wgrib2([download_file_pathA, '-new_grid_winds', 'earth', '-new_grid_interpolation', 'neighbor', - '-match', matchString, '-new_grid', HRRR_grid1, HRRR_grid2, HRRR_grid3, download_path_GB_A]) - pywgrib2_s.wgrib2([download_file_pathA, '-rewind_init', download_file_pathA, '-new_grid_winds', 'earth', '-new_grid_interpolation', - 'neighbor', '-match', 'APCP', '-append', '-new_grid', HRRR_grid1, HRRR_grid2, HRRR_grid3, download_path_GB_A, '-quit']) - pywgrib2_s.close(download_path_GB_A) - - # Add to NetCDF - pywgrib2_s.wgrib2([download_path_GB_A, '-append', - '-netcdf', download_path_NC_A]) - - pywgrib2_s.close(download_file_pathA) - pywgrib2_s.close(download_path_GB_A) - - os.remove(download_file_pathA) - os.remove(download_path_GB_A) - - -# Remove Chunk file if exists -if os.path.isfile(download_path_CK): - os.remove(download_path_CK) -if os.path.isfile(download_path + '/' + prod_A + '/' + fDate + '/' + runTime + '/' + prod_A + '.done'): - os.remove(download_path + '/' + prod_A + '/' + fDate + - '/' + runTime + '/' + prod_A + '.done') - - -print('###Chunk') -chkA = Dataset(download_path_CK, "w") -srcA = Dataset(download_path_NC_A, 'r', format="NETCDF3_CLASSIC") - -# Save Lat Lon -lats = srcA.variables['latitude'][:] -lons = srcA.variables['longitude'][:] - -np.save(download_path + '/' + prod_A + '/' + prod_A + '-lats.npy', lats.data) -np.save(download_path + '/' + prod_A + '/' + prod_A + '-lons.npy', lons.data) - -# copy global attributes all at once via dictionary -src = srcA -chk = chkA -chk.setncatts(srcA.__dict__) - -# copy dimensions for srcA and srcB -for name, dimension in src.dimensions.items(): - chk.createDimension( - name, (len(dimension) if not dimension.isunlimited() else None)) -# copy all file data except for the excluded -for name, variable in src.variables.items(): - print('##DIM##') - print(len(variable.dimensions)) - print(variable.shape) - if len(variable.dimensions) == 3: - if 'PRATE_surface' in name: - x = chk.createVariable(name, variable.datatype, variable.dimensions, chunksizes=[ - 30, 10, 10], zlib=True, least_significant_digit=4, complevel=1) - else: - x = chk.createVariable(name, variable.datatype, variable.dimensions, chunksizes=[ - 30, 10, 10], zlib=True, least_significant_digit=1, complevel=1) - else: - x = chk.createVariable(name, variable.datatype, variable.dimensions) - - chk[name][:] = src[name][:] - # copy variable attributes all at once via dictionary - for ncattr in src[name].ncattrs(): - if ncattr != '_FillValue': - chk[name].setncattr(ncattr, src[name].getncattr(ncattr)) - -src.close() -chk.close() - -# Remove runs greater than a week old -for runPath in range(0, 1): - if runPath == 0: - runDatesPath = download_path + '/' + prod_A - - runDates = [f for f in os.listdir(runDatesPath) if not os.path.isfile( - os.path.join(runDatesPath, f))] - runDatesInt = [int(i) for i in runDates] - - for g in runDatesInt: - # Cycle through dates - dataRuns_g = [f for f in os.listdir(runDatesPath + '/' + str( - g)) if not os.path.isfile(os.path.join(runDatesPath + '/' + str(g), f))] - dataRunsInt_g = [int(z[1:3]) for z in dataRuns_g] - - gfsRunDate = datetime(int(str(g)[0:4]), - int(str(g)[4:6]), - int(str(g)[6:8]), 00, 00, 00) - - if gfsRunDate < (datetime.now() - timedelta(days=7)): - shutil.rmtree(runDatesPath + '/' + str(g)) - print(runDatesPath + '/' + str(g)) - - -# Save completion files -f = open(download_path + '/' + prod_A + '/' + fDate + - '/' + runTime + '/' + prod_A + '.done', "w") -f.write(download_path_CK) -f.close() diff --git a/scripts/hrrrh_combined-fargate.py b/scripts/hrrrh_combined-fargate.py deleted file mode 100644 index 41034f6..0000000 --- a/scripts/hrrrh_combined-fargate.py +++ /dev/null @@ -1,258 +0,0 @@ -# Ingest script for docker image for hourly HRRR products - -import json -import os -import shutil -import sys -import uuid -from datetime import datetime, timedelta -from urllib.parse import unquote_plus - -import boto3 -import numpy as np -import pywgrib2_s -from botocore import UNSIGNED -from botocore.client import Config -from netCDF4 import Dataset, MFDataset - -s3_client = boto3.client('s3', config=Config(signature_version=UNSIGNED)) -s3 = boto3.resource('s3') - - -# Set variables from ENV -bucket = os.environ['bucket'] -download_path = os.environ['download_path'] -time_in = os.environ['time'] -temp_path = os.environ['temp_path'] - -# 1 hour 45 minute delay -datetime_IN = datetime.strptime(time_in, "%Y-%m-%dT%H:%M:%S%z") -datetime_RUN = datetime_RUN = datetime_IN - timedelta(hours=1, minutes=45) - -fDate = datetime_RUN.strftime("%Y%m%d") -runTime = "t" + datetime_RUN.strftime("%H") + "z" - -print(time_in) -print(datetime_IN) -print(datetime_RUN) -print(fDate) -print(runTime) - -# Setup paths - -prod_A = 'subh2' # SubHourly HRRR -prod_B = 'hrrrh2' # Hourly HRRR -if not os.path.exists(download_path + '/' + prod_A): - os.makedirs(download_path + '/' + prod_A) -if not os.path.exists(download_path + '/' + prod_A + '/' + fDate): - os.makedirs(download_path + '/' + prod_A + '/' + fDate) -if not os.path.exists(download_path + '/' + prod_A + '/' + fDate + '/' + runTime): - os.makedirs(download_path + '/' + prod_A + '/' + fDate + '/' + runTime) - -if not os.path.exists(download_path + '/' + prod_B): - os.makedirs(download_path + '/' + prod_B) -if not os.path.exists(download_path + '/' + prod_B + '/' + fDate): - os.makedirs(download_path + '/' + prod_B + '/' + fDate) -if not os.path.exists(download_path + '/' + prod_B + '/' + fDate + '/' + runTime): - os.makedirs(download_path + '/' + prod_B + '/' + fDate + '/' + runTime) - -# Merge paths -download_path_CK = download_path + '/' + prod_A + '/' + \ - fDate + '/' + runTime + '/' + 'out_' + prod_A + '_chunked.nc' -download_path_CK_B = download_path + '/' + prod_B + '/' + \ - fDate + '/' + runTime + '/' + 'out_' + prod_B + '_chunked.nc' - -download_path_NC_A = temp_path + '/' + prod_A + '_tmp.nc' -download_path_NC_B = temp_path + '/' + prod_B + '_tmp.nc' - -# Setup download file range -ncFileRange_A = range(1, 5) -ncFileRange_B = range(1, 19) - -# Setup grid transformation -HRRR_grid1 = 'lambert:262.500000:38.500000:38.500000:38.500000' -HRRR_grid2 = '237.280472:1799:3000.000000' -HRRR_grid3 = '21.138123:1059:3000.000000' - - -# SUBH -grbType = 'wrfsubhf' -for ncFileName in ncFileRange_A: - download_file_pathA = temp_path + '/subh.f' + \ - str(ncFileName).zfill(3) + '.' + grbType + '.grb' - download_path_GB_A = temp_path + '/subh.f' + \ - str(ncFileName).zfill(3) + '.' + grbType + '.grb.earth' - # Download from S3 - s3_filename = 'hrrr.' + fDate + '/conus/hrrr.' + runTime + \ - '.wrfsubhf' + str(ncFileName).zfill(2) + '.grib2' - print(s3_filename) - - s3_client.download_file(bucket, s3_filename, download_file_pathA) - - matchString = (":(TMP:2 m above ground|CRAIN:surface|CSNOW:surface|" - "CFRZR:surface|PRATE:surface|PRES:surface|CICEP:surface|" - "UGRD:10 m above ground:.*min fcst|" - "VGRD:10 m above ground:.*min fcst|" - "VIS:surface|DPT:2 m above ground|APCP:surface|" - "TCDC:entire atmosphere|GUST:surface):") - - # ((15|30|45|60|75|90|105|120|135|150|165|180|" - # "195|210|225|240|300|360|420|480|540|600|660|720|780|840|900|960|1020|1080) min fcst|.*acc fcst):") - - # Convert to earth oriented winds and select variables - pywgrib2_s.wgrib2([download_file_pathA, '-new_grid_winds', 'earth', '-new_grid_interpolation', 'neighbor', '-match', matchString, - '-new_grid', HRRR_grid1, HRRR_grid2, HRRR_grid3, download_path_GB_A]) - pywgrib2_s.close(download_path_GB_A) - - # Add to NetCDF - pywgrib2_s.wgrib2([download_path_GB_A, '-append', - '-netcdf', download_path_NC_A]) - - pywgrib2_s.close(download_file_pathA) - pywgrib2_s.close(download_path_GB_A) - - os.remove(download_file_pathA) - os.remove(download_path_GB_A) - -# HRRRH -grbType = 'wrfsfc' -for ncFileName in ncFileRange_B: - download_file_pathB = temp_path + '/hrrrh.f' + \ - str(ncFileName).zfill(3) + '.' + grbType + '.grb' - download_path_GB_B = temp_path + '/hrrrh.f' + \ - str(ncFileName).zfill(3) + '.' + grbType + '.grb.earth' - - # Download from S3 - s3_filename = 'hrrr.' + fDate + '/conus/hrrr.' + runTime + \ - '.wrfsfcf' + str(ncFileName).zfill(2) + '.grib2' - print(s3_filename) - - s3_client.download_file(bucket, s3_filename, download_file_pathB) - - matchString = (":(TMP:2 m above ground|CRAIN:surface|CSNOW:surface|" - "CFRZR:surface|PRATE:surface|PRES:surface|CICEP:surface|" - "UGRD:10 m above ground:.*hour fcst|" - "VGRD:10 m above ground:.*hour fcst|" - "VIS:surface|DPT:2 m above ground|TCDC:entire atmosphere|GUST:surface|RH:2 m above ground):") - - pywgrib2_s.wgrib2([download_file_pathB, '-new_grid_winds', 'earth', '-new_grid_interpolation', 'neighbor', - '-match', matchString, '-new_grid', HRRR_grid1, HRRR_grid2, HRRR_grid3, download_path_GB_B]) - pywgrib2_s.wgrib2([download_file_pathB, '-rewind_init', download_file_pathB, '-new_grid_winds', 'earth', '-new_grid_interpolation', - 'neighbor', '-match', 'APCP', '-append', '-new_grid', HRRR_grid1, HRRR_grid2, HRRR_grid3, download_path_GB_B, '-quit']) - pywgrib2_s.close(download_path_GB_B) - # Add to NetCDF - pywgrib2_s.wgrib2([download_path_GB_B, '-append', - '-netcdf', download_path_NC_B]) - - pywgrib2_s.close(download_file_pathB) - pywgrib2_s.close(download_path_GB_B) - - os.remove(download_file_pathB) - os.remove(download_path_GB_B) - - -# Remove Chunk file if exists -if os.path.isfile(download_path_CK): - os.remove(download_path_CK) -if os.path.isfile(download_path + '/' + prod_A + '/' + fDate + '/' + runTime + '/' + prod_A + '.done'): - os.remove(download_path + '/' + prod_A + '/' + fDate + - '/' + runTime + '/' + prod_A + '.done') - -if os.path.isfile(download_path_CK_B): - os.remove(download_path_CK_B) -if os.path.isfile(download_path + '/' + prod_B + '/' + fDate + '/' + runTime + '/' + prod_B + '.done'): - os.remove(download_path + '/' + prod_B + '/' + fDate + - '/' + runTime + '/' + prod_B + '.done') - -print('###Chunk') -chkA = Dataset(download_path_CK, "w") -chkB = Dataset(download_path_CK_B, "w") - -srcA = Dataset(download_path_NC_A, 'r', format="NETCDF3_CLASSIC") -srcB = Dataset(download_path_NC_B, 'r', format="NETCDF3_CLASSIC") - -# copy global attributes all at once via dictionary -grbTypes = ['wrfsubhf', 'wrfsfc'] -for grbType in grbTypes: - if grbType == 'wrfsubhf': - src = srcA - chk = chkA - chk.setncatts(srcA.__dict__) - prod = prod_A - elif grbType == 'wrfsfc': - src = srcB - chk = chkB - chk.setncatts(srcB.__dict__) - prod = prod_B - - # Save Lat Lon - lats = src.variables['latitude'][:] - lons = src.variables['longitude'][:] - - np.save(download_path + '/' + prod + '/' + prod + '-lats.npy', lats.data) - np.save(download_path + '/' + prod + '/' + prod + '-lons.npy', lons.data) - - # copy dimensions for srcA and srcB - for name, dimension in src.dimensions.items(): - chk.createDimension( - name, (len(dimension) if not dimension.isunlimited() else None)) - # copy all file data except for the excluded - for name, variable in src.variables.items(): - print('##DIM##') - print(len(variable.dimensions)) - print(variable.shape) - if len(variable.dimensions) == 3: - if 'PRATE_surface' in name: - x = chk.createVariable(name, variable.datatype, variable.dimensions, chunksizes=[ - 18, 10, 10], zlib=True, least_significant_digit=4, complevel=1) - else: - x = chk.createVariable(name, variable.datatype, variable.dimensions, chunksizes=[ - 18, 10, 10], zlib=True, least_significant_digit=1, complevel=1) - else: - x = chk.createVariable( - name, variable.datatype, variable.dimensions) - - chk[name][:] = src[name][:] - # copy variable attributes all at once via dictionary - for ncattr in src[name].ncattrs(): - if ncattr != '_FillValue': - chk[name].setncattr(ncattr, src[name].getncattr(ncattr)) - - src.close() - chk.close() - -# Remove runs greater than a week old -for runPath in range(0, 2): - if runPath == 0: - runDatesPath = download_path + '/' + prod_A - elif runPath == 1: - runDatesPath = download_path + '/' + prod_B - - runDates = [f for f in os.listdir(runDatesPath) if not os.path.isfile( - os.path.join(runDatesPath, f))] - runDatesInt = [int(i) for i in runDates] - - for g in runDatesInt: - # Cycle through dates - dataRuns_g = [f for f in os.listdir(runDatesPath + '/' + str( - g)) if not os.path.isfile(os.path.join(runDatesPath + '/' + str(g), f))] - dataRunsInt_g = [int(z[1:3]) for z in dataRuns_g] - - gfsRunDate = datetime(int(str(g)[0:4]), - int(str(g)[4:6]), - int(str(g)[6:8]), 00, 00, 00) - - if gfsRunDate < (datetime.now() - timedelta(days=7)): - shutil.rmtree(runDatesPath + '/' + str(g)) - print(runDatesPath + '/' + str(g)) - - -# Save completion files -f = open(download_path + '/' + prod_A + '/' + fDate + - '/' + runTime + '/' + prod_A + '.done', "w") -f.write(download_path_CK) -f.close() -f = open(download_path + '/' + prod_B + '/' + fDate + - '/' + runTime + '/' + prod_B + '.done', "w") -f.write(download_path_CK_B) -f.close() diff --git a/wgrib2/Dockerfile b/wgrib2/Dockerfile deleted file mode 100644 index 9367e6b..0000000 --- a/wgrib2/Dockerfile +++ /dev/null @@ -1,35 +0,0 @@ -# ====================================================================== -# Dockerfile to compile wgrib2 based on Alpine linux -# -# Homepage: http://www.cpc.ncep.noaa.gov/products/wesley/wgrib2/ -# Available versions: ftp://ftp.cpc.ncep.noaa.gov/wd51we/wgrib2/ -# ====================================================================== - -FROM ubuntu:20.04 AS base - -ENV CC=gcc -ENV FC=gfortran - -RUN apt-get update -RUN apt-get install -y build-essential gfortran wget file pip - -RUN wget -q -O /tmp/wgrib2.tgz https://ftp.cpc.ncep.noaa.gov/wd51we/wgrib2/wgrib2.tgz - -RUN tar -xf /tmp/wgrib2.tgz -C /tmp - -WORKDIR /tmp/grib2 -RUN ls -l - -RUN sed -i "s|MAKE_SHARED_LIB=0|MAKE_SHARED_LIB=1|g" makefile - -RUN make lib - -#RUN mkdir -p .local/lib/python3.8/site-packages - -RUN cp /tmp/grib2/lib/libwgrib2.so /usr/lib/python3.8/libwgrib2.so - -RUN python3 -m pip install numpy boto3 netCDF4 - -RUN wget -q -O /usr/lib/python3.8/pywgrib2_s.py https://ftp.cpc.ncep.noaa.gov/wd51we/pywgrib2_s/pywgrib2_s/pywgrib2_s.py - -ENTRYPOINT ["python3"] \ No newline at end of file diff --git a/wgrib2/README.md b/wgrib2/README.md deleted file mode 100644 index 1bf0b6b..0000000 --- a/wgrib2/README.md +++ /dev/null @@ -1,8 +0,0 @@ -# pywgrib2 Docker container - -This is a docker container setup to download and compile the latest pywgrib2 prgram. It's designed with a "from scratch" approach, compiling the raw source on an ubuntu 20:04 base image. -The image contains the wgrib2 libraries, python 3.8, as well as numpy. Additional Python modules will be added later. - -# Usage - -This is a work in progress image, and significant changes should be expected. The plan is to use it via a volume mounted that contains the Python script to run. \ No newline at end of file