HUC10 summary tool

[philipbaileynar]

Write a new CC tool that takes an RSContext project and writes inventory level metrics (min elevation, hypsometry, major veg classes) to S3 for use in Athena (and reports).

@MattReimer can you please advise on how best to handle getting the data to Athena if docker box doesn't have boto3 and Fargate can't use AWS CLI?

[MattReimer]

I'm not really sure what the best way to go here is. The riverscapes-tools cybercastor engine is designed to only be able to run tools and put the results up in the Data Exchange.

1. Adding Athena Access to the riverscapes-tools cybercastor engine

We can technically bake this in if we rebuild the cybercastor engine but given the fragility of the riverscapes-tools repo I'm nervous about touching it at all.

Furthermore it introduces a pretty egregious security concern whereby any tool can wipe out ALL of our athena data. I'm not concerned about deliberate naughtiness but this kind of thing is easy to do by accident and there are lots of people who can write code inside riverscapes-tools so opening that up might not be a great idea.

Also this means we need to add boto3 and/or aws cli to the docker box which I don't love because it collides with some of our other pip requirements and may in fact break our build.

2. Put this new tool on a branch like RiverscapesNZ and create a new Cybercastor engine just for this tool.

This would let us install boto3 or the aws cli on the dockerbox and give us lots more control. It's probably the right way to go but it involves a LOT of extra infrastructure to do something very small.

Also we might have the same problems building the docker box so this could be a lot of work to tease it out.

.... I'm going to keep thinking here. Neither of these options are sitting well with me.

[philipbaileynar]

Intrigued by your option 2.

Additonal info....

This tool will be doing GDAL operations, rasterio, pandas and zonal stats.

I see us writing a few of these tools.

One for HUC10 for sure... This tool I see writing a single JSON file per HUC10 and putting it on S3.

but potentially other GIS processors that go grab some data, reduce it and put on S3. i.e. we could have a StreamCAT tool that calls an API, gets some data and formats to JSON to go on S3 for Athena. In other words, bringing data to Athena for context to our metrics, but not needed by our models.

We don't have to go crazy. These are INFREQUENT operations (HUC10 elevation is not changing!) Version 1 could just run on our local machines.

[weblorin]

Just to clarify - Athena is essentially read only. For traditional Athena tables (like those using Parquet, CSV, JSON, etc. with the Hive metastore) that are not Iceberg, you must delete the underlying data files in Amazon S3. We can give riverscapes tools access only to specific S3 buckets.