Improve performance of SDAP-517

[RKuttruff]

Try to minimize time spent in xarray.open_zarr after SDAP startup. We should run open_zarr for each new dataset in webapp driver at least once upon discovery to ensure validity. This is an especially prevalent issue with Spark. The Spark workers would open ALL datasets every time they were given a task, which could introduce severe performance penalties.

This PR will:

• Share the opened dataset(s) required for Spark algorithms to the workers. (done: note 1)
• Minimize time for Zarr backend updates
  • Should avoid regular blocking call to open_zarr. Either make it lazy (open on use), or run it asynchronously (use a future)
    • Did lazy open with checks it's always done in driver
  • Credential rotation can stay as is re: blocking behavior

Note 1 - Spark Algs

Despite efforts to do so, I could not find a way to make this behavior automatic. There are some manual steps that need to be taken in the Spark algorithm definition. These are fortunately fairly simple and, if done incorrectly or if something goes wrong, the old behavior should be used as a fallback.

1. The Spark driver code should invoke NexusTileService.save_to_spark using the SparkContext object from the NexusCalcSparkHandler or the SDAP webservice.nexus_tornado.app_builders.SparkContextBuilder.SparkContextBuilder.get_spark_context() method and all the datasets that will be worked with.
2. The executor code should get its NexusTileService instance from the provided tile_service_factory with the kwargs spark=True, collections=[...] where the collections kwarg is a list of all the dataset names saved in step 1.

def spark_driver(tiles, ds1, ds2, tile_service_factory, sc, spark_nparts=1):
  NexusTileService.save_to_spark(sc, ds1, ds2)
  
  tiles_spark = [(ds1, ds2, tile) for tile in tiles]
  
  rdd = sc.parallelize(tiles_spark, spark_nparts)
  results = rdd.flatMap(partial(calc_executor, tile_service_factory)).collect()
  results = list(itertools.chain.from_iterable(results))
  return results

def calc_executor(tile_service_factory, spark_tile):
  tile, ds1, ds2 = spark_tile

  tile_service = tile_service_factory(spark=True, collections=[ds1, ds2])
  
  # Do work

  return result