Indexing policy for legacy unreplicated data

[lliming]

LLNL's Solr index served as an index for several ESGF data nodes that didn't have their own index.
Consequently, LLNL's index contains metadata for datasets that have never been stored at LLNL, ANL, or ORNL.
Q: Should ANL's or ORNL's Phase I indices contain these entries?
Q: Should the Phase II consolidated index contain these entries?

[bstrdsmkr]

ORNL believes these datasets have been replicated, does anyone have evidence to the contrary? If so, we'd like to get them replicated. This means the likely answer to both questions should be yes, though that might be a policy question for @climate-dude ?

[sashakames]

I think there might be some misunderstanding. These are datasets published by NASA, NOAA, CCCma (canadians) DIAS (Japanese) sites, Taiwan, and several of the Korean and Chinese sites. Not all these datasets are replicated. What is crucial are the records that are hosted in the LLNL Solr. We don't need multiple copies of the records migrated to all the DOE site indexes. It would probably be easiest to just migrate those along with the LLNL records when migration time comes. I will produce a list of dataset and file counts for the data nodes.

[jpnavarro]

Moved issue from esgf-1.5-design to esgf-1.5-storage-plan.

[jpnavarro]

All the Dataset entries and LLNL and all the File entries at LLNL, ORNL, and ANL will be combined into the single consolidate ESGF-1.5 Globus Search index, including LLNL Solr File entries for files not at LLNL, ORNL, or ANL.