🥇 Challenge : compute H2O on Radis-Lab

[erwanp]

In the latest commits to RADIS-Lab 567b025 I added the auto-download of HITEMP H2O lines.

Challenge is to be able to run a large range, high-temperature H2O computation directly on RADIS-Lab ! (no install needed !) 🥇 🥇

Currently, we are limited because :

1. the .h5 cache files are not generated during the Docker PostBuild phase (it crashed) ; probably because of Memory error while unpacking them.
2. you can still start computing HITEMP H2O from Radis-Lab, but it takes ages to generate the .h5 files from the .par files.
3. Then, it will be a challenge to know if 8 GB RAM provided by GESIS are enough to compute an H2O spectrum.

Fix :

1. could be helped a lot by using Vaex to stream the .par files into HDF5 files (a small change of the Pandas syntax used in HITEMP-Hdf5 improvement radis#203 and Auto fetch hitemp for CO2 & H2O radis#281). If .h5 are cached during Docker build; the Docker build will likely take 2 hrs ; but user computations will be instantaneous then ! ( kind reminder that we need Improve launch time : separate environment (+ databases) from content (notebooks) #13 to be able to add Example Notebooks here without restarting the Docker !)
2. May work after @gagan-aryan 's Reduce Memory Usage Reduce Memory usage radis#118 project

[Springder]

I failed to compute H2O on Radis-Lab(990-1000 cm-1), even if 8 GB RAM was provided by GESIS. It's really necessary to unzip all par files when I just need 990-1000 cm-1, but in fact, that's what it does, converting all files into h5 files. This may be the reason for the failure？

[erwanp]

Yes! Good news : in 0.9.30, we only download, unzip & parse the files needed. This is already implemented in develop version (radis/radis#281) and will be available on RADIS-Lab as soon as 0.9.30 is released

[Springder]

Any independent file conversion from .par to h5 will not exceed 8 GB RAM on Radis-Lab.
So, I try to compute H2O from 2260 to 2270 cm-1, then .par files were convertiing one by one until the last file (01_2250-2500_HITEMP2010) was unzipped and parsed. In current version, this is an ingenious way, because the interval we calculate is just in the last file, so we can ensure that there is enough RAM to unzip & parse the files.

[erwanp]

Hello ! RADIS-Lab was updated with the newest RADIS 0.10.x #25

I also managed to get it to pre-parse the full H2O database #26

Now, a 2260 - 2270 cm-1 works very well :

from radis import calc_spectrum
from astropy import units as u
s = calc_spectrum(wavenum_min = 2260 / u.cm, 
                  wavenum_max = 2270 / u.cm,
                  molecule = 'H2O',
                  isotope = '1,2,3',
                  pressure = 1.01325 * u.bar,
                  mole_fraction = 1,
                  path_length = 1 * u.cm,
                  Tgas=3000, 
                  databank='hitemp',
                  verbose=3,        # adding some extra info for the first computation
                  )
s.plot()
# without verbose=False this will show all the input parameters. 
# With verbose=2,3,etc... we get increasing number of details about the calculation. 

A larger spectrum (ex : 2000 - 4000 cm-1 , 1 isotope) is still out of reach with the limited 8 GB of GESIS, and I run out-of-memory at loading

Generated dataframe from /home/jovyan/.radisdb/hitemp/H2O-01_02000-02250_HITEMP2010.h5 in 31.88s (3283544 rows, 31 columns, 2068.63 MB)
Generated dataframe from /home/jovyan/.radisdb/hitemp/H2O-01_02250-02500_HITEMP2010.h5 in 29.26s (2971420 rows, 31 columns, 1871.99 MB)

But 31 columns are loaded, most of them useless for equilibrium calculations, so it could quickly be optimized further.

Index(['id', 'iso', 'wav', 'int', 'A', 'airbrd', 'selbrd', 'El', 'Tdpair',
       'Pshft', 'ierr', 'iref', 'lmix', 'gp', 'gpp', 'ju', 'Kau', 'Kcu', 'Fu',
       'symu', 'jl', 'Kal', 'Kcl', 'Fl', 'syml', 'v1u', 'v2u', 'v3u', 'v1l',
       'v2l', 'v3l'],
      dtype='object')

[erwanp]

@gagan-aryan with Radis version 0.12 the algorithm is even better: you can already compute much larger ranges. I tried the code above with the 2000 - 4000 cm-1 range and it worked. You can try to push it further.

[ariadnc]

Hello, I tried using the HITEMP-H2O data and it is not available in the jupyter-lab environment. I also tried running it in my computer but it seams like there is a problem converting most of the hitemp molecules .par files in .h5 files. The errors you get is that is not concatenated and that there is an error on the data stream.

Do you know if I can get the .h5 directly without having to convert it anywhere? I see that the path being path (example):
path: "/home/jovyan/.radisdb/hitemp/NO2-10_HITEMP2019.hdf5"

[erwanp]

Hello @ariadnc thanks for reporting

unfortunately Radis Lab is not maintained I'm even surprised it still works. Id suggest to run the codes on a local machine first. Do you have use cases that specifically require an online version ?

[ariadnc]

Radis Lab does not work anymore, I was in the virtual machine because the local machine did not work for the HITEMP database. The local machine does not convert the downloaded files from the HITRAN HITEMP website into the format that the library uses for analysis. This only happens with the HITEMP database, where it either does not find the .bz2 files or it just downloads a corrupted file that can not be extracted properly. But I did some changes in the code in my local machine and is solved now.

How I went around this problem is downloading it myself from the website and add it it to the hitemp folder in my database in my local machine. That method worked for molecules like CO or OH (witch only have one file). But for both H2O and CO2 molecules witch is composed of a list of .zip files it did not even have access to them.

I solved these issues by first adding the download url and path manually in the .json fie of the radis database. Then I downloaded the files manually from the website and added them to the database in my local maachine. And then I changed the function fetch urls and added the names of the files:

 def fetch_urlnames(self):
        r"""requires connection"""

        if self.urlnames is not None:
            return self.urlnames

        molecule = self.molecule

        if molecule in ["H2O", "CO2"]:

            base_url, Ntotal_lines_expected, _, _ = self.fetch_url_Nlines_wmin_wmax()
            response = urllib.request.urlopen(base_url)
            response_string = response.read().decode()
            print(response_string)
            if molecule == "CO2":
                base_url = "https://hitran.org/files/HITEMP/HITEMP-2010/CO2_line_list/"
            
                inputfiles = [ 
                    "02_00000-00500_HITEMP2010.zip",
                    "02_00500-00625_HITEMP2010.zip",
                    "02_00625-00750_HITEMP2010.zip",
                    "02_00750-01000_HITEMP2010.zip",
                    "02_01000-01500_HITEMP2010.zip",
                    "02_01500-02000_HITEMP2010.zip",
                    "02_02000-02125_HITEMP2010.zip",
                    "02_02125-02250_HITEMP2010.zip",
                    "02_02250-02500_HITEMP2010.zip",
                    "02_02500-03000_HITEMP2010.zip",
                    "02_03000-03250_HITEMP2010.zip",
                    "02_03250-03500_HITEMP2010.zip",
                    "02_03500-03750_HITEMP2010.zip",
                    "02_03750-04000_HITEMP2010.zip",
                    "02_04000-04500_HITEMP2010.zip",
                    "02_04500-05000_HITEMP2010.zip",
                    "02_05000-05500_HITEMP2010.zip",
                    "02_05500-06000_HITEMP2010.zip"
                    ]
            else:
                inputfiles = [ # fix here, add all zips YUSUF CTRL F
                    "01_00000-00050_HITEMP2010.zip",
                    "01_00050-00150_HITEMP2010.zip",
                    "01_00150-00250_HITEMP2010.zip",
                    "01_00250-00350_HITEMP2010.zip",
                    "01_00350-00500_HITEMP2010.zip",
                    "01_00500-00600_HITEMP2010.zip",
                    "01_00600-00700_HITEMP2010.zip",
                    "01_00700-00800_HITEMP2010.zip",
                    "01_00800-00900_HITEMP2010.zip",
                    "01_00900-01000_HITEMP2010.zip",
                    "01_01000-01150_HITEMP2010.zip",
                    "01_01150-01300_HITEMP2010.zip",
                    "01_01300-01500_HITEMP2010.zip",
                    "01_01500-01750_HITEMP2010.zip",
                    "01_01750-02000_HITEMP2010.zip",
                    "01_02000-02250_HITEMP2010.zip",
                    "01_02250-02500_HITEMP2010.zip",
                    "01_02500-02750_HITEMP2010.zip",
                    "01_02750-03000_HITEMP2010.zip",
                    "01_03000-03250_HITEMP2010.zip",
                    "01_03250-03500_HITEMP2010.zip",
                    "01_03500-04150_HITEMP2010.zip",
                    "01_04150-04500_HITEMP2010.zip",
                    "01_04500-05000_HITEMP2010.zip",
                    "01_05000-05500_HITEMP2010.zip",
                    "01_05500-06000_HITEMP2010.zip",
                    "01_06000-06500_HITEMP2010.zip"
                    ]
                
            urlnames = [join(base_url, f) for f in inputfiles]

With these changes I could access and converted the files that I needed to the .h5 format that allows me to access the data and make the calculations I need.