Instructions feedback

[LoannPeurey]

Add comments to this issue for every piece of feedback you would like to provide in regard to the instructions in the handbook

[alecristia]

• cosmetic : https://laac-lscp.github.io/docs/create-a-new-dataset.html#audio-files : I suspect the text following the word WARNING was intended to be put inside the same yellow box

[alecristia]

• https://laac-lscp.github.io/docs/create-a-new-dataset.html#create-the-metadata-from-the-its-information "You can install the package" <-- probably add hyperlink to instructions (ie same link as el1000 in the previous sentence)

[alecristia]

• cosmetic : https://laac-lscp.github.io/docs/create-a-new-dataset.html#audio-files : I suspect the text following the word WARNING was intended to be put inside the same yellow box

perhaps this applies also to the warning below https://laac-lscp.github.io/docs/create-a-new-dataset.html#create-the-metadata-from-the-its-information

[alecristia]

I wonder: I can see the instructions for creating metadata from LENA its, and from scratch. What if I have a dataset with both? in the tsimanem2018 case, I'm going to opt for not splitting & not using the its data for the metadata, even though it would be nice to have accurate timestamps... but it seems like it'll be a tremendous amount of hassle. This will probably happen incredibly infrequently - perhaps tsimane2017 & 2018 are the only datasets in the world with mixed recordings (and LENA are infrequent within those datasets)

[alecristia]

update https://childproject.readthedocs.io/en/latest/processors.html#basic-audio-conversion so that the examples match our current use, namely:

• remove the split at 15h

[alecristia]

I may be missing this, but where are the actual instructions for its import? In https://laac-lscp.github.io/docs/create-a-new-dataset.html#lena-its I see how to fix the timeline if you split recs, but if I don't, then is my script basically:

"""
This script uses metadata/recordings.csv to import its annotations. It assumes recordings.csv contains columns 'its_filename' and 'duration' to run the importation
"""
import pandas as pd
from ChildProject.projects import ChildProject
from ChildProject.annotations import AnnotationManager

dataset_path = "."

#load the project and annotation manager
project = ChildProject(dataset_path)
am = AnnotationManager(project)

# we take a copy of the recordings.csv file of the dataset, that suits us because we have one importation per recording, as is usually the case with automated annotations
input_frame = pd.DataFrame.copy(project.recordings)
input_frame = input_frame.sort_values(['its_filename', 'recording_filename'])

#make sure that the duration for the recordings is set in recordings.csv, it should be imported with the metadata of the its
input_frame["raw_filename"]= input_frame['its_filename']
input_frame["set"] = 'its'
input_frame["range_onset"] = "0" #from the start of the audio...
input_frame["range_offset"]= input_frame["duration"] # ...to the end

#AC commented the following two lines for cases in which no split has been done
#for its, df in input_frame.groupby('its_filename'):
#    input_frame.loc[df.index,'time_seek'] = - df['duration'].cumsum().shift(1, fill_value = 0)

am.import_annotations(input_frame)

[alecristia]

I went ahead and ran that, and got this error:

Traceback (most recent call last):
File "scripts/import_its.py", line 28, in
am.import_annotations(input_frame)
File "/scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages/ChildProject/annotations.py", line 521, in import_annotations
assert_columns_presence("input", input, required_columns)
File "/scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages/ChildProject/tables.py", line 31, in assert_columns_presence
raise MissingColumnsException(name, missing)
ChildProject.tables.MissingColumnsException: dataframe input misses the following required columns: time_seek

Note that my script has those two lines commented out, but it seems import_annotations requires time_seek (assert_columns_presence("input", input, required_columns)). So that means that probably somewhere in validation of the dataset, we need to tell users that it's an error if they're missing this column. And I guess we should tell people to add it somewhere.

I'm not sure where though -- annotations.csv in my case is empty & has that column...

[alecristia]

it would be nice to have an idea of how long processes last -- I know this is super dependent on hardware, but in my case, I was surprised that the tsimanem2018 data (which has only 350 files, from max 80 long-form recs) is taking over 20 mins to be standardized in the cluster... So perhaps we can give a couple of examples, so that people can plan? say one in which you only have 1 CPU, 1 16h rec, another with 4 CPUs 100 recs... just a thought!

[LoannPeurey]

I may be missing this, but where are the actual instructions for its import? In https://laac-lscp.github.io/docs/create-a-new-dataset.html#lena-its I see how to fix the timeline if you split recs, but if I don't, then is my script basically:

"""
This script uses metadata/recordings.csv to import its annotations. It assumes recordings.csv contains columns 'its_filename' and 'duration' to run the importation
"""
import pandas as pd
from ChildProject.projects import ChildProject
from ChildProject.annotations import AnnotationManager

dataset_path = "."

#load the project and annotation manager
project = ChildProject(dataset_path)
am = AnnotationManager(project)

# we take a copy of the recordings.csv file of the dataset, that suits us because we have one importation per recording, as is usually the case with automated annotations
input_frame = pd.DataFrame.copy(project.recordings)
input_frame = input_frame.sort_values(['its_filename', 'recording_filename'])

#make sure that the duration for the recordings is set in recordings.csv, it should be imported with the metadata of the its
input_frame["raw_filename"]= input_frame['its_filename']
input_frame["set"] = 'its'
input_frame["range_onset"] = "0" #from the start of the audio...
input_frame["range_offset"]= input_frame["duration"] # ...to the end

#AC commented the following two lines for cases in which no split has been done
#for its, df in input_frame.groupby('its_filename'):
#    input_frame.loc[df.index,'time_seek'] = - df['duration'].cumsum().shift(1, fill_value = 0)

am.import_annotations(input_frame)

I think you actually should not comment those two lines, they take care of adding time_seek when the recordings are split but also if they are not. It will simply always give a time_seek of 0. which is what we want.

[alecristia]

that helped! but I get a new error:

Traceback (most recent call last):
File "/scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages/pandas/core/indexes/base.py", line 3361, in get_loc
return self._engine.get_loc(casted_key)
File "pandas/_libs/index.pyx", line 76, in pandas._libs.index.IndexEngine.get_loc
File "pandas/_libs/index.pyx", line 108, in pandas._libs.index.IndexEngine.get_loc
File "pandas/_libs/hashtable_class_helper.pxi", line 5198, in pandas._libs.hashtable.PyObjectHashTable.get_item
File "pandas/_libs/hashtable_class_helper.pxi", line 5206, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'format'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "scripts/import_its.py", line 27, in
am.import_annotations(input_frame)
File "/scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages/ChildProject/annotations.py", line 540, in import_annotations
builtin = input[input["format"].isin(converters.keys())]
File "/scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages/pandas/core/frame.py", line 3458, in getitem
indexer = self.columns.get_loc(key)
File "/scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages/pandas/core/indexes/base.py", line 3363, in get_loc
raise KeyError(key) from err
KeyError: 'format'

[LoannPeurey]

obvious answer will be to add

input_frame['format'] = 'its'

But that surprises me because some importations were runnin fine without it. I'll take a deeper look soon

[LoannPeurey]

I wonder: I can see the instructions for creating metadata from LENA its, and from scratch. What if I have a dataset with both? in the tsimanem2018 case, I'm going to opt for not splitting & not using the its data for the metadata, even though it would be nice to have accurate timestamps... but it seems like it'll be a tremendous amount of hassle. This will probably happen incredibly infrequently - perhaps tsimane2017 & 2018 are the only datasets in the world with mixed recordings (and LENA are infrequent within those datasets)

In those cases, I guess the easiest way is to create 2 versions of metadata files, one generated with its files and one from scratch and then merging them. Maybe we should mention that possibility

[alecristia]

input_frame['format'] = 'its'

thanks! that got rid of that error, but a triad of new ones emerged. They look interesting:

type 1: trying to import an NA its

an error occured while processing './annotations/its/raw/NA'
Traceback (most recent call last):
File "/scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages/ChildProject/annotations.py", line 437, in _import_annotation
df = converter.convert(path, filter, **params)
File "/scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages/ChildProject/converters.py", line 265, in convert
xml = etree.parse(filename)
File "src/lxml/etree.pyx", line 3538, in lxml.etree.parse
File "src/lxml/parser.pxi", line 1876, in lxml.etree._parseDocument
File "src/lxml/parser.pxi", line 1902, in lxml.etree._parseDocumentFromURL
File "src/lxml/parser.pxi", line 1805, in lxml.etree._parseDocFromFile
File "src/lxml/parser.pxi", line 1177, in lxml.etree._BaseParser._parseDocFromFile
File "src/lxml/parser.pxi", line 615, in lxml.etree._ParserContext._handleParseResultDoc
File "src/lxml/parser.pxi", line 725, in lxml.etree._handleParseResult
File "src/lxml/parser.pxi", line 652, in lxml.etree._raiseParseError
OSError: Error reading file './annotations/its/raw/NA': failed to load external entity "./annotations/its/raw/NA"

type 2: NA error despite there being an actual its - oh wait, type 2 and 3 refer to the same file!

an error occured while processing './annotations/its/raw/annotations/raw/its/e20180716_170035_013097.its'
Traceback (most recent call last):
File "/scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages/ChildProject/annotations.py", line 437, in _import_annotation
df = converter.convert(path, filter, **params)
File "/scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages/ChildProject/converters.py", line 265, in convert
xml = etree.parse(filename)
File "src/lxml/etree.pyx", line 3538, in lxml.etree.parse
File "src/lxml/parser.pxi", line 1876, in lxml.etree._parseDocument
File "src/lxml/parser.pxi", line 1902, in lxml.etree._parseDocumentFromURL
File "src/lxml/parser.pxi", line 1805, in lxml.etree._parseDocFromFile
File "src/lxml/parser.pxi", line 1177, in lxml.etree._BaseParser._parseDocFromFile
File "src/lxml/parser.pxi", line 615, in lxml.etree._ParserContext._handleParseResultDoc
File "src/lxml/parser.pxi", line 725, in lxml.etree._handleParseResult
File "src/lxml/parser.pxi", line 652, in lxml.etree._raiseParseError
OSError: Error reading file './annotations/its/raw/NA': failed to load external entity "./annotations/its/raw/NA"

Type 3: path repeated

Traceback (most recent call last):
File "/scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages/ChildProject/annotations.py", line 437, in _import_annotation
df = converter.convert(path, filter, **params)
File "/scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages/ChildProject/converters.py", line 265, in convert
xml = etree.parse(filename)
File "src/lxml/etree.pyx", line 3538, in lxml.etree.parse
File "src/lxml/parser.pxi", line 1876, in lxml.etree._parseDocument
File "src/lxml/parser.pxi", line 1902, in lxml.etree._parseDocumentFromURL
File "src/lxml/parser.pxi", line 1805, in lxml.etree._parseDocFromFile
File "src/lxml/parser.pxi", line 1177, in lxml.etree._BaseParser._parseDocFromFile
File "src/lxml/parser.pxi", line 615, in lxml.etree._ParserContext._handleParseResultDoc
File "src/lxml/parser.pxi", line 725, in lxml.etree._handleParseResult
File "src/lxml/parser.pxi", line 652, in lxml.etree._raiseParseError
OSError: Error reading file './annotations/its/raw/annotations/raw/its/e20180716_170035_013097.its': failed to load external entity "./annotations/its/raw/annotations/raw/its/e20180716_170035_013097.its"

In this last one, notice the path in ... failed to load external entity "./annotations/its/raw/**annotations/raw/**its/e20180716_170035_013097.its"

[alecristia]

full error log: https://gin.g-node.org/LAAC-LSCP/tsimanem2018/src/main/extra/fullerror.txt

[LoannPeurey]

This happens when some of the recordings don't have an its -- so if you have both LENA and non-LENA recordings in the same corpus. Then, the lines of recordings that don't have an its_filename should be removed:

input_frame.dropna(subset= 'its_filename', inplace=True)

Alternatively, if the NA is being treated as text rather than nan, then do:

input_frame = input_frame[input_frame.its_filename!= "NA"]