Some MGnify genome collections are missing the file that uniquely names genes in a pangenome

[pbradz]

Issue: pangenomes only named with "group_XXXX" codes instead of MGYGXXXXX codes in some MGnify collections. (Panaroo is supposed to generate a file that links these but MGnify did not include this file by default in older versions of the pipeline.)

Approach is to match group IDs to genes/gene families by using exact matching from the .gff files.

[majernik14]

To fix this issue, I will be extracting the MYGY ids for the clustered genes per pangenome from the UHGP-50.tsv file. This should create a list of only the genes that are the representative sequence in the pangenome. Then I will use this list to parse the .gffs and use mmseqs or vsearch for exact match to the pan-genome.fna file (contains group_XXXX and DNA sequences).

[majernik14]

Not all databases in MGnify have this issue and this should no longer be an issue in new releases of MGnify databases.

State of databases in MGnify:

• human-gut: Missing linker
• chicken-gut: Has linker
• cow-rumen: Missing linker
• human-oral: Missing linker
• human-vaginal: Has linker
• marine: Has linker
• marine-eukaryotes: NO pangenomes
• mouse-gut: Has linker
• non-model-fish-gut: Has linker
• pig-gut: Missing linker
• sheep-gut: Has linker
• zebrafish-gut: Needs to be re-downloaded -- Missing linker

Of the databases with linkers they all have a gene-presence-absence.csv that lists: gene (group_##), non-unique gene name, annotation, gene name in the genome (MYGY###_##), again for different genome, again until all genomes are listed
This file will need to be parsed in next steps!

These two db organization seems to cover all dbs.

[majernik14]

I am currently working on identifying a good gff parser:
X BCBio tools (support discontinued 8/2024)

[pbradz]

I know we chatted about this but for posterity, I was able to get GFF with FASTA parsing working using gff3-py; see workflow/scripts/extract_ssu_from_gff.py in this repository. The one issue is that it generates a LOT of logging content, like multiple gigabytes (most GFFs are not perfectly compliant in some way so it "errors" on almost every line, but still actually parses the files just fine) so once I got it working I turned off logging just around the command that reads the GFFs in. Let me know if you have questions about that.