Switch taxon lookup away from NCBI (was: Incomplete display of Study 1103)

[bearkland]

I entered this study and entered 3 experiments. After refreshing a couple of times, I was still only seeing 2 experiments. Then the 3rd appeared. I also entered signatures but they are not showing.

[tosfos]

It looks like some caches are not being purged after we update certain modules. We're working on this issue.

[tosfos]

This individual study is being displayed, but let's keep this open while we work on a permanent fix. In the meantime, it is possible that some Studies, especially new ones, will not show all data. You should be able to fix that by clicking "..." and then "Refresh". Here is a screenshot:

[bearkland]

Thank you.

…

________________________________ From: Ike Hecht ***@***.***> Sent: Sunday, October 6, 2024 10:59 AM To: waldronlab/BugSigDB ***@***.***> Cc: Mary Bearkland ***@***.***>; Author ***@***.***> Subject: Re: [waldronlab/BugSigDB] Incomplete display of Study 1103 (Issue #248) You don't often get email from ***@***.*** Learn why this is important<https://aka.ms/LearnAboutSenderIdentification> * This email originates from a sender outside of CUNY. Verify the sender before replying or clicking on links and attachments. * This individual study is being displayed, but let's keep this open while we work on a permanent fix. In the meantime, it is possible that some Studies, especially new ones, will not show all data. You should be able to fix that by clicking "..." and then "Refresh". Here is a screenshot: image.png (view on web)<https://github.com/user-attachments/assets/b2cce4fb-b084-428e-90f8-5dc6a41c0891> — Reply to this email directly, view it on GitHub<#248 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AUTRT5ELPBR5G4HCUHRRT5TZ2FF5NAVCNFSM6AAAAABPA3RYZ2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGOJVGQ3TAMBQGU>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[Ka-di1]

I encountered this issue where the signatures weren’t showing up after saving. I found that making a small edit to the experiment and saving it again seems to resolve the problem.

[tosfos]

We're attacking the issues from a few different angles, so the problem is showing up less and less but still not fully resolved.

[tosfos]

We are currently not making any changes, but we want to continue further work on this in Staging.

We’re exploring options to optimize how we handle taxonomy data for BugSigDB. We have been seeing API rate limit errors when querying NCBI, and one potential solution we’ve been testing is Taxoniq, a Python-based tool for fast, offline querying of NCBI Taxonomy data. Our goal is to determine if Taxoniq could serve as a reliable alternative to our current approach of directly querying the NCBI API. To help us make an informed decision, we’d greatly appreciate your input on the following:

Data Completeness: The example at https://bugsigdb-staging.wikiworks.com/Project:Taxoniq (use admin as the user name and password) highlights one discrepancy between Taxoniq’s output and the NCBI API response. For instance, Taxoniq lacks the Lineage string (though we can derive it from LineageEx), and fields like MitoGeneticCode, Division, and OtherNames are missing. Are these gaps significant? Our current wiki usage seems limited to ScientificName, Rank, Lineage, and ParentTaxId—do you foresee any issues if we rely on Taxoniq’s subset of fields?

Update Frequency: Taxoniq’s documentation claims monthly updates, but updates might be less frequent. How critical is it for BugSigDB to have taxonomy data refreshed more often than, say, every few months? Would a delay in updates (e.g., 3-6 months) impact the accuracy or utility of the microbial signatures we track?

Additional Test Cases: To thoroughly evaluate Taxoniq, we’d like to test it against a broader range of taxonomy IDs relevant to BugSigDB. Could you suggest specific taxon examples—perhaps ones with complex lineages, synonyms, or frequent updates—that we should compare between Taxoniq and the NCBI API? This would help us confirm if Taxoniq consistently meets our needs across diverse use cases.

Practical Fit: Assuming we adapt our data handling (e.g., handling tag name differences like TaxId vs. tax_id) and derive Lineage from LineageEx, do you see Taxoniq as a viable drop-in replacement for our current system?

Looking forward to your thoughts!

Pinging @lwaldron @lgeistlinger

[tosfos]

Any thoughts here? The rate-limit issue is significant and should be resolved.

[lwaldron]

Argh, I'm sorry for having missed this, @tosfos. To answer your questions (@lgeistlinger @SvetlanaUP @cmirzayi please chime in anything else you have to add):

• Data Completeness - I can's see any problem with the more limited information bundled by taxoniq. We're not using any of it now and I can't think of why we would need it in the future. The discrepancy you noted looks like just seems a delay in taxoniq catching up to a recent change in NCBI, which is fine, see next point.
• Update Frequency - I don't think we need more frequent updates than every few months. We are currently following a 6-month release cycle at BugSigDBExports and that timescale is fine. As long as there is some program of regular updates, that's much more important for our purposes than what the actual frequency is. Using taxonomiq actually would have some advantage of reproducibility if we can list the taxonomiq version in BugSigDBExports, from an API call or page that will tell us what version of taxonomiq is in use.
• Practical fit - seems highly practical: less fragile to rate limiting or downtime from NCBI, and having the very latest from NCBI is not even relevant to us as long as we can get periodic updates.
• Additional Test Cases - I asked Gemini Deep Research (2.0 Flash) "What are some recent important changes in NCBI taxonomic classifications?" - output pasted below, and here is a spreadsheet with demonstrative NCBI IDs for each of the recent or upcoming changes listed below. I note that Gemini identified the 'TACK group' -> 'Thermoproteati' change you noticed, and the discontinuation of "superkingdom" that we've just had to adapt downstream software to.

• New Ranks: Domain and Realm: The rank "superkingdom" has been discontinued for Archaea, Bacteria, Eukaryota, and Viruses. "Domain" now replaces "superkingdom" for Archaea, Bacteria, and Eukaryota, while Viruses are now classified under the rank of "acellular root".[1, 2, 3, 4] Additionally, six major clades within Viruses have been assigned the new rank of "realm".[1, 2] These changes were implemented in early March 2025.[2]
• Updates to Prokaryotes (Bacteria and Archaea): The higher-level classification of prokaryotes has been updated with the introduction of the rank "kingdom". Several prominent groups have been renamed and elevated to the kingdom rank, including:
  • "DPANN group" is now kingdom Nanobdellati.
  • "TACK group" is now kingdom Thermoproteati.
  • Former phylum Euryarchaeota is now kingdom Methanobacteriati.
  • "Asgard group" is now kingdom Promethearchaeati.
  • "Terrabacteria group" is now kingdom Bacillati.
    Other phylum-level renamings include Nanoarchaeota to Nanobdellota and Candidatus Thermoplasmatota to Thermoplasmatota. These changes began in October 2024 and continued through December 2024.
• Upcoming Changes to Viruses: In spring 2025, NCBI plans to add binomial species names to approximately 3000 viruses to align with the International Code of Virus Classification and Nomenclature (ICVCN).[5, 4, 6, 7] The former species names will be retained in the lineage below the new binomial names.[5]
• Updates to Yeasts (Saccharomycotina): The subphylum Saccharomycotina has undergone a significant reorganization, with the introduction of six new classes and eleven new orders.[1, 4, 8] This update impacts over six million records in the NCBI Taxonomy database.[4, 8]

[tosfos]

Ok. We'll go ahead with this change. Thanks.

[cmirzayi]

I just wanted to check in on this. Some of our curators are still reporting that they don't see signatures right away and it takes 1-2 days to appear.

[tosfos]

I was hoping things were improved. We'll bump priority on this migration then and update you once it's ready for testing.

[tosfos]

The switch to Taxoniq has been deployed to staging. We're doing some final testing before deploying to production.