update evaluation code to include edge evaluation

[ntalluri]

This excalidraw contains the plans for the edge evaluation code: https://excalidraw.com/#json=JHcEVamDksDCAtv62ted5,ue4zzq3ZLvu3cPJvR-MQWA

The first step will be to integrate the edge files into the code so they can be executed exactly as intended (meaning edge gold standards should only run with edge evaluation, and node gold standards should only run with node evaluation).

For now, the plan is that each gold standard dataset will contain either a single edge file or a single node file; both cannot exist in the same dataset simultaneously. Also each edge dataset will include the transformed verison (fully undirected, fully directed, and mixed) of the data using transformation logic for gold standard edge files from interactome.py.

We will also need to update the file generation / evaluation logic in the Snakemake workflow. I’m considering updating this specific line:

dataset_gold_standard_pairs = [f"{dataset}-{gs_values['label']}" for gs_values in _config.config.gold_standards.values() for dataset in gs_values['dataset_labels']]

so that it can explicitly differentiate between edge and node files.

Once edge file integration is complete, the next steps will be:
2) Visualization updates: implement edge-specific visualization functions (based on the existing node code) in separate PRs.
3) General updates: add test cases, update naming conventions, and aggregate results per algorithm.

[ntalluri]

I'm hoping to not do a lot more of complicated input parsing of the gold standard data - avoiding a dataset.py file - but there is a chance I will need to.

[agitter]

I reviewed the diagram and agree with the plan. I left a couple comments.

For now, the plan is that each gold standard dataset will contain either a single edge file or a single node file

I would go further and say that should always be the case. If someone gives us an edge gold standard, they cannot add additional nodes to it. However, we can take the nodes that belong to those edges and do a node-based evaluation with them.

As Anna said, we want to make sure precision-recall plots always compare the same thing, where the gold standard is

• nodes
• edge list read in as a mixed graph
• edge list converted to undirected edges
• edge list converted to directed edges

[ntalluri]

@agitter did you leave comments on the excalidraw or are the comments what you posted on this issue? If on the excalidraw, we will need to get an updated link with your comments.

[ntalluri]

I would go further and say that should always be the case. If someone gives us an edge gold standard, they cannot add additional nodes to it. However, we can take the nodes that belong to those edges and do a node-based evaluation with them.

I agree with this, but for this first phase, that logic will be harder to implement. I think this is something that can be updated after in a phase two.

[agitter]

#375 added a function sort_and_deduplicate_undirected. We will need to apply that to the predicted pathway edges before evaluating them to make sure the undirected edges are sorted in the same manner.

[ntalluri]

Four more new PRs will come from this:

1. De-dupe and order undirected

• Apply the sort_and_deduplicate_undirected function to predicted pathway edges before evaluation to ensure consistent sorting of undirected edges.

2. Edge ensemble parameter selection and evaluation

• I will need to add in new functions to perform edge ensemble evaluation across the three edge evaluation types.
• Create two new Snakemake rules for edge-specific evaluations (aggregate per algorithm and dataset-specific)

3. Edge pca chosen parameter selection and evaluation evaluation: Edge evaluation pca-chosen #432

• Reuse existing code from PCA_chosen and the logic that determines which evaluation function to call by adding in edge porition
• Implement a new edge_precision_and_recall function and a new visualization function for edge data
• Create two new Snakemake rules for edge-specific evaluations (aggregate per algorithm and dataset-specific)

4. Edge no parameter selection and evaluation evaluation: edge evaluation no selection #437

• Reuse existing logic for selecting the appropriate evaluation function by adding in edge portion.
• Use the functions developed from the PCA-chosen evaluation (no new functions need to be added after evaluation: Edge evaluation pca-chosen #432 is merged)
• Create two new Snakemake rules for edge-specific evaluations (aggregate per algorithm and dataset-specific)