Merge branch 'main' into feat-torchmetrics-eval

Author: martibosch

CONTRIBUTING.md

@@ -1,6 +1,6 @@
 # Developer's Guide
 
-Depends on Python 3.5+
+Depends on Python 3.9+
 
 ## Getting started
 
@@ -9,7 +9,7 @@ Depends on Python 3.5+
 2. Clone your copy of the repository.
 
    - **Using ssh**:
-   
+
      ```bash
      git clone [email protected]:[your user name]/DeepForest.git
      ```
@@ -83,7 +83,7 @@ $ pytest -v
 
 We use [yapf](https://github.com/google/yapf) for code formatting and style checking.
 
-The easiest way to make sure your code is formatted correctly is to integrate it into your editor.  
+The easiest way to make sure your code is formatted correctly is to integrate it into your editor.
 See [EDITOR SUPPORT](https://github.com/google/yapf/blob/main/EDITOR%20SUPPORT.md).
 
 You can also run yapf from the command line to cleanup the style in your changes:
@@ -111,7 +111,7 @@ $ conda build conda_recipe/meta.yaml -c conda-forge -c defaults
 
 Update the Conda recipe after every release.
 
-Clone the [Weecology staged recipes](https://github.com/weecology/staged-recipes).  
+Clone the [Weecology staged recipes](https://github.com/weecology/staged-recipes).
 Checkout the deepforest branch, update the `deepforest/meta.yaml` with the new version and the sha256 values. Sha256 values are obtained from the source on [PYPI download files](https://pypi.org/project/deepforest/#files) using the deepforest-{version-number}.tar.gz.
 
 ```jinja
@@ -131,7 +131,7 @@ $ docformatter --in-place --recursive src/deepforest/
 
 ### Update Documentation
 
-The documentation is automatically updated for changes in functions.  
+The documentation is automatically updated for changes in functions.
 However, the documentation should be updated after the addition of new functions or modules.
 
 Change to the docs directory and use `sphinx-apidoc` to update the doc's `source`. Exclude the tests and setup.py documentation.
@@ -198,4 +198,4 @@ model.push_to_hub("weecology/deepforest-livestock")
 
 The model will be uploaded to [https://huggingface.co/weecology/[model-name]](https://huggingface.co/weecology/[model-name])
 
-Note: You must have appropriate permissions in the weecology organization to upload models.
+Note: You must have appropriate permissions in the weecology organization to upload models.

dev_requirements.txt

@@ -1,4 +1,4 @@
-albumentations>=1.0.0,<2.0.0
+albumentations>=2.0.0
 aiolimiter
 aiohttp
 bump-my-version

docs/source/deepforest.rst

@@ -8,6 +8,7 @@ Subpackages
    :maxdepth: 4
 
    deepforest.data
+   deepforest.datasets
 
 Submodules
 ----------
@@ -28,14 +29,6 @@ deepforest.callbacks module
    :undoc-members:
    :show-inheritance:
 
-deepforest.dataset module
--------------------------
-
-.. automodule:: deepforest.dataset
-   :members:
-   :undoc-members:
-   :show-inheritance:
-
 deepforest.evaluate module
 --------------------------
 

docs/user_guide/03_cropmodels.md

@@ -14,7 +14,7 @@ Why would you want to apply a model directly to each crop? Why not train a multi
 
 While that approach is certainly valid, there are a few key benefits to using CropModels, especially in common use cases:  
 
-- **Flexible Labeling**: Object detection models require that all objects of a particular class be annotated within an image, which can be impossible for detailed category labels. For example, you might have bounding boxes for all ‘trees’ in an image, but only have species or health labels for a small portion of them based on ground surveys. Training a multi-class object detection model would mean training on only a portion of your available data. 
+- **Flexible Labeling**: Object detection models require that all objects of a particular class be annotated within an image, which can be impossible for detailed category labels. For example, you might have bounding boxes for all 'trees' in an image, but only have species or health labels for a small portion of them based on ground surveys. Training a multi-class object detection model would mean training on only a portion of your available data. 
 - **Simpler and Extendable**: CropModels decouple detection and classification workflows, allowing separate handling of challenges like class imbalance and incomplete labels, without reducing the quality of the detections. Two-stage object detection models can be finicky with similar classes and often require expertise in managing learning rates. 
 - **New Data and Multi-sensor Learning**: In many applications, the data needed for detection and classification may differ. The CropModel concept provides an extendable piece that allows for advanced pipelines.
 
@@ -177,4 +177,225 @@ class CustomCropModel(CropModel):
 
 # Create an instance of the custom CropModel
 model = CustomCropModel()
+```
+
+## Making Predictions Outside of predict_tile
+
+While `predict_tile` provides a convenient way to run predictions on detected objects, you can also use the CropModel directly for classification tasks. This is useful when you have pre-cropped images or want to run classification independently.
+
+### Loading a Trained Model
+
+```python
+from deepforest.model import CropModel
+from pytorch_lightning import Trainer
+from torchvision.datasets import ImageFolder
+import numpy as np
+
+# Load a trained model from checkpoint
+cropmodel = CropModel.load_from_checkpoint("path/to/checkpoint.ckpt")
+
+# The model will automatically load:
+# - The model architecture and weights
+# - The label dictionary mapping class names to indices
+# - The number of classes
+# - Any hyperparameters saved during training
+```
+
+### Making Predictions on a Dataset
+
+```python
+# Create a validation dataset
+from torchvision.datasets import ImageFolder
+val_ds = ImageFolder(root=root_dir, transform=cropmodel.get_transform(augment=False))
+
+# Get predictions and labels
+images, labels, predictions = cropmodel.val_dataset_confusion(return_images=True)
+
+# Create dataloader
+crop_dataloader = cropmodel.predict_dataloader(val_ds)
+
+# Run prediction
+trainer = Trainer(
+    gpus=1, 
+    accelerator="gpu", 
+    max_epochs=1, 
+    logger=False, 
+    enable_checkpointing=False
+)
+crop_results = trainer.predict(cropmodel, crop_dataloader)
+
+# Process results using the built-in postprocessing method
+label, score = cropmodel.postprocess_predictions(crop_results)
+
+# Convert numeric labels to class names
+label_names = [cropmodel.numeric_to_label_dict[x] for x in label]
+```
+
+### Making Predictions on Single Images
+
+You can also make predictions on individual images or batches:
+
+```python
+import torch
+from PIL import Image
+
+# Load and preprocess a single image
+image = Image.open("path/to/image.jpg")
+transform = cropmodel.get_transform(augment=False)
+tensor = transform(image).unsqueeze(0)  # Add batch dimension
+
+# Make prediction
+with torch.no_grad():
+    output = cropmodel(tensor)
+    # Convert to numpy for postprocessing
+    output = output.cpu().numpy()
+    # Use the same postprocessing method
+    label, score = cropmodel.postprocess_predictions([output])
+    class_name = cropmodel.numeric_to_label_dict[label[0]]
+    confidence = score[0]
+```
+
+## Model Architecture and Training
+
+The CropModel uses a ResNet-50 backbone by default, but can be customized with any PyTorch model. The model includes:
+
+- A classification head with the specified number of classes
+- Standard image preprocessing (resize to 224x224, normalization)
+- Data augmentation during training (random horizontal flips)
+- Accuracy and precision metrics for evaluation
+
+### Training Process
+
+```python
+# Initialize model
+crop_model = CropModel(num_classes=2)
+
+# Create trainer
+crop_model.create_trainer(
+    max_epochs=10,
+    accelerator="gpu",
+    devices=1
+)
+
+# Load data
+crop_model.load_from_disk(
+    train_dir="path/to/train",
+    val_dir="path/to/val"
+)
+
+# Train
+crop_model.trainer.fit(crop_model)
+
+# Validate
+crop_model.trainer.validate(crop_model)
+
+# Save checkpoint
+crop_model.trainer.save_checkpoint("model.ckpt")
+```
+
+### Evaluation
+
+The model provides several evaluation metrics:
+
+```python
+# Get validation metrics
+metrics = crop_model.trainer.validate(crop_model)
+
+# Get confusion matrix
+images, labels, predictions = crop_model.val_dataset_confusion(return_images=True)
+```
+
+### Confusion Matrix Visualization
+
+You can visualize the confusion matrix in several ways:
+
+```python
+import matplotlib.pyplot as plt
+from torchmetrics.classification import MulticlassConfusionMatrix
+import seaborn as sns
+
+# Method 1: Using torchmetrics
+metric = MulticlassConfusionMatrix(num_classes=crop_model.num_classes)
+metric.update(preds=predictions, target=labels)
+fig, ax = metric.plot()
+plt.title("Confusion Matrix")
+plt.show()
+
+# Method 2: Using seaborn with val_dataset_confusion
+images, labels, predictions = crop_model.val_dataset_confusion(return_images=True)
+confusion_matrix = np.zeros((crop_model.num_classes, crop_model.num_classes))
+for true, pred in zip(labels, predictions):
+    confusion_matrix[true][pred] += 1
+
+# Plot with seaborn
+plt.figure(figsize=(10, 8))
+sns.heatmap(confusion_matrix, 
+            annot=True, 
+            fmt='g',
+            xticklabels=list(crop_model.label_dict.keys()),
+            yticklabels=list(crop_model.label_dict.keys()))
+plt.title("Confusion Matrix")
+plt.xlabel("Predicted")
+plt.ylabel("True")
+plt.show()
+
+# Get per-class metrics
+from torchmetrics.classification import MulticlassPrecision, MulticlassRecall, MulticlassF1Score
+
+precision = MulticlassPrecision(num_classes=crop_model.num_classes)
+recall = MulticlassRecall(num_classes=crop_model.num_classes)
+f1 = MulticlassF1Score(num_classes=crop_model.num_classes)
+
+precision_score = precision(torch.tensor(predictions), torch.tensor(labels))
+recall_score = recall(torch.tensor(predictions), torch.tensor(labels))
+f1_score = f1(torch.tensor(predictions), torch.tensor(labels))
+
+print(f"Precision: {precision_score:.3f}")
+print(f"Recall: {recall_score:.3f}")
+print(f"F1 Score: {f1_score:.3f}")
+```
+
+This will give you a comprehensive view of your model's performance, including:
+- A visual confusion matrix showing true vs predicted classes
+- Per-class precision, recall, and F1 scores
+- The ability to identify which classes are most commonly confused with each other
+
+The confusion matrix is particularly useful for:
+- Identifying class imbalance issues
+- Finding classes that are frequently confused
+- Understanding the model's strengths and weaknesses
+- Guiding decisions about data collection and model improvement
+
+## Advanced Usage
+
+### Custom Model Architecture
+
+You can use any PyTorch model as the backbone:
+
+```python
+from torchvision.models import resnet101
+
+# Initialize with custom model
+backbone = resnet101(weights='DEFAULT')
+crop_model = CropModel(
+    num_classes=2,
+    model=backbone
+)
+```
+
+### Custom Training Loop
+
+You can subclass CropModel to customize the training process:
+
+```python
+class CustomCropModel(CropModel):
+    def training_step(self, batch, batch_idx):
+        x, y = batch
+        outputs = self.forward(x)
+        loss = F.cross_entropy(outputs, y)
+        
+        # Add custom metrics
+        self.log("custom_metric", value)
+        
+        return loss
 ```

docs/user_guide/07_scaling.md

@@ -34,68 +34,28 @@ https://lightning.ai/docs/pytorch/latest/clouds/cluster_advanced.html#troublesho
 
 ## Prediction
 
-Often we have a large number of tiles we want to predict. DeepForest uses [PyTorch Lightning](https://lightning.ai/docs/pytorch/stable/) to scale inference. This gives us access to powerful tools for scaling without any changes to user code. DeepForest automatically detects whether you are running on GPU or CPU. The parallelization strategy is to run each tile on a separate GPU, we cannot parallelize crops from within the same tile across GPUs inside of main.predict_tile(). If you set m.create_trainer(accelerator="gpu", devices=4), and run predict_tile, you will only use 1 GPU per tile. This is because we need access to all crops to create a mosiac of the predictions.
+Often we have a large number of tiles we want to predict. DeepForest uses [PyTorch Lightning](https://lightning.ai/docs/pytorch/stable/) to scale inference. This gives us access to powerful tools for scaling without any changes to user code. DeepForest automatically detects whether you are running on GPU or CPU. 
 
-### Scaling prediction across multiple GPUs
+There are three dataset strategies that *balance cpu memory, gpu memory, and gpu utilization* using batch sizes. 
 
-There are a few situations in which it is useful to replicate the DeepForest module across many separate Python processes. This is especially helpful when we have a series of non-interacting tasks, often called 'embarrassingly parallel' processes. In these cases, no DeepForest instance needs to communicate with another instance. Rather than coordinating GPUs with the associated annoyance of overhead and backend errors, we can just launch separate jobs and let them finish on their own. One helpful tool in Python is [Dask](https://www.dask.org/). Dask is a wonderful open-source tool for coordinating large-scale jobs. Dask can be run locally, across multiple machines, and with an arbitrary set of resources.
+```python
+prediction_single = m.predict_tile(path=path, patch_size=300, dataloader_strategy="single")
+```
+The `dataloader_strategy` parameter has three options:
 
-### Example Dask and DeepForest integration using SLURM
+* **single**: Loads the entire image into CPU memory and passes individual windows to GPU.
 
-Imagine we have a list of images we want to predict using `deepforest.main.predict_tile()`. DeepForest does not allow multi-GPU inference within each tile, as it is too much of a headache to make sure the threads return the correct overlapping window. Instead, we can parallelize across tiles, such that each GPU takes a tile and performs an action. The general structure is to create a Dask client across multiple GPUs, submit each DeepForest `predict_tile()` instance, and monitor the results. In this example, we are using a SLURMCluster, a common job scheduler for large clusters. There are many similar ways to create a Dask client object that will be specific to a particular organization. The following arguments are specific to the University of Florida cluster, but will be largely similar to other SLURM naming conventions. We use the extra Dask package, `dask-jobqueue`, which helps format the call.
+* **batch**: Loads the entire image into GPU memory and creates views of the image as batches. Requires the entire tile to fit into GPU memory. CPU parallelization is possible for loading images.
 
+* **window**: Loads only the desired window of the image from the raster dataset. Most memory efficient option, but cannot parallelize across windows due to Python's Global Interpreter Lock, workers must be set to 0. 
 
-```python
-from dask_jobqueue import SLURMCluster
-from dask.distributed import Client
-
-cluster = SLURMCluster(processes=1,
-                        cores=10,
-                        memory="40 GB",
-                        walltime='24:00:00',
-                        job_extra=extra_args,
-                        extra=['--resources gpu=1'],
-                        nanny=False,
-                        scheduler_options={"dashboard_address": ":8787"},
-                        local_directory="/orange/idtrees-collab/tmp/",
-                        death_timeout=100)
-print(cluster.job_script())
-cluster.scale(10)
-
-dask_client = Client(cluster)
-```
+## Data Loading
 
-This job script gets a single GPUs with "40GB" of memory with 10 cpus. We then ask for 10 instances of this setup.
-Now that we have a dask client, we can send our custom function.
+DeepForest uses PyTorch's DataLoader for efficient data loading. One important parameter for scaling is the number of CPU workers, which controls parallel data loading using multiple CPU processes. This can be set 
 
-```python
-import os
-from deepforest import main
-
-def function_to_parallelize(tile):
-    m = main.deepforest()
-    m.load_model("weecology/deepforest-tree") # sub in the custom logic to load your own models
-    boxes = m.predict_tile(raster_path=tile)
-    # save the predictions using the tile pathname
-    filename = "{}.csv".format(os.path.splitext(os.path.basename(tile))[0])
-    filename = os.path.join(<savedir>,filename)
-    boxes.to_csv(filename)
-
-    return filename
 ```
-
-```python
-tiles = [<list of tiles to predict>]
-futures = []
-for tile in tiles:
-    future = client.submit(function_to_parallelize, tile)
-    futures.append(future)
+m.config["workers"] = 10
 ```
+0 workers runs without multiprocessing, workers > 1 runs with multiprocessing. Increase this value slowly, as IO constraints can lead to deadlocks among workers.
 
-We can wait to see the futures as they complete! Dask also has a beautiful visualization tool using bokeh.
 
-```python
-for x in futures:
-    completed_filename = x.result()
-    print(completed_filename)
-```