Training and Validation Curves

[macdatar]

Dear Ben/All,

Can someone post a code snipped to visualize Training and Validation curves during training? I would like to visualize Training Error vs Epochs and Validation Error vs Epochs during training.
I did read something about pytorch lightening logs but I was unable to make it work because I dont have much background in pytorch.
Any help is much appreciated.
regards,
Makarand

[macdatar]

To elaborate more, here are some more comments and examples

• Since I am unable to plot the performance metrics during training, my sense is that the model is likely overfit. I would like to figure that out by plotting training and validation curves. At present, when I give validation set during the trainng, I am unable to interpret the number that shows and the training also seems to not follow the specified validation interval.
• At prediction time, score_threshold value from model config seems to not take effect even after chaning it. I changed it a couple of ways but the output df still shows rows with score threshould lower than the specified value
• Below are some examples where the detection does ok but it misses some trees that I expect it to find; I would like to achieve a precision and recall of 0.9 (there are images that do well compared to the ones shown here but showing that dont)
• I would also like to know how to make the model hand multi-resolution images. Currently, with our resolution being finer than the data that was used to train deepforest, we are using a bigger tile-size in split raster but for unknown resolutions, we would like the model to work on any resolutions.
  Thank you for your help in advance. Much appreciated.

[bw4sz]

Let's start by

1. posting the entire code block so i can see what you are doing and then we can start to make some visualizations.
2. Add some detail on the number of images you have, the number of annotations, the config file, so we can get a sense of the size of the data.
3. Please show an example annotation image so we can see how the images have been annotated.
4. Have you held out data for model.evaluate, what is the current score? Show this code as well.

[macdatar]

Hi Ben,

I will do my best to provide all the information you have asked for. You may not be able to run it though because the orthomosaics we have are pretty large in size and hard to attach.

I have a total of 115 files with 60000 annotations. Each image covers 250-500 meter of land area in both length and width at 2.5 cm resolution. So these rasters are pretty large with one of them being 3 gb in size. I have attached a csv file with only first few lines shown as an example (my full csv files has 60000 lines). Also attached is an xml (extension changed to allow upload).

The evaluation value during training has been a small number like 0.1 or so for me but when I evaluate the train model and get the recall and precision using following code snippet, I get larger values like 0.8 at iou of 0.5 and around 0.88 for iou of 0.3. But it doesnt do too well on images with small trees. Overall, small trees is a smaller sample of the training data and some of them are really small. The model also struggles to distinguish trees with canopies touching each other. Most of our other predictions good but we would like to push the precision and recall over 0.9 and at least try to visualize training phase for underfitting, overfitting characteristics. The results seem ok but yet not robust.

annotation_df = utilities.xml_to_annotations(xml_path)
annotation_df["label"] = "Tree"
result = evaluate.evaluate_image(predictions=prediction_df, ground_df=annotation_df, root_dir='path to orthomosaics', savedir=None)
result.head()

result["match"] = result.IoU > 0.4
true_positive = sum(result["match"])
recall = true_positive / result.shape[0]
precision = true_positive / prediction_df.shape[0]

Also my biggest trouble is not able to plot train and validation curves. I have not been able to plot it from pytorch trainer with tensorboard.

csv_annotations_example.csv
pascal_vcc_annotation_xml.txt

Here are few samples of annotations

Here are some sample predictions on images completely hidden from training

Now the Code is described below.

Reading in annotations
The following code is inside a for that loops over all orthomosaics (i havent put all other supporting code).

• Reads in the annotations for the orthomosaic image in PASCAL VOC *.xml file format as a dataframe
• Writes it to a csv file,
• read the csv file inside split_raster function along with the raster file which outputs a annotation dataframe for a single orthomosaic as well as saves the split rasters.
• This is appended on to a combined annotation dataframe.
• At the end of the for loop, there is one single dataframe with varying image names. This can then be written as a csv.
  We use approximately 4 times larger tile because our resolution is 2.5 cm which is 4 times more compared to 10 cm of deepforest release model. This process is done for train and validation both. Orthomosaics are separated into train and test sets so both those are processed in the following way and saved with separately as csvs.

single_annotation = utilities.xml_to_annotations(xml_path)
csv_file_path = "filepath to be used to save the csv file"
single_annotation.to_csv(csv_file_path, index=False)
split_raster_save_dir = "directory to save all the split rasters"
single_split_annotation = preprocess.split_raster(path_to_raster = "path to one of the orthomosaic rasters",
                                           annotations_file = csv_file_path,
                                           base_dir = split_raster_save_dir,
                                           patch_size = 1500,
                                           patch_overlap = 0.15)
combined_annotations_df = pd.concat([combined_annotations_df, single_split_annotation])
combined_annotations_df.to_csv("path to a file where all annotations (train or val) should be saved as a csv", index=False)

Training
This section is line by line self explanatory

split_raster_save_dir = "path for a directory where all the split rasters are saved"
filename_save_model = "file path where the trained model should be saved"
train_annotations_path = "path for the csv file where all the training annotations are saved"
val_annotations_path = "path for the csv file where all the validation annotations are saved"

model = main.deepforest()
model.use_release()

model.config["train"]["csv_file"] = get_data(train_annotations_path)
model.config["train"]["root_dir"] = split_raster_save_dir
model.config["train"]["fast_dev_run"] = False

model.config["validation"]["csv_file"] = get_data(val_annotations_path)
model.config["validation"]["root_dir"] = split_raster_save_dir

model.config["validation"]["val_accuracy_interval"] = 2
model.config["validation"]['iou_threshold'] = 0.3

model.config["workers"] = 2
model.config["batch_size"] = 4
model.config["nms_thresh"] = 0.05

model.config["train"]['epochs'] = 30
model.config["train"]['lr'] = 0.001

model.create_trainer()
model.trainer.fit(model)

model.save_model(filename_save_model)

Making Predictions
Pretty self explanatory. A small function is used in the end and is described in appendix for ease of reading.

'''Choose a transferlearning-trained model or the release model: "ttrained" or  #"default"'''
test_type = "ttrained" 

if test_type == "default":
  model = main.deepforest()
  model.use_release()
elif test_type == "ttrained":
  model = main.deepforest.load_from_checkpoint("path to where a trained model file is saved")

'''here, I am not sure which takes effect. None of them seem to take effect for me'''
model.config["score_thresh"] = 0.2
model.config["score"] = 0.2
model.config['retinanet']['score_thresh'] = 0.2

model.config['nms_thresh'] = 0.5

prediction_df, prediction_image = PredictTrees(model, image_path_for_tree_detection, 1500, 1)

Appendix

def PredictTrees(model, image, patch_size, workers):
    '''
    Arguments:  deepforest tree detection model, image path, patch size, number of workers
    Returns:  predicted_df: pandas dataframe, predicted image: an image object ndarray with height by width by 3 channels
    '''

    model.config["workers"] = workers
    predicted_df = model.predict_tile(image, return_plot = False, patch_size=patch_size, patch_overlap=0.25, thickness=20)
    predicted_image = model.predict_tile(image, return_plot = True, patch_size=patch_size, patch_overlap=0.25, thickness=20)

    return predicted_df, predicted_image

[bw4sz]

Pytorch lightning has a logger that you can pass to the trainer. Checkout

https://lightning.ai/docs/pytorch/stable/extensions/logging.html

from deepforest import main
m = main.deepforest()
logger = <any support pytorch lightning logger>
m.create_trainer(logger=logger)

from there you can get the visualizations you are looking for, through tensorboard, comet, mlflow, wandb and all the others. DeepForest logs train and val loss each epoch.

[bw4sz]

Adding to docs.