Scalability on coarse resolution imagery

[AlysonEast]

I have been working with the DeepForest model to try and fine-tune it to recover tree crowns from the coarser resolution NAIP or MAXAR imagery in dense closed canopy forests. As far as I know, the application of DeepForest to NAIP imagery has been limited to sparse/open canopy urban environments. Is there any reason to think that taking advantage of additional spectral bands could result in better results in coarse resolution applications? How much additional training is necessary to incorporate more bands than were originally in the model? And is there any sample code you could point me to for such an application?

[ethanwhite]

Hi @AlysonEast - sorry for losing track of this during the end of semester rush.

In concept we do think that additional spectral bands should be useful and some of the CS students associated with our project have been trying to leverage this within the NEON data itself. The idea (which I suspect is motivating your question) is that crowns of different species that occur close to one another should be easier to distinguish using additional bands (which is why we use hyperspectral data for species classification). If we can tell that two blobs of pixels deviate significantly in multi-spectral space that should make crown detection easier for the algorithms. This could be particularly important at coarser resolutions where it's difficult to distinguish the edges of individual crowns.

That said, we've found that simple approaches of just adding in additional bands and training hasn't generally been helpful. This is (we think) because doing so loses everything that can be learned from pretraining on large amounts of RGB imagery. So, the CS folks we've been working with have been working on more complex infrastructures to try to leverage that information more effectively.

TLDR; I think you've got a good idea, but I don't have any easy recommendations for how to implement it because the simple stuff hasn't worked for us. I've pinged the CS folks working on this with a link to this issue in the hopes that they might have more useful directions for you.

[meilunzhou]

Hi @AlysonEast

Is there any reason to think that taking advantage of additional spectral bands could result in better results in coarse resolution applications?

Building off what Dr. White said above and based off my own experience working with the data, the additional spectral bands should be useful. My recent work has been trying to combine LiDAR, RGB, and HSI data into an interpretable latent space that can help inform an object detection network. I have seen evidence that combining the information present in different modalities can help with both classification and detection. However, the caveat here is that the ground truth itself is not the most precisely annotated and can cause some issues

Some other issues I've noticed:

• some tree stems are very close to each other and the lower resolution NEON data (1m products like HSI and LiDAR) will have overlapping information for differently labeled objects
• there is not a clear delineation for where one object ends and another begins
• the data products are 2D representations of a 3D object. some trees may be below other trees
• i expected the additional information to nearly always be useful for a downstream task but this is not the case. sometimes adding additional information that isn't useful will actually serve as noise and cause performance to drop

How much additional training is necessary to incorporate more bands than were originally in the model? And is there any sample code you could point me to for such an application?

One of my goals is to incorporate my multi-modal representation into the DeepForest architecture. However, I'm still building the representation model at the moment and have not gotten to retraining DeepForest yet so I cannot advise much on this.