Improving Rare Tree Species Classification using Domain Knowledge

by I. Harmon

Introduction

Forests are an integral part of life on earth. They are important for carbon sequestration, fuel, building materials, and animal habitats, etc. Monitoring and managing ecosystem health requires forest management. Forest management includes doing field surveys in addition to remote sensing. Field surveys are done manually over limited areas and are very labor intensive. On the other hand, remote sensing can cover large areas of forest and monitor millions of trees in a single survey.

Using data collected from surveys, ecologists can estimate forest parameters such as biomass, species abundance, and biodiversity. Some parameters, such as biomass, can be better estimated when the species of the tree is known. To efficiently determine species from remotely sensed data requires an automated means of species classification.

Common modalities for remotely sensed data are hyperspectral imagery (HSI), RGB imagery, LiDAR and synthetic aperture radar (SAR). Of these, RGB is by far the most common. In order to efficiently classify species in hundreds of square kilometers of remotely sensed data, deep learning models are commonly deployed. Applying deep learning models to remotely sensed forest data is not without challenges.

Problem

1. Deep models are not explainable
2. Deep models don't classify rare species well

Deep models are black boxes. The reasoning behind their inferences is not easily understood by the user. Furthermore, deep models require large amounts of data to train, therefore they perform poorly when classifying trees that are not common in a dataset.

Neuro-symbolic models are one solution to these problems. Neuro-symbolics combines neural models with symbolic reasoning to

• Improve explainability
• Allow incorporation of domain knowledge
• Create a more robust model

In our paper, "Improving Rare Tree Species Classification using Domain Knowledge" we show that neuro-symbolics is a means of improving rare species classification using a simple neuro-symbolic framework called DeepCtrl (Seo et al., 2021).

DeepCTRL

DeepCTRL is a framework created by Google that allows domain knowledge to be incorporated into any neural model using a regularization technique where the domain knowledge is formulated as a rule. This rule is written as a mathematical statement that can be evaluated during training and added as a term in the loss function. The architecture consists of a task encoder, a rule encoder and a decision block. Any neural model can act as the rule encoder or the task encoder. The output from each encoder is scaled by alpha, where alpha is a constant chosen during training. The output of each encoder is scaled by alpha and 1 - alpha before being concatenated and fed into the decision block.

Model

We use the fully convolutional CNN model from Fricker et al. as task and rule encoders. We supplement remotely sensed RGB data with crown height model (CHM) and digital elevation model (DEM) data.

Data

We use the dataset collected by the National Ecological Observatory Network (NEON) and curated by Fricker et al. The data is collected from a mixed conifer forest called the Tea Kettle Experimental Forest (TEAK). TEAK is located in the Sierra National Forest in Fresno California.

NEON data includes RGB, HSI, LiDAR, and CHM data. We supplement their data with the digital elevation model from the US Geological Survey (2013 - 2019).

The data is broken into six classes of tree species: White fir, Red fir, Incense cedar, Jeffrey pine, Sugar pine, Black oak, and Lodgepole pine. A seventh class called Dead is added. Dead represents standing dead trees of any species.

Analysis of the data shows that the Black Oak species is the rarest in the dataset. Also, species can be distinguished by their height crown distribution and preference for spatial distribution as a function of terrain elevation. The black oak species, the rarest in the dataset, is shorter and tends to have shorter crowns.

Rules

We create 4 rules. Rules 1 and 2 are based on the CHM. Rule 3 is based on the DEM and rule 4 is based on the green leaf index (GLI). In natural language the rules are:

1. Trees taller than 46 m are unlikely to be black oak.
2. Trees taller than 53.2 m are unlikely to be lodgepole pine.
3. Trees growing at less than 2072 m elevation are unlikely to be red fir.
4. Trees trees with a GLI less than 0.1 are unlikely to be incense cedar.

Each rule is converted into a mathematical equation where the sigmoid function is used to map an internal function that quantizes the tree's compliance with the rule to a range between 0 and 1. A summary of each rule and its formula is given in the table below.

Rule	Natural Language Description	Equation
1	Trees taller than 46 m are unlikely to be black oak.
2	Trees taller than 53.2 m are unlikely to be lodgepole pine.
3	Trees growing at less than 2072 m elevation are unlikely to be red fir.
4	Trees trees with a GLI less than 0.1 are unlikely to be incense cedar.

Results and Conclusion

The addition of rules significantly improved classifier performance particularly for the rarest species. The change in F1 for each rule is shown in the graph below. The addition of rules also increases the explainability of the model as the framework forces the model to adhere to the rule.

Works Cited

[1] Fricker, Geoffrey A., et al. "A convolutional neural network classifier identifies tree species in mixed-conifer forest from hyperspectral imagery." Remote Sensing 11.19 (2019): 2326.

[2] Harmon, I., et al. "Improving Rare Tree Species Classification using Domain Knowledge." IEEE Geoscience and Remote Sensing Letters (2023).

[3] Seo, Sungyong, et al. "Controlling neural networks with rule representations." Advances in neural information processing systems 34 (2021): 11196-11207.