Summary

This article investigates how large pre-trained deep learning models (Visual Language Models) can be used to classify the behavior (eating, moving, resting) of animals (chamois, red deer, roe deer) in crops from camera trap images. Notably, they use zero-shot learning, i.e. do not use any labelled training data. This is an interesting area of research and is in line of previous research, such as done for WildCLIP, that seeks to exploit “foundation models” for camera trap image analysis.

They compare six pre-trained models and assess their performance on a single dataset. They report accuracy, calibration error, and the difference in diel activity patterns. They use labels collected from participatory science. The results are impressive, with high accuracy and little deviation in diel activity patterns.

The study addresses a relevant issue and presents an interesting extension regarding the analysis of camera trap images to behaviors using zero-shot learning. This approach is likely of interest to many practitioners in the field.

The study, however, is conducted in a narrow setting, limiting its value and generalizability. It only considers one dataset (even though others are publicly available) and three species (although others are available). Researchers who wish to apply this methodology would be required to carefully evaluate it on their dataset. Additionally, the study investigates few methodological variations even though their stated goal is to assess the models more generally (line 61f) and to explore important variations, such as prompt formulation (66ff). This leaves me with doubts about whether the results are robust under variations of hyper-parameters, datasets, and species. Furthermore, some details in the motivation, chosen methodology and the analysis of the results are vague or not convincing (see major / minor comments).

Major Comments

Data

• lines 48ff: The authors claim that a large amount of training data is needed to model behaviors. Which is one of their justification to use zero-shot learning. Is there evidence for that statement? I would assume that models might easily learn to distinguish between the behaviors that were modelled (using transfer-learning on the classification model – which is a pre-requisite). A comparison with a supervised-baseline would be interesting to add to the study, using increasing numbers of labels.

• lines 82ff: The process of how the images were classified on Zooniverse is not clear enough. Did participants classify images, sequences (one burst) or events (all images within a minute)? Sequences/events can contain multiple images and (presumably) multiple individuals. This would imply different behaviors could be present, even the same individual could show different behavior. The implication of this should be discussed more clearly, including some statistics on the frequency of such instances.

• lines 151ff: The authors consider images (events?) for which the most voted for behavior has at least 90% of the votes. This is roughly one third of the data (please provide exact numbers). One could assume, that this “high agreement” dataset consists of “easy” samples, since citizen scientists observe only one behavior, with high agreement. This might introduce a positive bias to accuracy and ECE scores. This should be discussed. I suggest considering using the same approach as used for diel activity patterns: a 50% threshold, or if more than 50% of the users observed a specific behavior.

Prompt Engineering

• line 279: The reason to include night/day time information to the prompt is not well justified in my view. Line 279-282 seems to state a hypothesis that is not verified. The original publication (Radford) refers to single-word captions which are rare, and therefore a prompt template representing a full sentence is used for zero-shot classification. This is something you are already doing. Without knowing the distribution of the captions myself, I consider it unlikely that specifying the time of day is part of a typical caption. I would however expect the species to be part of a caption. This is something the authors could test since the species are known in practice. Radford also describe additional techniques, such as prompt ensembling to reduce the effect of a badly picked prompt. This could also be interesting. The effect of prompt engineering seems not well evaluated and could be improved substantially.

Evaluation

Generally: The study would benefit from investigating more variations (confidence thresholds, prompt engineering, etc.), also by including train and test splits to robustly evaluate the results.

• lines 155ff: The macro accuracy is calculated, which is not well justified (while it considers the classes equally, it might be unfairly influenced by rare classes, leading to high variance in the metrics). Simply reporting Precision and Recall for the behaviors seems sufficient.

• line 158: The authors aggregate predictions on image (or crop?) level to event-level. Which is necessary. Aggregation is done by averaging the probabilities of all behaviors. The authors are modelling a multi-label classification problem. Could this aggregation strategy not be problematic if multiple behaviors are present? For example: If one individual is eating with very high confidence, and others are clearly not, then the score of eating would be low?

• Line 175 ff; Activity patterns are only considered for events for which one behavior has at least 50% of the votes. What happens if multiple behaviors are present?

• Line 175 ff: A confidence score of 0.75 is applied to “reduce false positives”. This is not further justified. Was this value experimentally evaluated? It could implicitly favor certain models (e.g. models with a uniform bias might have more false negatives). This could be analysed and visualized in more detail (maybe precision-recall curves for individual classes and models).

• Lines 212 (table 2): The authors report metrics with 0.01 significant digits and without confidence estimates. It would be more transparent and interesting to report some measure of uncertainty. One could calculate the metrics on bootstrap samples for example.

Discussion

The authors say their approach is typically applied to images for which the species is known (line 34-35). In practice that might often mean that images (i.e. crops from MegaDetector) need to be labelled. If that is the case, the additional effort to label behaviors (alongside species) might actually be small. This should be discussed and might limit the usefulness of the proposed approach. The studies cited (line 37) in my view do not proof that species classification is possible out-of-the-box (without training).

Line 252: The authors suggest using a zero-shot multi-label classification approach might be useful. It seems easily possible to do that with zero-shot models. It would be interesting to know why this was not tested (since it is in fact a multi-label classification problem).

The discussion is a bit difficult to follow at times. Examples include:

lines 286ff: The discussion about prompt order sensitivity, which could be addressed in the methodological section.

Line 267: The surprise about WildCLIP's performance even though it was trained on behavioral data .

Lines 299: The discussion about the ability to analyze multiple images, which seems possible with some open-source models (e.g. OpenFlamingo).

I suggest to split it the discussion section into discussion and conclusion.

Minor Comments

• line 50: Is behaviorally annotated camera trap data “very rare”? Please justify. See lila (see line 112, 113) with its many Zooniverse-based datasets.

• line 53: it is not clear to me that a model learns species specific behavioral patterns. Please elaborate why a model needs re-training for new species.

• line 92: it would be interesting to know whether this is an issue with participatory labelling or whether such behavior is simply rare. One could use the model to look for possible candidates and verify this? (also see line 124)

• line 124: The authors motivate using animal crops to avoid the model focusing on other information in the scene. This might, however, introduce a distribution shift (cropped images might be of lower quality than real images, and might not represent examples as they occur in typical VLM trianing data). Furthermore, behaviors that include multiple animals (e.g. interacting) may not be modelled accurately if only crops are considered . Please justify this in more detail.

• line 129: The caption (prompt) implies the model is presented a camera trap picture. It might have seen such in the pre-training dataset, however, you are using crops and not full camera trap images. It might be a detail but you might want to test more variations of prompts. Furthemore, this seems to be the prompt used in WildClip. Please cite accordingly.

• line 135: “prompt assistant” → who is that?

• line 158: how are predictions aggregated? Reading reference should not be needed

• Line 196: What are the implications of low ECE scores?

• line 248 ff: Sentence “In fact ...” is not clear to me

• line 255: It would be interesting to see such examples of wrongly classified behaviors.

• line 277: How long would it take on average CPU only pc?

• line 294: positional biases could be mitigated by ensembling predictions on permuted prompts. While this increases inference, it could improve predictions.

• line 310: reference wrong

• line 383: reference incomplete

=========================
Summary
=========================

The manuscript represents a resubmission of an earlier version that I have reviewed. The authors have commented on all issues that I have raised (see my replies below).

The study addresses an interesting area of research: How to exploit powerful large pre-trained visual language models (VLMs) in analysing images from camera traps. Previous work, such as the WildCLIP paper presented promising findings in this area.

In this work, the goal is to infer diel activity patterns by classifying cropped images (using a separate neural network) from camera traps according to different animal behaviors. They use VLMs for zero-shot learning. This is different to WildCLIP, which required fine-tuning a model using labelled data. They find and report high accuracy of several models (e.g. CogVLM), which could be relevant to practitioners.

My main concerns in the first version were related to 1) generalizability of the method due to the narrow setting used in the study (with three species and three behaviors and only one location) and 2) methodological questions surrounding the variation of hyper-parameters, such as prompt formulation, and the evaluation of the results.

The authors conducted new experiments on an established dataset (Snapshot Serengeti) which seems to confirm their results. However, they only include details in the SI and only use a limited set of three species. I think it would substantially strengthen the study's findings and the broad applicability of the proposed method if these results were prominently placed in the main text, include all species of the Snapshot Serengeti dataset, and (ideally) even 1-2 more datasets. Additionally, prompt variation still seems limited. Furthermore, the text could be improved substantially for clarity and brevity.

=========================
Major Comments
=========================

Generalizability

• Using additional datasets to evaluate the method, such as the Snapshot Serengeti dataset, would strengthen the study's conclusions. These results should be prominently placed in the main article. Ideally 1-2 additional datasets, each with multiple species, should be included. In the current version, the authors only briefly mention results on Snapshot Serengeti for three species and put most of the results in the SI. There is also not enough justification for only using three species from the CREA dataset, as well as only three from the Snapshot dataset.

Hyper-Parameters

• The authors mainly use WildCLIPs prompt, with some variation regarding day/night time (which is still poorly justified, see previous comments) and species name. WildCLIP adapted pre-trained models using fine-tuning to improve matching text captions with images, and thus the exact caption might be less important. In this work, the authors rely on zero-shot learning which might increase the importance of picking good captions. I expect more emphasis on this aspect, and more experiments. Experiments using variations of the class names could also be conducted. Such as using a prompt like “… lying down” or “sleeping” as opposed to just using “resting”.

Evaluation

• line 197 ff: Instead of using “macro accuracy” I suggest naming it macro averaged recall. Macro accuracy does not really exist. Additionally, I suggest reporting macro averaged precision and, to have one global metric, macro averaged F1-score.

• You are using a fixed threshold of 0.75 for all models to consider a classification when evaluating for diel activity patterns. This means that even if a model assigns the highest probability to the correct behavior, the prediction is not considered. You added an experiment to the SI which confirms, that this hurts the CLIP base model substantially. This also might imply that other experiments that you conducted, such as comparing different prompts, might be affected, as they might influence the probability distributions. Instead of using a fixed threshold, you could use: i) a training set to optimize for this and then report the results of a test set, ii) optimize on the full set , or iii) simply assign the behavior with the highest probability.

• As previously mentioned, providing boostrap uncertainty / confidence intervals for the results could help assess how significant the differences are. The authors mention they used bootstrap samples when estimating the overlap index. I suggest using such samples for all metrics.

=========================
Minor Comments
=========================

• Figure 1: A table would be more informative. Additionally, the animal species are not appealingly formatted (“reddeer” vs “Red Deer” | “red deer”).

• Figure 2: Maybe include examples of each species. Caption “only the first is..” what if there are multiple individuals? Maybe just remove that sentence and include only events with one individual.

• line 211: reference error

• line 283: doesn't WildCLIP enable the same?

• The wording could be improved and the text be more compact throughout. It is often overly verbose. Examples:
  line 175ff sentence “Then, ...”
  line 187 ff regarding imbalance

• The “Results” section could be more compact as it includes a lot of discussion of the results, despite a separate “Results” section. While the “Discussion” section introduced new results. I suggest to either strip the “Results” sections from discussions or combine “Results & Discussion”.

=========================
Replies to previous Comments
=========================

• Comment 1: You claimed that modelling behaviors needs a large amount of labelled data. My suggestion to train a supervised model (with varying number of labels) would evaluate this claim. I understand your motivation and your practical concerns. They are valid. They, however, limit the use case to researchers who solely rely on pre-trained models (also for detection and classification).

• Comment 3: Thanks for your reply. It makes sense that the models perform better on the high-agreement dataset, including the supervised baseline. The question is how this selection influences the relevant metrics: in this study the diel patterns.

• Comment 4: The additional experiments are valuable. It seems convincing that prompt variations with species names may not help much.

• Comment 5: My suggestion regarding using a train/test split was in reference to hyper-parameter tuning, such as prompt engineering and confidence threshold selection. In this case, train / test splits enable better estimation of generalization performance. Using the Snapshot Serengeti dataset is important since it is truly an independent test set.

• Comment 6: Referring to “macro accuracy” myself, I was not precise enough. Macro accuracy is not a metric that I know. Class-specific accuracy is not defined (often it is Recall). Including Recall/Precision is better. However, I would (as a global metric) include the following: Accuracy and macro F1. Currently you are using macro -Recall I suppose?

• Comment 11: Thanks for this reply.

• Comment 12: You mention that on crop-level the behaviours are mutually exclusive. This enables multi-label classification on event-level by aggregating multi-class classification on crop-level out-of-the-box?