Skip to content

Conversation

@jveitchmichaelis
Copy link
Collaborator

@jveitchmichaelis jveitchmichaelis commented Oct 23, 2025

@bw4sz no direct training integration yet, but this is what I'm thinking for the modeling.

The implementation looks verbose, but that's because a lot of the pieces are copied from transformers with very minor changes - like a reference to bbox changed to point. To modify the arch we need:

DeformableDetrKeypointConfig: Same as object detection but explicit parameter for point loss cost.
DeformableDetrKeypointDetectionOutput: Mostly similar but name changes.
DeformableDetrForKeypointDetection: Model, changes number of decoder outputs to 2 and removes some bbox specific bits.
DeformableDetrKeypointMatcher: Hungarian matcher that uses L1 or L2 cost instead of IoU.
DeformableDetrKeypointLoss: L1 or L2.
DeformableDetrKeypointImageProcessor: Normalizes keypoints to relative pixel coords and some other bits.

I don't anticipate checking in the current test suite as the overfit test takes a minute or two to run, but it does converge.

@jveitchmichaelis jveitchmichaelis force-pushed the keypoint branch 4 times, most recently from 37e04fb to 5d3e5f5 Compare October 23, 2025 22:38
@jveitchmichaelis
Copy link
Collaborator Author

jveitchmichaelis commented Oct 23, 2025

High level:

  1. Dataset support, initially derived from existing box datasets where we can derive keypoints from box centers.
  2. Configuration task support which is tied to architecture, e.g. task = boxes by default (allows retinanet, detr), but can be task = points (detr), polygons (?)
  3. Modeling support (create/load models should open the appropriate architecture)
  4. Transforms + augmentation should be minimal for now, but ensure that they support keypoints if passed in
  5. Training support - ensure that forward pass works with .fit() and all functions in main work
  6. Metrics should be modified to be task-specific
  7. Evaluate should also allow keypoint precision/recall based on matching metrics, with a threshold for pixel distance instead of IoU

@codecov
Copy link

codecov bot commented Oct 23, 2025

Codecov Report

❌ Patch coverage is 78.57143% with 138 lines in your changes missing coverage. Please review.
✅ Project coverage is 85.52%. Comparing base (915a945) to head (4ed7f25).
⚠️ Report is 23 commits behind head on main.

Files with missing lines Patch % Lines
src/deepforest/models/keypoint.py 79.48% 64 Missing ⚠️
src/deepforest/datasets/training.py 71.42% 22 Missing ⚠️
src/deepforest/main.py 72.60% 20 Missing ⚠️
src/deepforest/evaluate.py 77.61% 15 Missing ⚠️
src/deepforest/predict.py 73.33% 8 Missing ⚠️
src/deepforest/utilities.py 80.95% 4 Missing ⚠️
src/deepforest/augmentations.py 57.14% 3 Missing ⚠️
src/deepforest/callbacks.py 75.00% 1 Missing ⚠️
src/deepforest/visualize.py 75.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1182      +/-   ##
==========================================
- Coverage   87.38%   85.52%   -1.86%     
==========================================
  Files          20       22       +2     
  Lines        2569     3165     +596     
==========================================
+ Hits         2245     2707     +462     
- Misses        324      458     +134     
Flag Coverage Δ
unittests 85.52% <78.57%> (-1.86%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@jveitchmichaelis
Copy link
Collaborator Author

jveitchmichaelis commented Oct 31, 2025

@bw4sz integration is done, I'll add tests. Lots of TODOs dotted around which relate to tidying up and merging some duplicated code, but I anticipate polygons/segmentation will follow in a similar way.

The pre-commit failure is weird. It thinks that pandas is undefined, but the import is unchanged and ruff passes locally.

@jveitchmichaelis
Copy link
Collaborator Author

Currently training a DETR backbone on the lidar pretrain dataset, will swap that in once done.


def load_image(self, idx):
img_name = os.path.join(self.root_dir, self.image_names[idx])
img_name = os.path.join(self.root_dir, os.path.basename(self.image_names[idx]))
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will remove this, or make it an option.

@jveitchmichaelis jveitchmichaelis force-pushed the keypoint branch 3 times, most recently from 8eb5caf to 53820cd Compare November 7, 2025 22:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant