Skip to content

Add support for Comet experiment data flush after saving checkpoint into CometLogger #20681

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
yaricom opened this issue Mar 27, 2025 · 2 comments · May be fixed by #20680
Open

Add support for Comet experiment data flush after saving checkpoint into CometLogger #20681

yaricom opened this issue Mar 27, 2025 · 2 comments · May be fixed by #20680
Labels
feature Is an improvement or enhancement needs triage Waiting to be triaged by maintainers won't fix This will not be worked on

Comments

@yaricom
Copy link

yaricom commented Mar 27, 2025 •
edited
Loading

Description & Motivation

Periodic flushing of Comet experiment data by CometLogger is sometimes necessary to ensure all metrics and data remain up to date in the Comet UI and aligned with saved model checkpoints. Some users accomplish this by creating custom Callbacks, though this approach can be rather cumbersome.

Pitch

It is proposed to enhance the CometLogger implementation to support flushing collected data to Comet after each checkpoint is saved. This would be configurable via the flush_every parameter of the CometLogger.

This improvement allows users to manage Comet experiment flushing using the standard PyTorch Lightning approach by registering appropriate ModelCheckpoint callbacks with the Trainer.

Alternatives

No response

Additional context

No response

cc @lantiga @Borda
@yaricom yaricom added feature Is an improvement or enhancement needs triage Waiting to be triaged by maintainers labels Mar 27, 2025
@yaricom yaricom linked a pull request Mar 27, 2025 that will close this issue
Open
7 tasks
@yaricom
Copy link
Author

yaricom commented Mar 27, 2025

The PR with implementation: #20680

@stale Stale
Copy link

stale bot commented Apr 28, 2025

This issue has been automatically marked as stale because it hasn't had any recent activity. This issue will be closed in 7 days if no further activity occurs. Thank you for your contributions - the Lightning Team!

@stale stale bot added the won't fix This will not be worked on label Apr 28, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
feature Is an improvement or enhancement needs triage Waiting to be triaged by maintainers won't fix This will not be worked on
Projects
None yet
Milestone
No milestone
1 participant
@yaricom