Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[WIP] Sam integration #7722

Draft
wants to merge 28 commits into
base: gen-ai-dev
Choose a base branch
from
Draft

[WIP] Sam integration #7722

wants to merge 28 commits into from
+1,383 −42

Conversation

vgrau98
Copy link
Contributor

@vgrau98 vgrau98 commented Apr 28, 2024 •
edited
Loading

Fixes #6357 .

Description

Could be splited in smallest pr

  • Image encoder (2D light vision transformer ViTDet). Output consistency checked with official sam weights.
  • Prompt encoder
  • Mask decoder
  • Segment Anything Network
  • Weights mapping/integration
  • Model zoo implementation
  • Monai label apps integration

Types of changes

  • Non-breaking change (fix or new feature that would not break existing functionality).
  • Breaking change (fix or new feature that would cause existing functionality to change).
  • New tests added to cover the changes.
  • Integration tests passed locally by running ./runtests.sh -f -u --net --coverage.
  • Quick tests passed locally by running ./runtests.sh --quick --unittests --disttests.
  • In-line docstrings updated.
  • Documentation updated, tested make html command in the docs/ folder.

vgrau98 added 11 commits April 28, 2024 23:11
@vgrau98
causal self attention
7acfc69 
Signed-off-by: vgrau98 <[email protected]>
@vgrau98
causal selfattention tests
568013c 
Signed-off-by: vgrau98 <[email protected]>
@vgrau98
integrate flash attention usage
12dca86 
Signed-off-by: vgrau98 <[email protected]>
@vgrau98
transformer block local window attention
71d7c9d 
Signed-off-by: vgrau98 <[email protected]>
@vgrau98
fix: window partition input shapes
a0325e4 
Signed-off-by: vgrau98 <[email protected]>
@vgrau98
fix: error handling
02c087c 
Signed-off-by: vgrau98 <[email protected]>
@vgrau98
local window attention tests
ab4a440 
Signed-off-by: vgrau98 <[email protected]>
@vgrau98
feat: 3d local window attention
b47d71a 
Signed-off-by: vgrau98 <[email protected]>
@vgrau98
3d local attention window tests
d4fec56 
Signed-off-by: vgrau98 <[email protected]>
@vgrau98
clean
a4ee3f4 
Signed-off-by: vgrau98 <[email protected]>
@vgrau98
refacto
e417ffe 
Signed-off-by: vgrau98 <[email protected]>
@vgrau98 vgrau98 changed the title Sam integration [WIP] Sam integration Apr 28, 2024
@ericspod ericspod requested a review from marksgraham April 29, 2024 11:53
@marksgraham
Copy link
Contributor

Hi @vgrau98

Thanks for this - would you be OK to hold off until this PR is merged into the gen-ai branch, which will hopefully be soon?

@vgrau98
Copy link
Contributor Author

vgrau98 commented May 2, 2024 •
edited
Loading

Hi @vgrau98

Thanks for this - would you be OK to hold off until this PR is merged into the gen-ai branch, which will hopefully be soon?

Of course, I think some tasks can be initiated while waiting for the refacto on the transformer blocks as soon as the pr is merged

atbenmurray
atbenmurray reviewed May 3, 2024
View reviewed changes
tests/test_selfattention.py
@@ -24,25 +24,29 @@
from monai.utils import optional_import

einops, has_einops = optional_import("einops")
xops, has_xformers = optional_import("xformers.ops")

TEST_CASE_SABLOCK = []
for dropout_rate in np.linspace(0, 1, 4):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd like to suggest using itertools.product here rather than the nested for loops

vgrau98 and others added 10 commits May 4, 2024 16:05
@vgrau98
feat: layer norm 2d
e91c289 
Signed-off-by: vgrau98 <[email protected]>
@vgrau98
fix: rel pos embedding with local attention
3726f57 
Signed-off-by: vgrau98 <[email protected]>
@vgrau98
feature: 2D sam image encoder
e4778d1 
Signed-off-by: vgrau98 <[email protected]>
@vgrau98
Merge pull request #1 from vgrau98/sam-integration-img-encoder
bd5318e 
Sam integration img encoder
@vgrau98
sam prompt encoder
ce3f32c 
Signed-off-by: vgrau98 <[email protected]>
@vgrau98
Merge pull request #2 from vgrau98/sam-integration-prompt-encoder
47e0eb8 
sam prompt encoder
@vgrau98
refacto mlp block for multilayer suppoert
50752a0 
Signed-off-by: vgrau98 <[email protected]>
@vgrau98
mask decoder
0250aef 
Signed-off-by: vgrau98 <[email protected]>
@vgrau98
attention block as defined in sam
c6cd2c0 
Signed-off-by: vgrau98 <[email protected]>
@vgrau98
two way transformer block and two way attention block
0a6dadb 
Signed-off-by: vgrau98 <[email protected]>
@marksgraham
Copy link
Contributor

Hi @vgrau98
Thanks for this - would you be OK to hold off until this PR is merged into the gen-ai branch, which will hopefully be soon?

Of course, I think some tasks can be initiated while waiting for the refacto on the transformer blocks as soon as the pr is merged

hey @vgrau98 , thanks for waiting - it's now merged into gen-ai-dev so go ahead with this PR :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@atbenmurray atbenmurray atbenmurray left review comments

@marksgraham marksgraham Awaiting requested review from marksgraham

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@vgrau98 @marksgraham @atbenmurray