Qualcomm AI Engine Direct - Fix static cache #16423
Conversation
- Fix t5 and whisper - Fix online prepare for mutable buffer - Fix minor issues
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/16423
Note: Links to docs will display an error until the docs builds have been completed. ❗ 1 Active SEVsThere are 1 currently active SEVs. If your PR is affected, please view them below: ✅ You can merge normally! (1 Unrelated Failure)As of commit ba138a5 with merge base c730feb ( UNSTABLE - The following job is marked as unstable, possibly due to flakiness on trunk:
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
meta-cla
bot
added
the
CLA Signed
This PR needs a
|
chenweng-quic
added
the
module: qnn
mergennachin
left a comment
mergennachin
left a comment
There was a problem hiding this comment.
See inline comments
| ) | ||
| multiples = [] | ||
| for i in range(len(reshape_tensor.shape)): | ||
| multiples.append(output_tensor.shape[i] // reshape_tensor.shape[i]) |
There was a problem hiding this comment.
assert output_tensor.shape[i] % reshape_tensor.shape[i] == 0, f"Shape mismatch at dim {i}: {output_tensor.shape[i]} not divisible by {reshape_tensor.shape[i]}"
| for idx in range(len(self.static_cache.layers)): | ||
| self.register_buffer(f"key_cache_{idx}", self.static_cache.layers[idx].keys) | ||
| for idx in range(len(self.static_cache.layers)): | ||
| self.register_buffer( | ||
| f"value_cache_{idx}", self.static_cache.layers[idx].values | ||
| ) |
There was a problem hiding this comment.
nitpick: can do in one loop
| ) -> QuantizationConfig: | ||
| # the smallest scale: 0.0001 / 255 | ||
| extra_args: Dict[str, Any] = {"eps": 2**-21} | ||
| # the smallest scale defaults to 0.0001 / 255 |
There was a problem hiding this comment.
# At top of file
DEFAULT_EPS_8BIT = 0.0001 / 255
DEFAULT_EPS_16BIT = 0.0001 / 65535
# In functions
extra_args: Dict[str, Any] = {"eps": eps if eps else DEFAULT_EPS_8BIT}
You are repeating this pattern over and over again. 2 times for 8it and 6 times for 16bit
2 participants
Summary
Test plan
cc @cccclai @cbilgin