Describe the bug
Flex runs the inference it in around 6 seconds. With cubecl cpu backend I left it running for 3-5 minutes and it didn't finish.

To Reproduce

1. Clone https://github.com/SzczurekYT/burn-wav2vec2-multipa-repro
2. Follow readme to setup the model
3. Run it with the flex backend - takes < 10 seconds
4. Swap the backend to cubecl cpu in code
5. Run again - doesn't finish after multiple minutes

Desktop (please complete the following information):

• OS: Nobara (Fedora fork) Linux
• Burn version - v0.21.0