If I'm understanding the problem correctly, this should be solved by pnpm [1]. It stores packages in a global cache, and hardlinks to the local node_packages. So running install in a new worktree should be instant.

[1]: https://pnpm.io/motivation