I wonder whether LoRAs could be useful for U-Net training. Especially thinking of CNN-based U-Net models with pre-trained encoders (but randomly initialized decoders). At least, it seems possible that normal weight updates on the decoder and LoRA training on the encoder could improve efficiency.