(Disclaimer, I'm working for Google)
In my opinion this is a great question. I believe you can go two ways:
1) Get your hands on few physical computers and "old" gpus (like nVidia 1000 series or something like that). Put them as part of a K8S cluster and you have an amazing setup to play around with. Try to squeeze as much flops from your hardware as you can. Bonus points if you also architect around failures and test that by pulling off network cables.
2) Using some Cloud provider use preemptible/spot instances with a bunch of GPUs for few hours at the time. Not sure with other clouds, but with GCP you can create a GKE nodepool that only uses spot instances and in conjuction with cluster autoscaler makes what I described very easy and you don't really have to clean much after you're done messing around for the day. GPUs like K80, T4, or P4s are relatively cheap and, if you use them for just 10s of hours a month, you can get away with a bill of 10s of dollars [1].
Either option works fine (IMHO).
Another option I am unsure about because I never tried it is to use something like [2] to multiplex your GPU(s) so that you can pretend you have more GPUs to mess around with. However, if your goal is to learn how to manage/write software for multi-gpus/multi-machine clusters this is somewhat limiting because 1) it doesn't teach you much about data locality/compact placement since transfers between virtual GPUs will be extremely fast (i.e. they are sharing the same VRAM pool after all) and 2) you will still have a single machine (even if you are using multiple virtual K8s nodes).
1) Get your hands on few physical computers and "old" gpus (like nVidia 1000 series or something like that). Put them as part of a K8S cluster and you have an amazing setup to play around with. Try to squeeze as much flops from your hardware as you can. Bonus points if you also architect around failures and test that by pulling off network cables.
2) Using some Cloud provider use preemptible/spot instances with a bunch of GPUs for few hours at the time. Not sure with other clouds, but with GCP you can create a GKE nodepool that only uses spot instances and in conjuction with cluster autoscaler makes what I described very easy and you don't really have to clean much after you're done messing around for the day. GPUs like K80, T4, or P4s are relatively cheap and, if you use them for just 10s of hours a month, you can get away with a bill of 10s of dollars [1].
Either option works fine (IMHO).
Another option I am unsure about because I never tried it is to use something like [2] to multiplex your GPU(s) so that you can pretend you have more GPUs to mess around with. However, if your goal is to learn how to manage/write software for multi-gpus/multi-machine clusters this is somewhat limiting because 1) it doesn't teach you much about data locality/compact placement since transfers between virtual GPUs will be extremely fast (i.e. they are sharing the same VRAM pool after all) and 2) you will still have a single machine (even if you are using multiple virtual K8s nodes).
[1] 3 instances with 4 GPUs each used for 24h in a month cost you 46$ according to: https://cloud.google.com/products/calculator/?hl=en&dl=CiRhY...
[2] https://github.com/NVIDIA/k8s-device-plugin