Not really, but note that adding your user to the docker group is precisely identical to disabling sudo's password authentication for your user (adding users to the docker group gives a free privilege escalation from that user to root).
Now, that said, the Docker client does quite a few things (such as unpacking archives) that you might not want to be done as root. Especially if the client is running on a different machine.
is there any easy way to give each user their own docker runtime? I always thought it would be very useful, this way you can give all of your users the ability to easily run all the software they want.
Yes, but it'd currently require giving root access to your users. Rootless containers[1] is a project I started a while ago, and now (with some patches) you can run Docker (and Kubernetes) as an unprivileged user. There are some caveats, but I'd recommend checking it out.
Now, that said, the Docker client does quite a few things (such as unpacking archives) that you might not want to be done as root. Especially if the client is running on a different machine.