The down side is that you don't get paid. And I think that's pretty consistent -- no one is going to pay market rates for you to spend 80% of your time on "whatever you like".