NNs are not a black box in the sense that you cannot see what is going on in them, but in the sense that we currently just very badly understand. You have a function approximator that you have to translate into some other representation. As long as you know how basic building blocks map or can be approximated, it might be possible.
I don't know anything about decompiling or ML, but doesn't the black box nature of NN rules this out?