My "only" is an exaggeration, but the AI labs are definitely the societal force I'm most worried about. If they were closed down or some talented person figures out how to control AIs (so they stay controlled even if the become more capable than us) I'd be fairly optimistic about humanity's future.
