Oh man ! I wish someone had told me this before. Back when i did research for my BSc thesis, I had an 24 core machine and wrote a script to create 24 bash scripts to split up my task which i would than run in parallel.
Sometimes I have the felling that I am wasting time when I check HN too often, but stuff like this makes me think otherwise.
Like, async spidering with parallel is a one-liner, but comparatively, have you seen how much absurdly syntaxed pycurl boilerplate you need? It's crazy. I for one will take the simpler method any day.
Unlike xargs? That's just plain wrong, see xargs' -P flag.