In my experience, WAL-E is very brittle and fails with weird errors at times such as when your database restarts, or when it fails to upload a segment.
I have gone back to using pg_dump even though its not a real-time snapshot. In the end, replication + pg_dump gets the job done