This is the same for me. I've also learned how to recognize an album that I WILL like after more listening, which feels really weird, but it works.

Also, very catchy albums that immediately grab me rarely prove to have long-term listenability for me. An example would be the Postal Service album, I remember thinking that was really catchy and great, but after a few listens I hated it.