You can't compare context to memory. Context is simply all the text the LLM can use to generate a likely continuation. Imagine you're a relationship expert and I'm asking you for relationship advice. You don't know me so the best you can give me is "be yourself!" or "be confident!". It doesn't matter how good you are---lack of information about me is your limit. Now imagine you have a complete view of my dating history, including in-depth reviews from ex-girlfriends and whatnot. You could come up with some sharp and very fine-tuned advice just for me. Or maybe it still would be "be yourself!" cause dating advice is pseudoscience but you get my point.