The LLM is more like a Ouija board than a reliable tool.
>I can ask it to write user stories or prototypes
By the time I write enough to explain thoroughly to an LLM what to write in "user stories" or "prototypes", I could have just written it myself, without the middleman(bot), and without the LLM hallucinating.
If half the time I spend with an LLM is telling it what to do, and then another half is correcting what it did, then I'm not really saving any time at all by using it.
For me, it is a different experience. Yes, it takes time to provide the LLM with the necessary context, but overall it can save me 3-4 times the time I would have invested. Maybe I am just slower at software development myself ;-)
However, the work you do is indeed more that of a product owner than that of a developer. To avoid hallucinations, providing extensive documentation and allowing the LLM to perform test-driven development can be a game-changer.
When you do that, generation time is highly correlated (negatively) with code quality. So when the AI solves the tasks quickly and easily, you have a good chance of it generating good code. As soon as the AI has to try and try again to build something working, you should be very skeptical of the result.
Over the past months and years, I have used this method, and because it is somewhat reproducible, you can see the progress that LLMs are making. Sessions where the model does stupid things are becoming fewer, and sessions where the model finds a good solution become more frequent.
Writing the prompt takes time, but in many cases it's like rubber-duck debugging - it helps you to figure out the actual requirements, flow, or even incorrect/missing assumptions. Is it worth the time? Subjective.
The LLM is more like a Ouija board than a reliable tool.
>I can ask it to write user stories or prototypes
By the time I write enough to explain thoroughly to an LLM what to write in "user stories" or "prototypes", I could have just written it myself, without the middleman(bot), and without the LLM hallucinating.
If half the time I spend with an LLM is telling it what to do, and then another half is correcting what it did, then I'm not really saving any time at all by using it.