Not crazy! I listened to a software engineering daily episode about pieces.app. Right now it’s some dev productivity tool or something, but in the interview the guy laid out a crazy vision that sounds like what you’re talking about.
He was talking about eventually having an agent that watches your screen and remembers what you do across all apps, and can store it and share it with you team.
So you could say “how does my teammate run staging builds?” or “what happened to the documentation on feature x that we never finished building”, and it’ll just know.
Obviously that’s far away, and it was just the ramblings of excited founder, but it’s fun to think about. Not sure if I hate it or love it lol
Being able to ask about stuff other people do seems like it could be ripe with privacy issues, honestly. Even if the model was limited to only recording work stuff, I don't think I would want that. Imagine "how often does my coworker browse to HN during work" or "list examples of dumb mistakes my coworkers have made" for some not-so-bad examples.
Even later it will be ingesting camera feeds from your AR glasses and listening in on your conversations, so you can remember what you agreed on. Just like automated meeting notes with Zoom which already exists, but it will be for real life 24/7.
Speech-to-text works. OCR works. LLMs are quite good at getting the semantics of the extracted text. Image understanding is pretty good too already. Just with the things that already exist right now, you can go most of the way.
And the CCTV cameras will also all be processed through something like it.
He was talking about eventually having an agent that watches your screen and remembers what you do across all apps, and can store it and share it with you team.
So you could say “how does my teammate run staging builds?” or “what happened to the documentation on feature x that we never finished building”, and it’ll just know.
Obviously that’s far away, and it was just the ramblings of excited founder, but it’s fun to think about. Not sure if I hate it or love it lol