Thank you for writing about this topic! To go back to your early question, "how do you know who to trust in business?", the irony is that the people who do mention that they're not sure about the information they're advancing, and who therefore *look* less confident, are actually the ones we should put more trust in, compared to those who are consistently confident about AI-resurfaced information.
Awareness of low confidence > clueless high confidence, but the latter may still inspire more trust for many people
This has been bothering me for a while, so thank you for putting it into words.
I think we’ve lost sight of the fact that producing a doc was never the end goal — it was a forcing function for gathering data, reflecting on it, and thinking through what to do about it. When we use an LLM to produce that output and don’t bother to verify it, we’re giving up the whole thinking process that helps us develop good judgment in the first place.
Unfortunately, I see leaders reinforcing this mindset: building is easy with LLMs, so just try stuff and see what works. This is probably mostly true at the solution level, but picking the right problems to solve is a different matter. It’s still expensive to spend weeks and months on the wrong track, even if you’re spitting out code at 10x speed.
If I had a dollar for every time Claude tells me “that reframes everything” when I remember a detail mid-conversation, I’d probably break even on token spend. It’s a helpful reminder as to why fully outsourcing product strategy to an LLM is a bad idea and we need to keep our hands on the wheel.
This is what Claude once told me - “I constructed a fake proof to make a confident-sounding answer, and it talked you out of the right one.” And I was sloppy here, doing a data analysis for leadership meet, I had just accepted Claude output when it gave me some confident sounding narrative on initial pushback. I have come up with a framework to avoid such moments, I call OHS framework- named after my OH Shit moment! Basically, I ask for observation, hypothesis and supporting evidence; not a definitive claim. It is certainly not foolproof, but does help. I wrote a more detailed piece on it as well https://evalsense.substack.com/p/ohs-framework-avoid-oh-sht-moments
The number of times I've asked one of the AI tools to send me their data source, only to be told there isn't one and they had just 'guessed' is surreal. We absolutely need to validate all hypothesis with legit data, including AI hypothesis.
No method by itself is sufficient to this task, but a fantastic starting point is Socratic Dialogue, with or without AI.
Deeper than that is our posture towards knowledge within the context of Reality, defined as that state that does not negotiate with us. Human failure is an ineffective action in the face of Reality. Reality is the ultimate denominator relative to everything else. It’s for that reason that we learn “Life Lessons” as we mature, but Life learns no lessons about us. It doesn’t have to. It does not care nor does it cater.
Thank you for writing about this topic! To go back to your early question, "how do you know who to trust in business?", the irony is that the people who do mention that they're not sure about the information they're advancing, and who therefore *look* less confident, are actually the ones we should put more trust in, compared to those who are consistently confident about AI-resurfaced information.
Awareness of low confidence > clueless high confidence, but the latter may still inspire more trust for many people
This has been bothering me for a while, so thank you for putting it into words.
I think we’ve lost sight of the fact that producing a doc was never the end goal — it was a forcing function for gathering data, reflecting on it, and thinking through what to do about it. When we use an LLM to produce that output and don’t bother to verify it, we’re giving up the whole thinking process that helps us develop good judgment in the first place.
Unfortunately, I see leaders reinforcing this mindset: building is easy with LLMs, so just try stuff and see what works. This is probably mostly true at the solution level, but picking the right problems to solve is a different matter. It’s still expensive to spend weeks and months on the wrong track, even if you’re spitting out code at 10x speed.
If I had a dollar for every time Claude tells me “that reframes everything” when I remember a detail mid-conversation, I’d probably break even on token spend. It’s a helpful reminder as to why fully outsourcing product strategy to an LLM is a bad idea and we need to keep our hands on the wheel.
This is what Claude once told me - “I constructed a fake proof to make a confident-sounding answer, and it talked you out of the right one.” And I was sloppy here, doing a data analysis for leadership meet, I had just accepted Claude output when it gave me some confident sounding narrative on initial pushback. I have come up with a framework to avoid such moments, I call OHS framework- named after my OH Shit moment! Basically, I ask for observation, hypothesis and supporting evidence; not a definitive claim. It is certainly not foolproof, but does help. I wrote a more detailed piece on it as well https://evalsense.substack.com/p/ohs-framework-avoid-oh-sht-moments
The number of times I've asked one of the AI tools to send me their data source, only to be told there isn't one and they had just 'guessed' is surreal. We absolutely need to validate all hypothesis with legit data, including AI hypothesis.
No method by itself is sufficient to this task, but a fantastic starting point is Socratic Dialogue, with or without AI.
Deeper than that is our posture towards knowledge within the context of Reality, defined as that state that does not negotiate with us. Human failure is an ineffective action in the face of Reality. Reality is the ultimate denominator relative to everything else. It’s for that reason that we learn “Life Lessons” as we mature, but Life learns no lessons about us. It doesn’t have to. It does not care nor does it cater.