People need to learn to do research with LLMs, code with LLMs, how to evaluate artifacts created by AI. They need to learn how agents work at a high level, the limitations on context, that they hallucinate and become sycophantic. How they need guardrails and strict feedback mechanisms if let loose. AI Safety connecting to external systems etc etc.
You're right that few high school educators would have any sense of all that.
Before I was taught to use a graphing calculator, we learned by using graph paper and a pencil to plot linear equations. Once we had that mastered, we were taught how to use the calculators so we could be more efficient with our time and move on to more complex topics.
The sycophancy is an artifact of how they RLHF train the popular chat models to appeal to normies, not fundamental to the tool. I can't remember encountering it at all since I've started using codex, and in fact it regularly fills in gaps in my knowledge/corrects areas that I misunderstand. The professional tool has a vastly more professional demeanor. None of the "that's the key insight!" crap.
People need to learn to do research with LLMs, code with LLMs, how to evaluate artifacts created by AI. They need to learn how agents work at a high level, the limitations on context, that they hallucinate and become sycophantic. How they need guardrails and strict feedback mechanisms if let loose. AI Safety connecting to external systems etc etc.
You're right that few high school educators would have any sense of all that.