My comments will be inadequate, necessarily. Your curiosity and interests lead me to share a few of the questions I have found to be salient to themes in your post and I hope they might be useful for you also. First, I think the hypothesis within your conditional statement concluding section 3 is true and I generally agree with the sentence beginning section 4. However, I think the central unresolved problem(s) are probably substantially different than the factors you refer to with the apt term "threshold-setting." In my view, although model evaluations should be oriented as you propose, and the three dimensions you specify as particularly requiring measurement should be included among the important goals of any adequate testing regime, I have the impression that you presuppose an assumption that, in my estimation, is unwarranted. I think that adequate oversight and the required testing must prioritize reasonable calculations of risk probabilities, based on roles given to AI processes within dynamical systems, in relation to the limits that can predictably be imposed on the predictable possible consequences inherent in the overall system dynamics. I know that's a mouthful. Here's an example. The famous "human-in-the-loop" constraint required of AI-powered weapons, such as drones or targeting mechanisms incorporated into human-controlled aircraft, tanks, ships, etc., necessarily constrains the dynamics of the (otherwise autonomous) control systems in ways that intrinsically limit the utility of the intelligence. The *evaluations* provided by the system cannot be so far beyond the overall capabilities of the human cognition "in the loop" that the human cannot assess the quality of the intelligence. But that limit defeats much of the purpose of building the intelligence into the system in the first place, and will always be in tension with the reasons for building weaponry more lethal than the enemy's. At some point in the development of autonomous systems, any human supposedly in the loop becomes irrelevant, because the only source of information the human can rely on for making the "kill-don't kill" decision is the intelligence embedded in the killing machine itself. To reiterate, that's an example of a much broader set of issues that must be incorporated into the basic formulations of risk-assessment as applied to AI in general.
Hi Jordan, thanks for the interesting reply! I was planning to write about the complexity of cognitive oversight and the alignment problems.
I agree that it makes sense to shift the framing from "is this AI safe" to "is this deployment context capable of absorbing its failure modes"; it's closer to systems safety than to AI-centric alignment.
Calculating risk probabilities from AI's role in system dynamics requires modeling its contribution to consequences well enough to bound it — which is itself an evaluation problem, structurally similar to the one HITL runs into. The HITL argument is about the output-assessment problem — the human can't verify what the AI produces.
Cognitive oversight moves the risk-assessment target: don't verify outputs, verify the internal reasoning. This partially sidesteps HITL's weaknesses, since we're not asking the human to match the AI's cognition; we're asking whether the cognitive processes look structurally sound.
One example is AI-driven controllability of its own chain of thought. So far it seems that increased AI capabilities actually work in favor of AI oversight: models struggle to control their own chain of thought in longer reasoning traces. Cognitive oversight works for now, but continued evaluation matters as models advance.
My comments will be inadequate, necessarily. Your curiosity and interests lead me to share a few of the questions I have found to be salient to themes in your post and I hope they might be useful for you also. First, I think the hypothesis within your conditional statement concluding section 3 is true and I generally agree with the sentence beginning section 4. However, I think the central unresolved problem(s) are probably substantially different than the factors you refer to with the apt term "threshold-setting." In my view, although model evaluations should be oriented as you propose, and the three dimensions you specify as particularly requiring measurement should be included among the important goals of any adequate testing regime, I have the impression that you presuppose an assumption that, in my estimation, is unwarranted. I think that adequate oversight and the required testing must prioritize reasonable calculations of risk probabilities, based on roles given to AI processes within dynamical systems, in relation to the limits that can predictably be imposed on the predictable possible consequences inherent in the overall system dynamics. I know that's a mouthful. Here's an example. The famous "human-in-the-loop" constraint required of AI-powered weapons, such as drones or targeting mechanisms incorporated into human-controlled aircraft, tanks, ships, etc., necessarily constrains the dynamics of the (otherwise autonomous) control systems in ways that intrinsically limit the utility of the intelligence. The *evaluations* provided by the system cannot be so far beyond the overall capabilities of the human cognition "in the loop" that the human cannot assess the quality of the intelligence. But that limit defeats much of the purpose of building the intelligence into the system in the first place, and will always be in tension with the reasons for building weaponry more lethal than the enemy's. At some point in the development of autonomous systems, any human supposedly in the loop becomes irrelevant, because the only source of information the human can rely on for making the "kill-don't kill" decision is the intelligence embedded in the killing machine itself. To reiterate, that's an example of a much broader set of issues that must be incorporated into the basic formulations of risk-assessment as applied to AI in general.
Hi Jordan, thanks for the interesting reply! I was planning to write about the complexity of cognitive oversight and the alignment problems.
I agree that it makes sense to shift the framing from "is this AI safe" to "is this deployment context capable of absorbing its failure modes"; it's closer to systems safety than to AI-centric alignment.
Calculating risk probabilities from AI's role in system dynamics requires modeling its contribution to consequences well enough to bound it — which is itself an evaluation problem, structurally similar to the one HITL runs into. The HITL argument is about the output-assessment problem — the human can't verify what the AI produces.
Cognitive oversight moves the risk-assessment target: don't verify outputs, verify the internal reasoning. This partially sidesteps HITL's weaknesses, since we're not asking the human to match the AI's cognition; we're asking whether the cognitive processes look structurally sound.
One example is AI-driven controllability of its own chain of thought. So far it seems that increased AI capabilities actually work in favor of AI oversight: models struggle to control their own chain of thought in longer reasoning traces. Cognitive oversight works for now, but continued evaluation matters as models advance.