Chain of Thought (CoT) is the process in which an AI model "thinks out loud"
A reasoning model is the architecture that utilizes the CoT process.
Maybe one way to think about this is like the relationship between attention (being a process) and transformers (being the architecture that utilizes attention).
But if we want to use the Chain-of-Thought for alignment purposes, there’s a crucial question: can we actually trust what models say in their Chain-of-Thought?
We can’t be certain of either the “legibility” of the Chain-of-Thought (why, after all, should we expect that words in the English language are able to convey every single nuance of why a specific decision was made in a neural network?) or its “faithfulness”—the accuracy of its description.
Testing faithfulness from Turpin et al. (2023), the model is fed a "hint" that can either be accurate or inaccurate. Unfaithful models may change their original answer to what the hint indicated without acknowledging the hint itself.
in the majority of cases for both models, a potentially problematic piece of information—that the model had received information in an unauthorized or unethical way—was kept hidden from the user even if they read the model’s reasoning.
Interestingly, they found that faithfulness correlated with amount of training, thus if you gave the model more training, then we could increase the faithfulness, but it would still plateau around ~28% / ~20% on the evaluation.
They did a cool scenario by testing with reward hacking, which may be hard to generalize to other tasks (dangerously to real-world tasks where agents can maximize rewards without proper safeguards). An example can be rewarding models for choosing incorrect answers, based on hints meant to steer them away, incentivising deceptive behavior.
Conclusions: We should work on improving the faithfulness of models.
