我对强化学习的看法
2 分•作者: umjunsik132•4 个月前
我认为强化学习（RL）是一种通过模型预测来生成训练数据的方法——它直接促使模型扩展其输出范围，因为数据的多样性增加了。然而，强化学习的根本在于它依赖于自举法（bootstrapping）并存在移动目标问题（moving target problem），这也是其稳定性差的原因。近似价值函数（value function）最易处理的方法之一是时序差分（TD）学习，但这会导致样本噪声、函数逼近器误差和移动目标问题。我认为我们需要在贝尔曼方程（Bellman equation）层面扩展纯粹的强化学习理论，以实现更稳定的强化学习。因此，我们需要一个更好的价值函数数学基础和一个与之协调且没有问题的易处理的近似方法。
查看原文
I think RL as a method which produces training data by model's predictions — It directly leads the model to extend its output range because of increased diversity of the data. However, fundamentally RL relies on bootstrapping and has moving target problem which are the reason of its poor stability. One of the most tractable method to approximate value function is TD which causes sample noise, function approximator error and moving target problems. I argue that we need to extend pure RL theory at the level of the Bellman equation to achieve more stable RL. Consequently, we need both a better mathematical foundation for value functions and a tractable approximation method that are aligned with each other — free from problems