神经网络变得轻松（第四十二部分）：模型拖延症、原因和解决方案·进阶篇
（2/3）·强化学习模型卡顿不是玄学，环境、目标与反馈三线失配才是主因
◍ Actor 与 Critic 的网络层堆叠
在 MT5 用 OpenCL 跑强化学习策略时，Actor 和 Critic 两套网络都靠 CLayerDescription 逐个堆层。下面这段就是两层网络的前向结构定义，外汇与贵金属行情噪声大，这种带批归一化的卷积栈对过拟合有一定抑制作用，但实盘仍属高风险。 Actor 输入层 window=0、无激活、优化器 ADAM；第一层 BatchNorm 的 batch 写死 1000，count 沿用上一层；第二层卷积把 prev_count 减 2 后作为通道数，window=3、step=1、window_out=2，激活用 LReLU；第三层卷积 count 取 (prev_count+1)/2，window=2、step=2、window_out=4，激活换 SIGMOID 收口。 Critic 清场后重搭：输入层 count 直接算成 HistoryBars*12+9，同样 window=0、ADAM；后面 BatchNorm 与两层卷积的参数和 Actor 几乎镜像，只是第三层在截图处截断。注意每层 new 失败或 Add 失败都会 delete descr 并 return false，内存泄漏点就藏在这些 early return 里。
descr.window = class="num">0; descr.activation = None; descr.optimization = ADAM; if(!actor.Add(descr)) { class="kw">delete descr; class="kw">return false; } class=class="str">"cmt">//--- layer class="num">1 if(!(descr = new CLayerDescription())) class="kw">return false; descr.type = defNeuronBatchNormOCL; descr.count = prev_count; descr.batch = class="num">1000; descr.activation = None; descr.optimization = ADAM; if(!actor.Add(descr)) { class="kw">delete descr; class="kw">return false; } class=class="str">"cmt">//--- layer class="num">2 if(!(descr = new CLayerDescription())) class="kw">return false; descr.type = defNeuronConvOCL; prev_count=descr.count = prev_count-class="num">2; descr.window = class="num">3; descr.step = class="num">1; descr.window_out = class="num">2; prev_count*=descr.window_out; descr.activation = LReLU; descr.optimization = ADAM; if(!actor.Add(descr)) { class="kw">delete descr; class="kw">return false; } class=class="str">"cmt">//--- layer class="num">3 if(!(descr = new CLayerDescription())) class="kw">return false; descr.type = defNeuronConvOCL; descr.count = (prev_count+class="num">1)/class="num">2; descr.window = class="num">2; descr.step = class="num">2; descr.window_out = class="num">4; descr.activation = SIGMOID; descr.optimization = ADAM; if(!actor.Add(descr)) { class="kw">delete descr; class="kw">return false; } class=class="str">"cmt">//--- Critic critic.Clear(); class=class="str">"cmt">//--- Input layer if(!(descr = new CLayerDescription())) class="kw">return false; descr.type = defNeuronBaseOCL; prev_count = descr.count = (class="type">int)(HistoryBars * class="num">12 + class="num">9); descr.window = class="num">0; descr.activation = None; descr.optimization = ADAM; if(!critic.Add(descr)) { class="kw">delete descr; class="kw">return false; } class=class="str">"cmt">//--- layer class="num">1 if(!(descr = new CLayerDescription())) class="kw">return false; descr.type = defNeuronBatchNormOCL; descr.count = prev_count; descr.batch = class="num">1000; descr.activation = None; descr.optimization = ADAM; if(!critic.Add(descr)) { class="kw">delete descr; class="kw">return false; } class=class="str">"cmt">//--- layer class="num">2 if(!(descr = new CLayerDescription())) class="kw">return false; descr.type = defNeuronConvOCL; prev_count=descr.count = prev_count-class="num">2; descr.window = class="num">3; descr.step = class="num">1; descr.window_out = class="num">2; prev_count*=descr.window_out; descr.activation = LReLU; descr.optimization = ADAM; if(!critic.Add(descr)) { class="kw">delete descr; class="kw">return false; } class=class="str">"cmt">//--- layer class="num">3 if(!(descr = new CLayerDescription()))
「判别器与调度器的层堆配置」
这段配置在 MT5 的 OpenCL 神经网络框架里分别给 critic（判别器）和 scheduler（调度器）堆层。critic 从第 3 层起用卷积层 defNeuronConvOCL，窗口 2、步长 2、输出窗口 4，激活统一 SIGMOID，优化用 ADAM；第 5、6 层退回到全连接 defNeuronBaseOCL，各 500 节点、TANH 激活。 第 7 层是分位数输出层 defNeuronFQF，count=4 且 window_out=32，意味着把状态压成 4 个分位数、每个展 32 维特征，给后续分布式决策用。任何一层 Add 失败都走 delete descr 并 return false，防止半吊子网络被训练。 scheduler 的输入层直接吃 9+40=49 维原始特征（prev_count=49），紧跟一层 defNeuronBatchNormOCL 做批量归一，batch 设 1000、激活 None。外汇与贵金属行情噪声大，这种归一前置能缓解不同品种量纲差异导致的梯度抖动，但高杠杆下模型误判仍可能迅速放大亏损。 开 MT5 把这段贴进 EA 的神经网络初始化函数，改 descr.count 或 batch 就能直观看到显存占用和回测稳定性的变化。
class="kw">return false; descr.type = defNeuronConvOCL; descr.count = (prev_count+class="num">1)/class="num">2; descr.window = class="num">2; descr.step = class="num">2; descr.window_out = class="num">4; descr.activation = SIGMOID; descr.optimization = ADAM; if(!critic.Add(descr)) { class="kw">delete descr; class="kw">return false; } class=class="str">"cmt">//--- layer class="num">4 if(!(descr = new CLayerDescription())) class="kw">return false; descr.type = defNeuronConvOCL; descr.count = class="num">150; descr.window = class="num">2; descr.step = class="num">2; descr.window_out = class="num">4; descr.activation = SIGMOID; descr.optimization = ADAM; if(!critic.Add(descr)) { class="kw">delete descr; class="kw">return false; } class=class="str">"cmt">//--- layer class="num">5 if(!(descr = new CLayerDescription())) class="kw">return false; descr.type = defNeuronBaseOCL; descr.count = class="num">500; descr.optimization = ADAM; descr.activation = TANH; if(!critic.Add(descr)) { class="kw">delete descr; class="kw">return false; } class=class="str">"cmt">//--- layer class="num">6 if(!(descr = new CLayerDescription())) class="kw">return false; descr.type = defNeuronBaseOCL; descr.count = class="num">500; descr.activation = TANH; descr.optimization = ADAM; if(!critic.Add(descr)) { class="kw">delete descr; class="kw">return false; } class=class="str">"cmt">//--- layer class="num">7 if(!(descr = new CLayerDescription())) class="kw">return false; descr.type = defNeuronFQF; descr.count = class="num">4; descr.window_out = class="num">32; descr.optimization = ADAM; if(!critic.Add(descr)) { class="kw">delete descr; class="kw">return false; } class=class="str">"cmt">//--- Scheduler scheduler.Clear(); class=class="str">"cmt">//--- Input layer if(!(descr = new CLayerDescription())) class="kw">return false; descr.type = defNeuronBaseOCL; prev_count = descr.count = (class="num">9 + class="num">40); descr.window = class="num">0; descr.activation = None; descr.optimization = ADAM; if(!scheduler.Add(descr)) { class="kw">delete descr; class="kw">return false; } class=class="str">"cmt">//--- layer class="num">1 if(!(descr = new CLayerDescription())) class="kw">return false; descr.type = defNeuronBatchNormOCL; descr.count = prev_count; descr.batch = class="num">1000; descr.activation = None; descr.optimization = ADAM; if(!scheduler.Add(descr)) { class="kw">delete descr; class="kw">return false; } class=class="str">"cmt">//--- layer class="num">2 if(!(descr = new CLayerDescription())) class="kw">return false; descr.type = defNeuronBaseOCL;
堆叠网络层与状态结构的写法
这段代码在调度器里连续登记了 5 层神经网络：第 3、4 层都是 256 个神经元的 base 层，激活函数用 TANH，优化器统一走 ADAM；第 4 层输出维度降到 10，且没设激活函数；第 5 层是 SoftMax 层，count 也是 10、step 为 1，同样挂 ADAM。每一层 new 出 CLayerDescription 后若 Add 失败就 delete 并 return false，能保证资源不漏。 SState 结构体把历史状态摊平成 float 数组：state 长度 = HistoryBars * 12，account 固定 9 维，并重载了 = 运算符用 ArrayCopy 直接拷两块内存，省掉逐元素循环。 外部输入参数给了三个双精度变量：ProfitToSave=10、MoneyTP=10、MoneySL=5，单位依上下文可能是点数或金额，外汇与贵金属品种波动剧烈，实盘套用前需在 MT5 策略测试器里把这组数值和压力测试跑通，参数不当可能放大回撤。 开 MT5 把这段贴进 EA 的层定义函数，重点看第 4 层少写 activation 后网络是否仍能编译通过，借此确认你用的 MT5 构建版本对默认激活的处理逻辑。
descr.count = class="num">256; descr.optimization = ADAM; descr.activation = TANH; if(!scheduler.Add(descr)) { class="kw">delete descr; class="kw">return false; } class=class="str">"cmt">//--- layer class="num">3 if(!(descr = new CLayerDescription())) class="kw">return false; descr.type = defNeuronBaseOCL; descr.count = class="num">256; descr.optimization = ADAM; descr.activation = TANH; if(!scheduler.Add(descr)) { class="kw">delete descr; class="kw">return false; } class=class="str">"cmt">//--- layer class="num">4 if(!(descr = new CLayerDescription())) class="kw">return false; descr.type = defNeuronBaseOCL; descr.count = class="num">10; descr.optimization = ADAM; if(!scheduler.Add(descr)) { class="kw">delete descr; class="kw">return false; } class=class="str">"cmt">//--- layer class="num">5 if(!(descr = new CLayerDescription())) class="kw">return false; descr.type = defNeuronSoftMaxOCL; descr.count = class="num">10; descr.step = class="num">1; descr.optimization = ADAM; if(!scheduler.Add(descr)) { class="kw">delete descr; class="kw">return false; } class=class="str">"cmt">//--- class="kw">return true; } class="kw">struct SState { class="type">float state[HistoryBars * class="num">12]; class="type">float account[class="num">9]; class=class="str">"cmt">//--- SState(class="type">void); class=class="str">"cmt">//--- class="type">bool Save(class="type">int file_handle); class="type">bool Load(class="type">int file_handle); class=class="str">"cmt">//--- overloading class="type">void class="kw">operator=(const SState &obj) { ArrayCopy(state, obj.state); ArrayCopy(account, obj.account); } }; class=class="str">"cmt">//+------------------------------------------------------------------+ class=class="str">"cmt">//| Input parameters | class=class="str">"cmt">//+------------------------------------------------------------------+ input class="type">class="kw">double ProfitToSave = class="num">10; input class="type">class="kw">double MoneyTP = class="num">10; input class="type">class="kw">double MoneySL = class="num">5; class=class="str">"cmt">//+------------------------------------------------------------------+ class=class="str">"cmt">//| Expert tick function | class=class="str">"cmt">//+------------------------------------------------------------------+ class="type">void OnTick() { class=class="str">"cmt">//---
◍ 把K线与账户状态压进一个特征向量
做价格行为建模时，最麻烦的不是算指标，而是把不同维度的数据规整成模型能直接吃的连续数组。下面这段逻辑在每个新Bar触发时跑一次，先把历史K线拉下来，再逐根把涨跌、振幅、成交量、时间分量和技术指标塞进固定宽度的浮点数组。 特征宽度定死为12，意味着每根Bar在 sState.state 里占 12 个 float 位：0~2 是 close/high/low 相对开盘价的偏移，3 是 tick_volume 除以 1000，4~6 是小时/星期几/月份，7~11 依次是 RSI、CCI、ATR、MACD主线、MACD信号。这样 HistoryBars 根K线就线性铺成 b*12 的偏移，回测或推理时不用再反复查结构体。 账户侧另开 account[5]，依次装余额、净值、空闲保证金、保证金水平百分比、浮动盈亏。外汇和贵金属杠杆高，保证金水平掉到阈值以下会被强平，把这个量送进特征能让模型感知当前风险暴露，但任何模型输出都只是概率倾向，不代表未来方向。 持仓扫描只统计当前品种，用 PositionsTotal 遍历并跳过非本符号仓位，为后续区分多空市值和浮盈留接口。开 MT5 把这段直接贴进 EA 的 OnTick 前面，改 HistoryBars 从 100 调到 300，能看到 state 数组长度从 1200 涨到 3600，特征密度立刻变厚。
if(!IsNewBar()) class="kw">return; class=class="str">"cmt">//--- class="type">int bars = CopyRates(Symb.Name(), TimeFrame, iTime(Symb.Name(), TimeFrame, class="num">1), HistoryBars, Rates); if(!ArraySetAsSeries(Rates, true)) class="kw">return; class=class="str">"cmt">//--- RSI.Refresh(); CCI.Refresh(); ATR.Refresh(); MACD.Refresh(); class=class="str">"cmt">//--- class="type">MqlDateTime sTime; for(class="type">int b = class="num">0; b < (class="type">int)HistoryBars; b++) { class="type">float open = (class="type">float)Rates[b].open; TimeToStruct(Rates[b].time, sTime); class="type">float rsi = (class="type">float)RSI.Main(b); class="type">float cci = (class="type">float)CCI.Main(b); class="type">float atr = (class="type">float)ATR.Main(b); class="type">float macd = (class="type">float)MACD.Main(b); class="type">float sign = (class="type">float)MACD.Signal(b); if(rsi == EMPTY_VALUE || cci == EMPTY_VALUE || atr == EMPTY_VALUE || macd == EMPTY_VALUE || sign == EMPTY_VALUE) class="kw">continue; class=class="str">"cmt">//--- sState.state[b * class="num">12] = (class="type">float)Rates[b].close - open; sState.state[b * class="num">12 + class="num">1] = (class="type">float)Rates[b].high - open; sState.state[b * class="num">12 + class="num">2] = (class="type">float)Rates[b].low - open; sState.state[b * class="num">12 + class="num">3] = (class="type">float)Rates[b].tick_volume / class="num">1000.0f; sState.state[b * class="num">12 + class="num">4] = (class="type">float)sTime.hour; sState.state[b * class="num">12 + class="num">5] = (class="type">float)sTime.day_of_week; sState.state[b * class="num">12 + class="num">6] = (class="type">float)sTime.mon; sState.state[b * class="num">12 + class="num">7] = rsi; sState.state[b * class="num">12 + class="num">8] = cci; sState.state[b * class="num">12 + class="num">9] = atr; sState.state[b * class="num">12 + class="num">10] = macd; sState.state[b * class="num">12 + class="num">11] = sign; } class=class="str">"cmt">//--- sState.account[class="num">0] = (class="type">float)AccountInfoDouble(ACCOUNT_BALANCE); sState.account[class="num">1] = (class="type">float)AccountInfoDouble(ACCOUNT_EQUITY); sState.account[class="num">2] = (class="type">float)AccountInfoDouble(ACCOUNT_MARGIN_FREE); sState.account[class="num">3] = (class="type">float)AccountInfoDouble(ACCOUNT_MARGIN_LEVEL); sState.account[class="num">4] = (class="type">float)AccountInfoDouble(ACCOUNT_PROFIT); class=class="str">"cmt">//--- class="type">class="kw">double buy_value = class="num">0, sell_value = class="num">0, buy_profit = class="num">0, sell_profit = class="num">0; class="type">int total = PositionsTotal(); for(class="type">int i = class="num">0; i < total; i++) { if(PositionGetSymbol(i) != Symb.Name()) class="kw">continue;
「持仓分类与强化学习信号的分发逻辑」
这段逻辑先把当前账户里的多空持仓拆开统计。switch 按 POSITION_TYPE 分支，把每笔买单的量与浮动盈利累加进 buy_value / buy_profit，卖单同理进 sell_value / sell_profit，随后写进 sState.account 数组的第 5~8 位（索引从 0 计），供后续网络推理使用。 推理阶段把账户状态送进 Actor 网络（feedForward 隐藏层 12、训练标志 false），拿到 Result 后再把账户数组与 Result 拼接，喂给 Schedule 网络得到动作编号 act。若总浮盈 profit 越过 MoneyTP 或跌破 -MathAbs(MoneySL)，则强制 act=2，相当于硬止损止盈覆盖模型输出。外汇与贵金属杠杆高，这种硬阈值若设得过窄，可能频繁截断模型原本的持仓倾向。 Train 函数里走的是带评论家的强化更新：Actor 与 Critic 出结果后，用 Buffer 里的奖励减 Critic 最大值再除 100 得到缩放 reward，按所记录 action 列做 Scheduler 输出的 softmax 加权，再乘 reward 反传。CriticResult 对应动作位被直接赋成当期奖励，然后分别 backProp 更新 Scheduler 与 Critic。 让小布替你跑这套 把 MoneyTP / MoneySL 先置成账户净值的 1% 与 0.8% 做离线回测，观察 act=2 触发次数；若占比超 30%，说明策略对硬止损依赖过强，可能削弱模型在趋势段的持仓能力。
class="kw">switch((class="type">int)PositionGetInteger(POSITION_TYPE)) { case POSITION_TYPE_BUY: buy_value += PositionGetDouble(POSITION_VOLUME); buy_profit += PositionGetDouble(POSITION_PROFIT); class="kw">break; case POSITION_TYPE_SELL: sell_value += PositionGetDouble(POSITION_VOLUME); sell_profit += PositionGetDouble(POSITION_PROFIT); class="kw">break; } } sState.account[class="num">5] = (class="type">float)buy_value; sState.account[class="num">6] = (class="type">float)sell_value; sState.account[class="num">7] = (class="type">float)buy_profit; sState.account[class="num">8] = (class="type">float)sell_profit; State1.AssignArray(sState.state); if(!Actor.feedForward(GetPointer(State1), class="num">12, false)) class="kw">return; Actor.getResults(Result); State1.AssignArray(sState.account); State1.AddArray(Result); if(!Schedule.feedForward(GetPointer(State1), class="num">12, false)) class="kw">return; class="type">int act = GetAction(Result, Schedule.getSample(), Models); class="type">class="kw">double profit = buy_profit + sell_profit; if(profit >= MoneyTP || profit <= -MathAbs(MoneySL)) act = class="num">2; class="type">void Train(class="type">void) { ........ ........ Actor.getResults(ActorResult); Critic.getResults(CriticResult); State1.AssignArray(Buffer[tr].States[i].account); State1.AddArray(ActorResult); if(!Scheduler.feedForward(GetPointer(State1), class="num">12, false)) class="kw">return; Scheduler.getResults(SchedulerResult); class=class="str">"cmt">//--- class="type">class="kw">ulong actions = ActorResult.Size() / Models; matrix<class="type">float> temp; temp.Init(class="num">1, ActorResult.Size()); temp.Row(ActorResult, class="num">0); temp.Reshape(Models, actions); class="type">float reward=(Buffer[tr].Revards[i] - CriticResult.Max())/class="num">100; class="type">int action=Buffer[tr].Actions[i]; SchedulerResult=SchedulerResult*temp.Col(action); SchedulerResult.Activation(SchedulerResult,AF_SOFTMAX); SchedulerResult = SchedulerResult * reward; Result.AssignArray(SchedulerResult); class=class="str">"cmt">//--- if(!Scheduler.backProp(GetPointer(Result))) class="kw">return; CriticResult[action] = Buffer[tr].Revards[i]; Result.AssignArray(CriticResult); class=class="str">"cmt">//--- if(!Critic.backProp(GetPointer(Result), class="num">0.0f, NULL)) class="kw">return; class=class="str">"cmt">//--- for(class="type">int r = class="num">0; r < Models; r++)
策略梯度回写时的行内权重修正
强化学习代理在更新 Actor 网络时，会把单条经验的动作价值直接叠加进对应动作的概率行。下面这段代码展示了从临时矩阵取一行、按 reward 缩放、再做 Softmax 归一化的过程。 [CODE] { vector<float> row = temp.Row(r); row[action] += row[action] * reward; row.Activation(row, AF_SOFTMAX); temp.Row(row, r); } [/CODE] 逐行看：先取出第 r 行的浮点向量；将该行中被执行动作的下标值乘以 reward 后累加，相当于局部提升或压低该动作权重；随后对整行做 Softmax 激活，把数值转回概率分布；写回原矩阵行。 temp.Reshape(1, ActorResult.Size()) 把矩阵压成单行，Result.AssignArray(temp.Row(0)) 提取为最终输出。最后若 backProp 失败直接 return，避免用脏梯度更新网络。 在 MT5 里跑这套时，reward 符号决定该动作下次被采样的倾向——正奖励抬高概率，负奖励压低，但外汇与贵金属杠杆高，策略过拟合单条经验会引发实盘漂移，验证时建议先关掉 live 交易只用历史 tick 回放。
{
vector<class="type">float> row = temp.Row(r);
row[action] += row[action] * reward;
row.Activation(row, AF_SOFTMAX);
temp.Row(row, r);
}
temp.Reshape(class="num">1, ActorResult.Size());
Result.AssignArray(temp.Row(class="num">0));
class=class="str">"cmt">//---
if(!Actor.backProp(GetPointer(Result)))
class="kw">return;