神经网络变得轻松（第四十八部分）：降低 Q-函数高估的方法·进阶篇
遍历持仓把多空盈亏拆进特征向量
这段逻辑干的事很直接：把当前账户下指定交易品种的所有持仓扫一遍，把买入手数、卖出手数、对应浮动盈亏分别累加，同时算一个带时间衰减的持仓折扣值。multiplyer 被设成 1.0/(60.0*60.0*10.0)，也就是 1/36000，意味着每持仓 36000 秒（10 小时）折扣系数大约吃掉等同利润的绝对量，时间越长对 position_discount 的拖累越明显。 循环里先用 PositionGetSymbol 过滤非目标品种，再用 POSITION_TYPE 分支把 volume 和 profit 归到 buy/sell 两组。position_discount 的累加式是 profit 减去「当前时间减开仓时间」乘 multiplyer 再乘利润绝对值，所以亏损单和盈利单都会按时间被打折，只是方向不同。 扫完之后把 7 个量写进 sState.account 的索引 2~6，再经 Account.Clear 和 Add 序列换算成相对 PrevBalance / PrevEquity 的比率，最后送进 Actor.feedForward 做推理。如果你在 MT5 里复刻，重点盯 multiplyer 这个常量——把它从 36000 调到 18000，时间衰减速度会翻倍，特征向量对隔夜单的敏感度会显著变化。外汇与贵金属杠杆高，这类特征工程改动会放大回测与实盘偏差，需自行验证。
class="type">class="kw">double buy_value = class="num">0, sell_value = class="num">0, buy_profit = class="num">0, sell_profit = class="num">0; class="type">class="kw">double position_discount = class="num">0; class="type">class="kw">double multiplyer = class="num">1.0 / (class="num">60.0 * class="num">60.0 * class="num">10.0); class="type">int total = PositionsTotal(); class="type">class="kw">datetime current = TimeCurrent(); for(class="type">int i = class="num">0; i < total; i++) { if(PositionGetSymbol(i) != Symb.Name()) class="kw">continue; class="type">class="kw">double profit = PositionGetDouble(POSITION_PROFIT); class="kw">switch((class="type">int)PositionGetInteger(POSITION_TYPE)) { case POSITION_TYPE_BUY: buy_value += PositionGetDouble(POSITION_VOLUME); buy_profit += profit; break; case POSITION_TYPE_SELL: sell_value += PositionGetDouble(POSITION_VOLUME); sell_profit += profit; break; } position_discount += profit - (current - PositionGetInteger(POSITION_TIME)) * multiplyer * MathAbs(profit); } sState.account[class="num">2] = (class="type">float)buy_value; sState.account[class="num">3] = (class="type">float)sell_value; sState.account[class="num">4] = (class="type">float)buy_profit; sState.account[class="num">5] = (class="type">float)sell_profit; sState.account[class="num">6] = (class="type">float)position_discount; class=class="str">"cmt">//--- Account.Clear(); Account.Add((class="type">float)((sState.account[class="num">0] - PrevBalance) / PrevBalance)); Account.Add((class="type">float)(sState.account[class="num">1] / PrevBalance)); Account.Add((class="type">float)((sState.account[class="num">1] - PrevEquity) / PrevEquity)); Account.Add(sState.account[class="num">2]); Account.Add(sState.account[class="num">3]); Account.Add((class="type">float)(sState.account[class="num">4] / PrevBalance)); Account.Add((class="type">float)(sState.account[class="num">5] / PrevBalance)); Account.Add((class="type">float)(sState.account[class="num">6] / PrevBalance)); if(Account.GetIndex() >= class="num">0) if(!Account.BufferWrite()) class="kw">return; class=class="str">"cmt">//--- if(!Actor.feedForward(GetPointer(State), class="num">1, false, GetPointer(Account))) class="kw">return; PrevBalance = sState.account[class="num">0]; PrevEquity = sState.account[class="num">1]; class=class="str">"cmt">//--- vector<class="type">float> temp; Actor.getResults(temp); class="type">float delta = MathAbs(ActorResult - temp).Sum(); ActorResult = temp; if(AddSmooth) { class="type">int err = class="num">0; for(class="type">ulong i = class="num">0; i < temp.Size(); i++)
◍ 多空双向的头寸与止损闸门
这段逻辑把买、卖两套控制完全对称地处理，核心是先拿经纪商限制卡一道门槛：最小手数、挂单步长、以及止损止盈距离不能低于 StopsLevel 换算出的 stops。以 XAUUSD 为例，若 StopsLevel 返回 10、Point 为 0.01，stops 就是 0.1 美元，任何低于该值的 TP/SL 会直接触发平仓而非开仓。 temp 数组前三个元素管买侧：temp[0] 是目标仓位占比，temp[1]、temp[2] 分别是 TP、SL 对应的 MaxTP/MaxSL 倍数。只要 temp[0] 算出来比 min_lot 还小，或 TP/SL 折算点数不超过 stops，现有多单会被 CloseByDirection 清掉；否则按 step_lot 圆整手数，用 Ask 加减点数算出 buy_tp、buy_sl，仓位不符就补单或 ClosePartial 减仓。 卖侧把 temp[3~5] 镜像用一遍，价格基准换成 Bid，SL 挂在 Bid 之上、TP 在 Bid 之下。最后 reward 以账户净值 Account[0] 起步，若多空总仓位为 0，再扣掉 atr 与 PrevBalance 的比值作为空载惩罚——这意味着横盘无持仓时，波动率越高对评分越不友好。 开 MT5 把这段贴进 EA 的 OnTick，把 MaxTP、MaxSL 先设成 50、30 跑回测，重点看 StopsLevel 在周末休市前后是否跳变导致异常平仓。外汇与贵金属杠杆高，stops 计算偏差可能瞬间放大滑点风险。
temp[i] += (class="type">float)(temp[i] * Math::MathRandomNormal(class="num">0, class="num">0.3, err)); temp.Clip(class="num">0.0f, class="num">1.0f); } class="type">class="kw">double min_lot = Symb.LotsMin(); class="type">class="kw">double step_lot = Symb.LotsStep(); class="type">class="kw">double stops = MathMax(Symb.StopsLevel(), class="num">1) * Symb.Point(); class=class="str">"cmt">//--- buy control if(temp[class="num">0] < min_lot || (temp[class="num">1] * MaxTP * Symb.Point()) <= stops || (temp[class="num">2] * MaxSL * Symb.Point()) <= stops) { if(buy_value > class="num">0) CloseByDirection(POSITION_TYPE_BUY); } else { class="type">class="kw">double buy_lot = min_lot+MathRound((class="type">class="kw">double)(temp[class="num">0]-min_lot) / step_lot) * step_lot; class="type">class="kw">double buy_tp = NormalizeDouble(Symb.Ask() + temp[class="num">1] * MaxTP * Symb.Point(), Symb.Digits()); class="type">class="kw">double buy_sl = NormalizeDouble(Symb.Ask() - temp[class="num">2] * MaxSL * Symb.Point(), Symb.Digits()); if(buy_value > class="num">0) TrailPosition(POSITION_TYPE_BUY, buy_sl, buy_tp); if(buy_value != buy_lot) { if(buy_value > buy_lot) ClosePartial(POSITION_TYPE_BUY, buy_value - buy_lot); else Trade.Buy(buy_lot - buy_value, Symb.Name(), Symb.Ask(), buy_sl, buy_tp); } } class=class="str">"cmt">//--- sell control if(temp[class="num">3] < min_lot || (temp[class="num">4] * MaxTP * Symb.Point()) <= stops || (temp[class="num">5] * MaxSL * Symb.Point()) <= stops) { if(sell_value > class="num">0) CloseByDirection(POSITION_TYPE_SELL); } else { class="type">class="kw">double sell_lot = min_lot+MathRound((class="type">class="kw">double)(temp[class="num">3]-min_lot) / step_lot) * step_lot;; class="type">class="kw">double sell_tp = NormalizeDouble(Symb.Bid() - temp[class="num">4] * MaxTP * Symb.Point(), Symb.Digits()); class="type">class="kw">double sell_sl = NormalizeDouble(Symb.Bid() + temp[class="num">5] * MaxSL * Symb.Point(), Symb.Digits()); if(sell_value > class="num">0) TrailPosition(POSITION_TYPE_SELL, sell_sl, sell_tp); if(sell_value != sell_lot) { if(sell_value > sell_lot) ClosePartial(POSITION_TYPE_SELL, sell_value - sell_lot); else Trade.Sell(sell_lot - sell_value, Symb.Name(), Symb.Bid(), sell_sl, sell_tp); } } class=class="str">"cmt">//--- class="type">float reward = Account[class="num">0]; if((buy_value + sell_value) == class="num">0) reward -= (class="type">float)(atr / PrevBalance);
「TD3训练回路里的样本抽取与目标构造」
这段逻辑是双延迟DDPG（TD3）智能体在MT5里做离线训练的核心循环。先看一下相关输入参数与网络实例声明，它们决定了训练规模与软更新强度。 input int Iterations = 1000000; input int UpdatePolicy = 3; input int UpdateTargets = 100; input float Tau = 0.01f; CNet Actor; CNet Critic1; CNet Critic2; CNet TargetActor; CNet TargetCritic1; CNet TargetCritic2; Iterations 设成 1000000 意味着最多跑百万次梯度步；Tau=0.01f 表示目标网络每次只吸收主网络 1% 的权重，属于慢跟随。UpdatePolicy=3 与 UpdateTargets=100 控制策略网络与目标网络的更新频率。 在 OnDeinit 里，EA 卸载前把三个目标网络用 Tau 做软更新并落盘成 .nnw 权重文件，其中 Critic 还写入了近期平均误差。这样下次加载可续训，不必从随机初始化重来。 Train 函数开头取回放池大小 total_tr，并用 GetTickCount 记时。循环里先用 MathRand()/32767.0 均匀抽一条轨迹 tr，再用 MathRand()*MathRand()/32767^2 的平方分布抽该轨迹内步号 i——平方分布让早期样本被抽中的概率更低，倾向聚焦轨迹中段以后的转移。 构造目标时，把下一状态 Buffer[tr].States[i+1].state 填进 State，并以账户权益变化率等 6 个量写进 Account 向量：余额变化率、权益/余额比、权益变化率，以及另外三项账户特征。这些归一化特征让 Critic 估算的 Q 值对绝对资金规模不敏感，在外汇或贵金属这种高杠杆品种上可能更稳定，但实盘仍属高风险，需自行回测验证。
input class="type">int Iterations = class="num">1000000; input class="type">int UpdatePolicy = class="num">3; input class="type">int UpdateTargets = class="num">100; input class="type">float Tau = class="num">0.01f; CNet Actor; CNet Critic1; CNet Critic2; CNet TargetActor; CNet TargetCritic1; CNet TargetCritic2; class="type">void OnDeinit(const class="type">int reason) { class=class="str">"cmt">//--- TargetActor.WeightsUpdate(GetPointer(Actor), Tau); TargetCritic1.WeightsUpdate(GetPointer(Critic1), Tau); TargetCritic2.WeightsUpdate(GetPointer(Critic2), Tau); TargetActor.Save(FileName + "Act.nnw", class="num">0, class="num">0, class="num">0, TimeCurrent(), true); TargetCritic1.Save(FileName + "Crt1.nnw", TargetCritic1.getRecentAverageError(), class="num">0, class="num">0, TimeCurrent(), true); TargetCritic1.Save(FileName + "Crt2.nnw", TargetCritic2.getRecentAverageError(), class="num">0, class="num">0, TimeCurrent(), true); class="kw">delete Result; } class="type">void Train(class="type">void) { class="type">int total_tr = ArraySize(Buffer); class="type">uint ticks = GetTickCount(); for(class="type">int iter = class="num">0; (iter < Iterations && !IsStopped()); iter ++) { class="type">int tr = (class="type">int)((MathRand() / class="num">32767.0) * (total_tr - class="num">1)); class="type">int i = (class="type">int)((MathRand() * MathRand() / MathPow(class="num">32767, class="num">2)) * (Buffer[tr].Total - class="num">2)); class=class="str">"cmt">//--- Target State.AssignArray(Buffer[tr].States[i + class="num">1].state); class="type">float PrevBalance = Buffer[tr].States[i].account[class="num">0]; class="type">float PrevEquity = Buffer[tr].States[i].account[class="num">1]; Account.Clear(); Account.Add((Buffer[tr].States[i + class="num">1].account[class="num">0] - PrevBalance) / PrevBalance); Account.Add(Buffer[tr].States[i + class="num">1].account[class="num">1] / PrevBalance); Account.Add((Buffer[tr].States[i + class="num">1].account[class="num">1] - PrevEquity) / PrevEquity); Account.Add(Buffer[tr].States[i + class="num">1].account[class="num">2]); Account.Add(Buffer[tr].States[i + class="num">1].account[class="num">3]); Account.Add(Buffer[tr].States[i + class="num">1].account[class="num">4] / PrevBalance);
双评论员网络下的奖励回传写法
这段逻辑跑在经验回放的训练循环里，先把下一状态（i+1）的账户特征按 PrevBalance 归一化后塞进 Account 缓冲，索引非负就落盘。随后 TargetActor 做前向推理，若失败直接 ExpertRemove 并 break，避免脏数据继续污染网络。 目标侧用两个 Critic 网络（TargetCritic1/2）对 Actor 输出做评估，任一 feedForward 失败同样终止。取两者结果的最小值乘以折扣因子 DiscFactor，再叠上相邻两步奖励差（Buffer[tr].Revards[i] - Buffer[tr].Revards[i+1]），构成保守 Q 值更新的 reward，降低外汇与贵金属行情突变时的过估计倾向。 当前状态 i 的特征重建时，PrevBalance / PrevEquity 取 i-1 与 0 的较大下标，防止越界；Account 清零后重新装入 7 个归一化字段（含余额差比、权益比、持仓数等）。实盘验证时可在 MT5 策略测试器把 DiscFactor 从 0.99 调到 0.95，观察奖励曲线波动是否收敛更快。 主网络 Actor 与 Critic1/2 的前向调用若返回 false，打印函数名与行号后移除 EA。这种双评论员 + 目标网络的代码结构，在 EURUSD 15 分钟复盘里可能比单 Critic 的回撤更浅，但仍属高风险验证，需自行跑样本外数据。
Account.Add(Buffer[tr].States[i + class="num">1].account[class="num">5] / PrevBalance); Account.Add(Buffer[tr].States[i + class="num">1].account[class="num">6] / PrevBalance); if(Account.GetIndex() >= class="num">0) Account.BufferWrite(); if(!TargetActor.feedForward(GetPointer(State), class="num">1, false, GetPointer(Account))) { PrintFormat("%s -> %d", __FUNCTION__, __LINE__); ExpertRemove(); break; } if(!TargetCritic1.feedForward(GetPointer(TargetActor), LatentLayer, GetPointer(TargetActor)) || !TargetCritic2.feedForward(GetPointer(TargetActor), LatentLayer, GetPointer(TargetActor))) { PrintFormat("%s -> %d", __FUNCTION__, __LINE__); ExpertRemove(); break; } TargetCritic1.getResults(Result); class="type">float reward = Result[class="num">0]; TargetCritic2.getResults(Result); reward = DiscFactor * MathMin(reward, Result[class="num">0]) + (Buffer[tr].Revards[i] - Buffer[tr].Revards[i + class="num">1]); class=class="str">"cmt">//--- Q-function study State.AssignArray(Buffer[tr].States[i].state); PrevBalance = Buffer[tr].States[MathMax(i - class="num">1, class="num">0)].account[class="num">0]; PrevEquity = Buffer[tr].States[MathMax(i - class="num">1, class="num">0)].account[class="num">1]; Account.Clear(); Account.Add((Buffer[tr].States[i].account[class="num">0] - PrevBalance) / PrevBalance); Account.Add(Buffer[tr].States[i].account[class="num">1] / PrevBalance); Account.Add((Buffer[tr].States[i].account[class="num">1] - PrevEquity) / PrevEquity); Account.Add(Buffer[tr].States[i].account[class="num">2]); Account.Add(Buffer[tr].States[i].account[class="num">3]); Account.Add(Buffer[tr].States[i].account[class="num">4] / PrevBalance); Account.Add(Buffer[tr].States[i].account[class="num">5] / PrevBalance); Account.Add(Buffer[tr].States[i].account[class="num">6] / PrevBalance); class=class="str">"cmt">//--- if(Account.GetIndex() >= class="num">0) Account.BufferWrite(); class=class="str">"cmt">//--- if(!Actor.feedForward(GetPointer(State), class="num">1, false, GetPointer(Account))) { PrintFormat("%s -> %d", __FUNCTION__, __LINE__); ExpertRemove(); break; } if(!Critic1.feedForward(Result,class="num">1,false, GetPointer(Actions)) || !Critic2.feedForward(Result,class="num">1,false, GetPointer(Actions))) {
◍ 策略网络在回放缓冲里的更新节奏
这段逻辑跑在强化学习训练循环里，核心是在满足迭代间隔时才做策略（Actor）与评论家（Critic）的前向推演。用 MathRand() 除以 32767.0 取均匀随机轨迹索引 tr，再用两次 MathRand() 乘积除以 32767 的平方做二级随机，把采样点 i 压到单条轨迹可用长度减 2 以内，避免越界读状态。 账户特征做了相对化：当前权益与上一帧权益的比值差、余额差值除上一余额等 8 个量写入 Account 向量，再喂给 Actor.feedForward。任何一步 feedForward 或 backProp 返回失败，就 PrintFormat 打出函数名与行号并 ExpertRemove 退出 EA——实盘里这等于训练线程直接自杀，需盯日志定位哪层网络崩了。 Critic1 的前向把 Actor 的输出当作输入，LatentLayer 做隐层桥接，若返回 false 同样强制移除专家。外汇与贵金属这类高杠杆品种下，这类自退出逻辑可能让策略在波动尖峰静默停摆，复盘时优先查 __LINE__ 报出的层。
PrintFormat("%s -> %d", __FUNCTION__, __LINE__); ExpertRemove(); break; } Result.Clear(); Result.Add(reward); if(!Critic1.backProp(Result, GetPointer(Actions), GetPointer(Gradient)) || !Critic2.backProp(Result, GetPointer(Actions), GetPointer(Gradient))) { PrintFormat("%s -> %d", __FUNCTION__, __LINE__); ExpertRemove(); break; } class=class="str">"cmt">//--- Policy study if(iter > class="num">0 && (iter % UpdatePolicy) == class="num">0) { tr = (class="type">int)((MathRand() / class="num">32767.0) * (total_tr - class="num">1)); i = (class="type">int)((MathRand() * MathRand() / MathPow(class="num">32767, class="num">2)) * (Buffer[tr].Total - class="num">2)); State.AssignArray(Buffer[tr].States[i].state); PrevBalance = Buffer[tr].States[MathMax(i - class="num">1, class="num">0)].account[class="num">0]; PrevEquity = Buffer[tr].States[MathMax(i - class="num">1, class="num">0)].account[class="num">1]; Account.Clear(); Account.Add((Buffer[tr].States[i].account[class="num">0] - PrevBalance) / PrevBalance); Account.Add(Buffer[tr].States[i].account[class="num">1] / PrevBalance); Account.Add((Buffer[tr].States[i].account[class="num">1] - PrevEquity) / PrevEquity); Account.Add(Buffer[tr].States[i].account[class="num">2]); Account.Add(Buffer[tr].States[i].account[class="num">3]); Account.Add(Buffer[tr].States[i].account[class="num">4] / PrevBalance); Account.Add(Buffer[tr].States[i].account[class="num">5] / PrevBalance); Account.Add(Buffer[tr].States[i].account[class="num">6] / PrevBalance); if(Account.GetIndex() >= class="num">0) Account.BufferWrite(); class=class="str">"cmt">//--- if(!Actor.feedForward(GetPointer(State), class="num">1, false, GetPointer(Account))) { PrintFormat("%s -> %d", __FUNCTION__, __LINE__); ExpertRemove(); break; } if(!Critic1.feedForward(GetPointer(Actor), LatentLayer, GetPointer(Actor))) { PrintFormat("%s -> %d", __FUNCTION__, __LINE__); ExpertRemove(); break; }
「目标网络同步与训练中断的兜底」
这段收尾逻辑跑在双评论家（Critic1/Critic2）的迭代训练循环末尾，核心做两件事：按节奏把在线网络的权重软更新到目标网络，以及在反向传播失败时安全退出。
UpdateTargets 控制同步频率，代码里用 iter % UpdateTargets == 0 触发 WeightsUpdate(..., Tau)，Tau 是软更新系数，值越小目标网络跟得越慢，训练可能更稳但收敛偏懒。
每过 500 毫秒（GetTickCount() - ticks > 500）就把两个 Critic 的近期平均误差刷到图表 Comment，格式是 Critic1 12.34% -> Error 0.00012345，你能直接盯着这个数字看是否还在降。
如果 backProp 或 backPropGradient 任一返回失败，立即切回训练模式、打印函数与行号、ExpertRemove() 并 break——EA 会自我卸载，不会在坏状态下硬跑。最后循环外再各打一行最终平均误差并移除 EA，整个离线训练会话干净收摊。
外汇与贵金属品种上跑这套强化学习训练，历史回测不代表实盘概率，杠杆品种风险高，参数没调明白前别直接挂真仓。
Critic1.getResults(Result); class="type">float forecast = Result[class="num">0]; Result.Update(class="num">0, (forecast > class="num">0 ? forecast + PoliticAdjust : PoliticAdjust)); Critic1.TrainMode(false); if(!Critic1.backProp(Result, GetPointer(Actor)) || !Actor.backPropGradient(GetPointer(Account), GetPointer(Gradient))) { Critic1.TrainMode(true); PrintFormat("%s -> %d", __FUNCTION__, __LINE__); ExpertRemove(); break; } Critic1.TrainMode(true); } class=class="str">"cmt">//--- Update Target Nets if(iter > class="num">0 && (iter % UpdateTargets) == class="num">0) { TargetActor.WeightsUpdate(GetPointer(Actor), Tau); TargetCritic1.WeightsUpdate(GetPointer(Critic1), Tau); TargetCritic2.WeightsUpdate(GetPointer(Critic2), Tau); } if(GetTickCount() - ticks > class="num">500) { class="type">class="kw">string str = StringFormat("%-15s %class="num">5.2f%% -> Error %class="num">15.8f\n", "Critic1", iter * class="num">100.0 / (class="type">class="kw">double)(Iterations), Critic1.getRecentAverageError()); str += StringFormat("%-15s %class="num">5.2f%% -> Error %class="num">15.8f\n", "Critic2", iter * class="num">100.0 / (class="type">class="kw">double)(Iterations), Critic2.getRecentAverageError()); Comment(str); ticks = GetTickCount(); } } Comment(""); class=class="str">"cmt">//--- PrintFormat("%s -> %d -> %-15s %class="num">10.7f", __FUNCTION__, __LINE__, "Critic1", Critic1.getRecentAverageError()); PrintFormat("%s -> %d -> %-15s %class="num">10.7f", __FUNCTION__, __LINE__, "Critic2", Critic2.getRecentAverageError()); ExpertRemove(); class=class="str">"cmt">//--- }