神经网络变得轻松（第四十四部分）：动态学习技能·进阶篇
（2/3）· DIAYN 教出的技能难以预测令规划头疼？DADS 如何在 2020 年破解可预测多样性难题
◍ 判别器网络的四层堆叠与结果校验
在 MT5 中用 OpenCL 后端搭判别器时，输入层维度由历史 K 线数与账户特征共同决定：prev_count = HistoryBars * BarDescr + AccountDescr，激活函数设为 None，优化器走 ADAM。 第一层接 BatchNorm，batch 写死 1000，用于抑制输入分布的偏移；第二、三层是标准全连接，神经元各 256 个，分别用 TANH 和 LReLU 做非线性，这两层是判别特征抽取的主体。 第四层输出维度压到 NSkills*AccountDescr，激活回 None，直接喂给 getResults 做后续调度。若 DiscriminatorResult.Size() 不等于 NSkills * AccountDescr，会打印「scope does not match」并返回 INIT_FAILED，这种尺寸不对的初始化在 EA 加载时就会崩。 调度器与 actor 的更新目标都用 MathMax(Iterations/100, 500000/Batch) 兜底，避免迭代步数过低时更新频率失控。外汇与贵金属杠杆高，这类 RL 模块若初始化失败，实盘前务必在策略测试器里先跑通尺寸校验。
class="kw">return false; } class=class="str">"cmt">//--- Discriminator discriminator.Clear(); class=class="str">"cmt">//--- Input layer if(!(descr = new CLayerDescription())) class="kw">return false; descr.type = defNeuronBaseOCL; prev_count = descr.count = (HistoryBars * BarDescr + AccountDescr); descr.window = class="num">0; descr.activation = None; descr.optimization = ADAM; if(!discriminator.Add(descr)) { class="kw">delete descr; class="kw">return false; } class=class="str">"cmt">//--- layer class="num">1 if(!(descr = new CLayerDescription())) class="kw">return false; descr.type = defNeuronBatchNormOCL; descr.count = prev_count; descr.batch = class="num">1000; descr.activation = None; descr.optimization = ADAM; if(!discriminator.Add(descr)) { class="kw">delete descr; class="kw">return false; } class=class="str">"cmt">//--- layer class="num">2 if(!(descr = new CLayerDescription())) class="kw">return false; descr.type = defNeuronBaseOCL; descr.count = class="num">256; descr.optimization = ADAM; descr.activation = TANH; if(!discriminator.Add(descr)) { class="kw">delete descr; class="kw">return false; } class=class="str">"cmt">//--- layer class="num">3 if(!(descr = new CLayerDescription())) class="kw">return false; descr.type = defNeuronBaseOCL; descr.count = class="num">256; descr.optimization = ADAM; descr.activation = LReLU; if(!discriminator.Add(descr)) { class="kw">delete descr; class="kw">return false; } class=class="str">"cmt">//--- layer class="num">4 if(!(descr = new CLayerDescription())) class="kw">return false; descr.type = defNeuronBaseOCL; descr.count = NSkills*AccountDescr; descr.optimization = ADAM; descr.activation = None; if(!discriminator.Add(descr)) { class="kw">delete descr; class="kw">return false; } Discriminator.getResults(DiscriminatorResult); if(DiscriminatorResult.Size() != NSkills * AccountDescr) { PrintFormat("The scope of the discriminator does not match the skills count(%d <> %d)", NSkills * AccountDescr, Result.Total()); class="kw">return INIT_FAILED; } Scheduler.getResults(SchedulerResult); Scheduler.SetUpdateTarget(MathMax(Iterations / class="num">100, class="num">500000 / SchedulerBatch)); if(SchedulerResult.Size() != NSkills) { PrintFormat("The scope of the scheduler does not match the skills count(%d <> %d)", NSkills, Result.Total()); class="kw">return INIT_FAILED; } Actor.getResults(ActorResult); Actor.SetUpdateTarget(MathMax(Iterations / class="num">100, class="num">500000 / AgentBatch * NSkills)); if(ActorResult.Size() != NActions) { PrintFormat("The scope of the actor does not match the actions count(%d <> %d)", NActions, Result.Total()); class="kw">return INIT_FAILED; }
「状态推演与智能体奖励的代码骨架」
强化学习里每一步都要由旧账户状态推出新状态。下面这段函数接收上一帧的 float 数组 prev_account、动作编号 action 以及单手收益率 prof_1l，返回下一帧状态向量。 case 0 与 case 1 分别对应加多、加空：在索引 5 或 6 上累加 SymbolInfoDouble(_Symbol, SYMBOL_VOLUME_MIN) 的最小交易量，再按 prof_1l 重算浮动盈亏（索引 7、8），最后刷新总权益（索引 4、1）。 case 2 是平仓动作，把未实现盈亏并入口袋（索引 0、1、2 对齐），并从索引 3 开始把持仓相关字段清零。case 3 是不动仓位只按最新 prof_1l 重估权益，常用于持仓观望步。 GetAgentReward 则把判别器向量 reshape 成 NSkills×AccountDescr 的矩阵，取第 skill 行作为预测；用 forecast[5]-prev_account[5] 与 forecast[6]-prev_account[6] 衡量买卖仓位变化，若两者均小于 0 说明模型倾向空仓。外汇与贵金属杠杆高，这套状态机只是概率推演，实盘需先在 MT5 策略测试器跑通。 让小布替你跑这套 把 SYMBOL_VOLUME_MIN 换成你经纪商实际最小手数，否则回测里加仓步长会和实盘错位，EURUSD 常见值为 0.01。
vector<class="type">float> GetNewState(class="type">float &prev_account[], class="type">int action, class="type">class="kw">double prof_1l) { vector<class="type">float> result; class=class="str">"cmt">//--- result.Assign(prev_account); class="kw">switch(action) { case class="num">0: result[class="num">5] += (class="type">float)SymbolInfoDouble(_Symbol, SYMBOL_VOLUME_MIN); result[class="num">7] += result[class="num">5] * (class="type">float)prof_1l; result[class="num">8] -= result[class="num">6] * (class="type">float)prof_1l; result[class="num">4] = result[class="num">7] + result[class="num">8]; result[class="num">1] = result[class="num">0] + result[class="num">4]; break; case class="num">1: result[class="num">6] += (class="type">float)SymbolInfoDouble(_Symbol, SYMBOL_VOLUME_MIN); result[class="num">7] += result[class="num">5] * (class="type">float)prof_1l; result[class="num">8] -= result[class="num">6] * (class="type">float)prof_1l; result[class="num">4] = result[class="num">7] + result[class="num">8]; result[class="num">1] = result[class="num">0] + result[class="num">4]; break; case class="num">2: result[class="num">0] += result[class="num">4]; result[class="num">1] = result[class="num">0]; result[class="num">2] = result[class="num">0]; for(class="type">int i = class="num">3; i < AccountDescr; i++) result[i] = class="num">0; break; case class="num">3: result[class="num">7] += result[class="num">5] * (class="type">float)prof_1l; result[class="num">8] -= result[class="num">6] * (class="type">float)prof_1l; result[class="num">4] = result[class="num">7] + result[class="num">8]; result[class="num">1] = result[class="num">0] + result[class="num">4]; break; } class=class="str">"cmt">//--- class="kw">return result class="kw">return result; } vector<class="type">float> GetAgentReward(class="type">int skill, vector<class="type">float> &discriminator, class="type">float &prev_account[]) { class=class="str">"cmt">//--- prepare matrix<class="type">float> discriminator_matrix; discriminator_matrix.Init(class="num">1, discriminator.Size()); discriminator_matrix.Row(discriminator,class="num">0); discriminator_matrix.Reshape(NSkills, AccountDescr); vector<class="type">float> forecast = discriminator_matrix.Row(skill); class=class="str">"cmt">//--- check action class="type">int action = class="num">3; class="type">float buy = forecast[class="num">5] - prev_account[class="num">5]; class="type">float sell = forecast[class="num">6] - prev_account[class="num">6]; if(buy < class="num">0 && sell < class="num">0)
奖励计算与三阶段训练循环
这段逻辑先把多空力量比较结果映射成离散动作：买压大于卖压置 action=0（倾向做多），卖压大于买压置 action=1（倾向做空），其余情况 action=2（观望）。随后用 forecast 与判别矩阵均值之比再取均值，套一层 MathLog(MathAbs(mean)) 作为该动作的奖励值，意味着预测偏离越大、对数奖励越高，但外汇与贵金属波动剧烈，这种奖励设计只是概率倾向而非确定性信号。 训练主函数 Train() 按 Iterations 轮迭代，每轮拆成 3 个 phase：phase 0 跑 DiscriminatorBatch、phase 1 跑 AgentBatch、phase 2 跑 SchedulerBatch，三个批次数彼此独立可调。内层 batch_iter 用 MathRand() 做样本抽取，其中 i 的计算用了 MathRand() 平方除以 32767 平方，使早期样本索引更偏向 0 附近，是一种非均匀回放策略。 实际在 MT5 里跑这套，建议先打印 total_tr 和各 phase 的 batch 值确认数据量；若 DiscriminatorBatch 设得过大，单轮耗时可能翻倍。贵金属品种点差跳变频繁，训练前务必用 IsStopped() 保底中断，避免卡死终端。
action = class="num">2; else if(buy > sell) action = class="num">0; else if(buy < sell) action = class="num">1; class=class="str">"cmt">//--- calculate reward vector<class="type">float> result = vector<class="type">float>::Zeros(NActions); class="type">float mean = (forecast / discriminator_matrix.Mean(class="num">0)).Mean(); result[action] = MathLog(MathAbs(mean)); class=class="str">"cmt">//--- class="kw">return result class="kw">return result; } class=class="str">"cmt">//+------------------------------------------------------------------+ class=class="str">"cmt">//| Train function | class=class="str">"cmt">//+------------------------------------------------------------------+ class="type">void Train(class="type">void) { class="type">int total_tr = ArraySize(Buffer); class="type">uint ticks = GetTickCount(); vector<class="type">float> account, reward; class="type">int bar, action; class="type">int skill, shift; class=class="str">"cmt">//--- for(class="type">int iter = class="num">0; (iter < Iterations && !IsStopped()); iter ++) { for(class="type">int phase = class="num">0; phase < class="num">3; phase++) { class="type">int batch = class="num">0; class="kw">switch(phase) { case class="num">0: batch = DiscriminatorBatch; break; case class="num">1: batch = AgentBatch; break; case class="num">2: batch = SchedulerBatch; break; class="kw">default: PrintFormat("Incorrect phase %d"); batch = class="num">0; break; } for(class="type">int batch_iter = class="num">0; batch_iter < batch; batch_iter++) { class="type">int tr = (class="type">int)(((class="type">class="kw">double)MathRand() / class="num">32767.0) * (total_tr - class="num">1)); class="type">int i = (class="type">int)((MathRand() * MathRand() / MathPow(class="num">32767, class="num">2)) * (Buffer[tr].Total - class="num">2)); State.AssignArray(Buffer[tr].States[i].state);
◍ 账户状态与单根盈亏的特征拼装
这段逻辑在做一件事：把上一根和当前根的账户快照，压成一组可喂给判别网络的标准化特征。外汇与贵金属杠杆高、滑点随机，这类比值特征对极端回撤的敏感度，往往比绝对金额更有参考价值。 前两句先用 MathMax(i-1,0) 兜底，避免 i=0 时越界取上一状态；随后连续 State.Add 把「余额增量/前余」「净值/前余」「净值波动/前净」等 10 个量依次入栈，分母统一用 PrevBalance 或 PrevEquity，相当于把规模效应消去。 bar = (HistoryBars-1)*BarDescr 定位到下一根收盘偏移，再用 SYMBOL_TRADE_TICK_VALUE_PROFIT 除以 SYMBOL_POINT，把价格跳动换算成该符号下的单跳盈亏金额 prof_1l，这一步直接绑定具体品种点值，换到 XAUUSD 和 EURUSD 数值会差出两位数。 循环里插了 IsStopped 的硬退出：脚本被终止就 PrintFormat 打行号并 ExpertRemove，防止半截特征写进 Buffer。phase==0 时走 Discriminator.feedForward 把指针喂入，失败同样移除并跳出，后面才初始化 SchedulerResult 为零向量。
class="type">float PrevBalance = Buffer[tr].States[MathMax(i - class="num">1, class="num">0)].account[class="num">0]; class="type">float PrevEquity = Buffer[tr].States[MathMax(i - class="num">1, class="num">0)].account[class="num">1]; State.Add((Buffer[tr].States[i].account[class="num">0] - PrevBalance) / PrevBalance); State.Add(Buffer[tr].States[i].account[class="num">1] / PrevBalance); State.Add((Buffer[tr].States[i].account[class="num">1] - PrevEquity) / PrevEquity); State.Add(Buffer[tr].States[i].account[class="num">2] / PrevBalance); State.Add(Buffer[tr].States[i].account[class="num">4] / PrevBalance); State.Add(Buffer[tr].States[i].account[class="num">5]); State.Add(Buffer[tr].States[i].account[class="num">6]); State.Add(Buffer[tr].States[i].account[class="num">7] / PrevBalance); State.Add(Buffer[tr].States[i].account[class="num">8] / PrevBalance); bar = (HistoryBars - class="num">1) * BarDescr; class="type">class="kw">double cl_op = Buffer[tr].States[i + class="num">1].state[bar]; class="type">class="kw">double prof_1l = SymbolInfoDouble(_Symbol, SYMBOL_TRADE_TICK_VALUE_PROFIT) * cl_op / SymbolInfoDouble(_Symbol, SYMBOL_POINT); PrevBalance = Buffer[tr].States[i].account[class="num">0]; PrevEquity = Buffer[tr].States[i].account[class="num">1]; if(IsStopped()) { PrintFormat("%s -> %d", __FUNCTION__, __LINE__); ExpertRemove(); break; } class="kw">switch(phase) { case class="num">0: if(!Discriminator.feedForward(GetPointer(State))) { PrintFormat("%s -> %d", __FUNCTION__, __LINE__); ExpertRemove(); break; } for(skill = class="num">0; skill < NSkills; skill++) { SchedulerResult = vector<class="type">float>::Zeros(NSkills);
「把动作塞进判别器前先算清账户偏移」
这段逻辑发生在策略跑完一次 actor 采样之后：先给当前 skill 标记 SchedulerResult 为 1，再把 State 数组和调度结果拼进 StateSkill，准备喂给神经网络。若终端被停止或 feedForward 返回失败，立刻打印函数名与行号并 break，避免半截状态污染后续回放缓冲区。 action 由 Actor.getSample() 抽出，紧接着用 GetNewState 基于上一根 buffer 的状态、动作和单级利润 prof_1l 算出新账户数组。shift 按 skill * AccountDescr 跳到判别器结果段的对应槽位，从这里开始连续写 9 个归一化指标：第 0 偏移是余额相对 PrevBalance 的收益率，第 1 偏移是权益占 PrevBalance 比，第 2 偏移是权益相对 PrevEquity 的变化率，后面依次铺开占用、回撤等账户特征。 若外层 Result 指针还为空，代码会 new 一个 CBufferFloat 来承接。外汇与贵金属这类高杠杆品种，账户归一化若 PrevBalance 取错周期，判别器学到的奖励信号可能完全偏移，建议直接在 MT5 里把 PrevBalance 打印出来核对每个 skill 切换时的基准值。
SchedulerResult[skill] = class="num">1; StateSkill.AssignArray(GetPointer(State)); StateSkill.AddArray(SchedulerResult); if(IsStopped()) { PrintFormat("%s -> %d", __FUNCTION__, __LINE__); break; } if(!Actor.feedForward(GetPointer(State), class="num">1, false)) { PrintFormat("%s -> %d", __FUNCTION__, __LINE__); break; } action = Actor.getSample(); account = GetNewState(Buffer[tr].States[i].account, action, prof_1l); shift = skill * AccountDescr; DiscriminatorResult[shift] = (account[class="num">0] - PrevBalance) / PrevBalance; DiscriminatorResult[shift + class="num">1] = account[class="num">1] / PrevBalance; DiscriminatorResult[shift + class="num">2] = (account[class="num">1] - PrevEquity) / PrevEquity; DiscriminatorResult[shift + class="num">3] = account[class="num">2] / PrevBalance; DiscriminatorResult[shift + class="num">4] = account[class="num">4] / PrevBalance; DiscriminatorResult[shift + class="num">5] = account[class="num">5]; DiscriminatorResult[shift + class="num">6] = account[class="num">6]; DiscriminatorResult[shift + class="num">7] = account[class="num">7] / PrevBalance; DiscriminatorResult[shift + class="num">8] = account[class="num">8] / PrevBalance; } if(!Result) { Result = new CBufferFloat(); if(!Result) {
判别器训练与推理的退出分支
这段逻辑处在 EA 主循环里，按 mode 切到 0 或 1：mode 0 负责反向传播训练，mode 1 负责前向推理并构造技能状态向量。
当缓冲区创建失败时，会打印 Error of create buffer %d 并带上 GetLastError() 返回码，随即 ExpertRemove() 退出，防止空缓冲区继续跑。训练阶段若 backProp(Result) 返回 false，同样打印函数名与行号后移除 EA。
推理分支里先 feedForward 拿判别器输出，再对 NSkills 个技能逐个置 1 生成 SchedulerResult，拼到 State 后面送进 StateSkill。循环体每次都查 IsStopped()，终端手动停止时会立刻 ExpertRemove() 断干净，不拖泥带水。
外汇与贵金属杠杆高、滑点跳空频繁，这类自研神经网络 EA 在实盘前务必用 MT5 策略测试器以 tick 级数据跑通退出逻辑，确认不会在断线重连时卡死。
PrintFormat("Error of create buffer %d", GetLastError()); ExpertRemove(); break; } } Result.AssignArray(DiscriminatorResult); if(!Discriminator.backProp(Result)) { PrintFormat("%s -> %d", __FUNCTION__, __LINE__); ExpertRemove(); break; } break; case class="num">1: if(!Discriminator.feedForward(GetPointer(State))) { PrintFormat("%s -> %d", __FUNCTION__, __LINE__); ExpertRemove(); break; } Discriminator.getResults(DiscriminatorResult); for(skill = class="num">0; skill < NSkills; skill++) { SchedulerResult = vector<class="type">float>::Zeros(NSkills); SchedulerResult[skill] = class="num">1; StateSkill.AssignArray(GetPointer(State)); StateSkill.AddArray(SchedulerResult); if(IsStopped()) { PrintFormat("%s -> %d", __FUNCTION__, __LINE__); ExpertRemove(); break; }