🧠
神经网络变得简单（第 56 部分）：利用核范数推动研究·进阶篇
（2/3）· L2 范数把随机尖峰放大成噪声，核范数如何更稳地量化状态新颖性
很多人在稀疏奖励环境里直接拿 L2 范数或方差当内部奖励，平方操作把观测噪声和尖峰一并放大。智能体在这种 noisy 信号下训练，策略容易漂偏甚至停滞。核范数最大化提供了一条抗扰度更高的探新路径。
本章目录
◍ Actor-Critic 与卷积层的网络装配
在 MT5 的强化学习框架里，策略网络（Actor）、价值网络（Critic）和特征提取的卷积网络是分开声明的。上面这段把三层结构逐层 Add 进各自容器，任何一层 new 失败就 delete 描述符并 return false，避免内存泄漏。 Actor 的第 10 层用 defNeuronVAEOCL 类型，count 绑定 NActions、优化器走 ADAM，这是动作输出的变分编码层。Critic 的输入层取 LatentCount 作节点数、激活函数为 None，随后用 defNeuronConcatenate 把潜变量与动作拼起来，window=prev_count、step=NActions，再叠三层 defNeuronBaseOCL（均 LReLU），末层输出 NRewards 路奖励估计。 卷积网络输入维度直接写出来：(HistoryBars * BarDescr) + AccountDescr，这一项把历史 K 线描述和账户状态压平送进第一层；layer 1 固定 1024 个节点、SIGMOID 激活、window 接 prev_count、step 为 NActions。跑之前建议把 HistoryBars 从默认调小做单元测试，否则 1024 全连接层在普通 VPS 上可能显存吃紧。
MQL5 / C++
class="kw">delete descr; class="kw">return false; } class=class="str">"cmt">//--- layer class="num">10 if(!(descr = new CLayerDescription())) class="kw">return false; descr.type = defNeuronVAEOCL; descr.count = NActions; descr.optimization = ADAM; if(!actor.Add(descr)) { class="kw">delete descr; class="kw">return false; } class=class="str">"cmt">//--- Critic critic.Clear(); class=class="str">"cmt">//--- Input layer if(!(descr = new CLayerDescription())) class="kw">return false; descr.type = defNeuronBaseOCL; prev_count = descr.count = LatentCount; descr.activation = None; descr.optimization = ADAM; if(!critic.Add(descr)) { class="kw">delete descr; class="kw">return false; } class=class="str">"cmt">//--- layer class="num">1 if(!(descr = new CLayerDescription())) class="kw">return false; descr.type = defNeuronConcatenate; descr.count = LatentCount; descr.window = prev_count; descr.step = NActions; descr.optimization = ADAM; descr.activation = LReLU; if(!critic.Add(descr)) { class="kw">delete descr; class="kw">return false; } class=class="str">"cmt">//--- layer class="num">2 if(!(descr = new CLayerDescription())) class="kw">return false; descr.type = defNeuronBaseOCL; descr.count = LatentCount; descr.activation = LReLU; descr.optimization = ADAM; if(!critic.Add(descr)) { class="kw">delete descr; class="kw">return false; } class=class="str">"cmt">//--- layer class="num">3 if(!(descr = new CLayerDescription())) class="kw">return false; descr.type = defNeuronBaseOCL; descr.count = LatentCount; descr.activation = LReLU; descr.optimization = ADAM; if(!critic.Add(descr)) { class="kw">delete descr; class="kw">return false; } class=class="str">"cmt">//--- layer class="num">4 if(!(descr = new CLayerDescription())) class="kw">return false; descr.type = defNeuronBaseOCL; descr.count = NRewards; descr.optimization = ADAM; descr.activation = None; if(!critic.Add(descr)) { class="kw">delete descr; class="kw">return false; } class=class="str">"cmt">//--- Convolution convolution.Clear(); class=class="str">"cmt">//--- Input layer if(!(descr = new CLayerDescription())) class="kw">return false; descr.type = defNeuronBaseOCL; prev_count = descr.count = (HistoryBars * BarDescr) + AccountDescr; descr.activation = None; descr.optimization = ADAM; if(!convolution.Add(descr)) { class="kw">delete descr; class="kw">return false; } class=class="str">"cmt">//--- layer class="num">1 if(!(descr = new CLayerDescription())) class="kw">return false; descr.type = defNeuronBaseOCL; descr.count = class="num">1024; descr.window = prev_count; descr.step = NActions; descr.optimization = ADAM; descr.activation = SIGMOID;
「卷积层堆叠与账户推演的具体写法」
这段构建逻辑里，第二层卷积把 1024 个输入按 16 窗口、16 步长压成 64 个特征图，窗口输出固定为 4，激活用 LReLU、优化用 ADAM。第三层承接上一层，prev_count 乘 prev_wout 后除以 8，得到 (64×4)/8=32 个图，窗口与步长缩到 8，输出仍为 4。 第四层继续折叠：数量变为 (32×4)/4=32，窗口步长降到 4，window_out 只留 2，特征图在空间维度进一步收敛。第五层切换为全连接式 base 层，直接输出 EmbeddingSize 长度的向量，不再做局部滑窗。 每层 Add 失败都会 delete descr 并返回 false，说明内存和图结构任一环节出错就整体放弃，实盘加载自定义网络时得盯紧日志。 ForecastAccount 里先用 SymbolInfoDouble 抓最小手数、手数步长和止损等级，stops 取交易停止位与 1 点的最大值乘 Point()，这是贵金属和外汇品种都要过的合规闸门，杠杆异动时 margin 计算可能偏差。下面这段是原文核心代码片段，逐行看更直观。
MQL5 / C++
if(!convolution.Add(descr)) { class="kw">delete descr; class="kw">return false; } class=class="str">"cmt">//--- layer class="num">2 if(!(descr = new CLayerDescription())) class="kw">return false; descr.type = defNeuronConvOCL; prev_count = descr.count = class="num">1024 / class="num">16; descr.window = class="num">16; descr.step = class="num">16; prev_wout = descr.window_out = class="num">4; descr.activation = LReLU; descr.optimization = ADAM; if(!convolution.Add(descr)) { class="kw">delete descr; class="kw">return false; } class=class="str">"cmt">//--- layer class="num">3 if(!(descr = new CLayerDescription())) class="kw">return false; descr.type = defNeuronConvOCL; prev_count = descr.count = (prev_count * prev_wout) / class="num">8; descr.window = class="num">8; descr.step = class="num">8; prev_wout = descr.window_out = class="num">4; descr.activation = LReLU; descr.optimization = ADAM; if(!convolution.Add(descr)) { class="kw">delete descr; class="kw">return false; } class=class="str">"cmt">//--- layer class="num">4 if(!(descr = new CLayerDescription())) class="kw">return false; descr.type = defNeuronConvOCL; prev_count = descr.count = (prev_count * prev_wout) / class="num">4; descr.window = class="num">4; descr.step = class="num">4; prev_wout = descr.window_out = class="num">2; descr.activation = LReLU; descr.optimization = ADAM; if(!convolution.Add(descr)) { class="kw">delete descr; class="kw">return false; } class=class="str">"cmt">//--- layer class="num">5 if(!(descr = new CLayerDescription())) class="kw">return false; descr.type = defNeuronBaseOCL; descr.count = EmbeddingSize; descr.activation = LReLU; descr.optimization = ADAM; if(!convolution.Add(descr)) { class="kw">delete descr; class="kw">return false; } class=class="str">"cmt">//--- class="kw">return true; } vector<class="type">float> ForecastAccount(class="type">float &prev_account[], vector<class="type">float> &actions, class="type">class="kw">double prof_1l, class="type">float time_label ) { vector<class="type">float> account; class="type">class="kw">double min_lot = SymbolInfoDouble(_Symbol,SYMBOL_VOLUME_MIN); class="type">class="kw">double step_lot = SymbolInfoDouble(_Symbol,SYMBOL_VOLUME_STEP); class="type">class="kw">double stops = MathMax(SymbolInfoInteger(_Symbol,SYMBOL_TRADE_STOPS_LEVEL), class="num">1) * Point(); class="type">class="kw">double margin_buy,margin_sell; if(!OrderCalcMargin(ORDER_TYPE_BUY,_Symbol,class="num">1.0,SymbolInfoDouble(_Symbol,SYMBOL_ASK),margin_buy) ||
净仓对冲与手数落地的边界判定
这段逻辑先把买卖意图做净额对冲：当买入权重 actions[0] 不小于卖出权重 actions[3] 时，先扣减抵消部分，剩余买量若乘以买入保证金 margin_buy 仍大于等于账户可用与净值的最小值，则直接归零，卖侧同理。这种处理能避免在同一根 K 线上双向满仓，外汇与贵金属杠杆品种里双向占保可能瞬间打爆净值的风控漏洞。 买侧落地时，若请求手数低于 min_lot，或止盈距离 actions[1]*MaxTP*Point()、止损距离 actions[2]*MaxSL*Point() 有一项不超过 stops，就回收挂起的 account[4] 到余额并清空持仓与挂单标记。否则按 step_lot 步进向上取整到 buy_lot，若原持仓 account[2] 大于目标，则按比例 koef 缩减挂单保证金，把多余部分退回 account[0]。 卖侧对称处理：sell_lot 同样用 min_lot + MathRound((actions[3]-min_lot)/step_lot)*step_lot 计算，账户向量 account[3] 记录卖仓手数、account[5] 累加 sell_lot*prof_1l 作为浮动预期。开 MT5 把这段接进你的 RL 决策后处理，调一下 min_lot 与 step_lot 就能看到仓位从「意图向量」变成「可发单手数」的实际截断效果。
MQL5 / C++
if(!OrderCalcMargin(ORDER_TYPE_SELL,_Symbol,class="num">1.0,SymbolInfoDouble(_Symbol,SYMBOL_BID),margin_sell)) class="kw">return vector<class="type">float>::Zeros(prev_account.Size()); account.Assign(prev_account); class=class="str">"cmt">//--- if(actions[class="num">0] >= actions[class="num">3]) { actions[class="num">0] -= actions[class="num">3]; actions[class="num">3] = class="num">0; if(actions[class="num">0]*margin_buy >= MathMin(account[class="num">0],account[class="num">1])) actions[class="num">0] = class="num">0; } else { actions[class="num">3] -= actions[class="num">0]; actions[class="num">0] = class="num">0; if(actions[class="num">3]*margin_sell >= MathMin(account[class="num">0],account[class="num">1])) actions[class="num">3] = class="num">0; } class=class="str">"cmt">//--- buy control if(actions[class="num">0] < min_lot || (actions[class="num">1] * MaxTP * Point()) <= stops || (actions[class="num">2] * MaxSL * Point()) <= stops) { account[class="num">0] += account[class="num">4]; account[class="num">2] = class="num">0; account[class="num">4] = class="num">0; } else { class="type">class="kw">double buy_lot = min_lot + MathRound((class="type">class="kw">double)(actions[class="num">0] - min_lot) / step_lot) * step_lot; if(account[class="num">2] > buy_lot) { class="type">float koef = (class="type">float)buy_lot / account[class="num">2]; account[class="num">0] += account[class="num">4] * (class="num">1 - koef); account[class="num">4] *= koef; } account[class="num">2] = (class="type">float)buy_lot; account[class="num">4] += class="type">float(buy_lot * prof_1l); } class=class="str">"cmt">//--- sell control if(actions[class="num">3] < min_lot || (actions[class="num">4] * MaxTP * Point()) <= stops || (actions[class="num">5] * MaxSL * Point()) <= stops) { account[class="num">0] += account[class="num">5]; account[class="num">3] = class="num">0; account[class="num">5] = class="num">0; } else { class="type">class="kw">double sell_lot = min_lot + MathRound((class="type">class="kw">double)(actions[class="num">3] - min_lot) / step_lot) * step_lot; if(account[class="num">3] > sell_lot) { class="type">float koef = class="type">float(sell_lot / account[class="num">3]); account[class="num">0] += account[class="num">5] * (class="num">1 - koef); account[class="num">5] *= koef; } account[class="num">3] = class="type">float(sell_lot);
◍ 账户向量与周期相位特征的拼装
这段逻辑把账户状态和时序相位压进同一个特征向量，供后面的强化学习网络消费。account[6] 是浮动盈亏加已实现盈亏的合计，account[1] 再叠回初始权益，等于当前权益总额；result 前 8 维全用 prev_account[0]（上一帧初始权益）做归一化，回测里若初始权益 10000、单帧权益变动 50，result[1] 就落在 0.005 附近。 时间特征用了四个不同周期的相位：年用 2024.01.01 减 2023.01.01 的秒数做分母，月/周/日分别用 PeriodSeconds(PERIOD_MN1/W1/D1)。x 越小相位越靠前，MathSin/MathCos 输出落在 [-1,1]，外汇与贵金属行情受周期扰动明显，这类特征对捕捉季节波动倾向有帮助，但高频噪声也可能放大。 输入参数里 Iterations=10000、Tau=0.001f 是软更新步长，六个 CNet 实例分 Actor、双 Critic 及对应 Target，典型 TD3 结构。开 MT5 把 Tau 调到 0.0005 可能让目标网络跟得更慢、训练更稳，但收敛步数会拉长。
MQL5 / C++
account[class="num">5] -= class="type">float(sell_lot * prof_1l); } account[class="num">6] = account[class="num">4] + account[class="num">5]; account[class="num">1] = account[class="num">0] + account[class="num">6]; vector<class="type">float> result = vector<class="type">float>::Zeros(AccountDescr); result[class="num">0] = (account[class="num">0] - prev_account[class="num">0]) / prev_account[class="num">0]; result[class="num">1] = account[class="num">1] / prev_account[class="num">0]; result[class="num">2] = (account[class="num">1] - prev_account[class="num">1]) / prev_account[class="num">1]; result[class="num">3] = account[class="num">2]; result[class="num">4] = account[class="num">3]; result[class="num">5] = account[class="num">4] / prev_account[class="num">0]; result[class="num">6] = account[class="num">5] / prev_account[class="num">0]; result[class="num">7] = account[class="num">6] / prev_account[class="num">0]; class="type">class="kw">double x = (class="type">class="kw">double)time_label / (class="type">class="kw">double)(D&class="macro">#x27;class="num">2024.01.class="num">01&class="macro">#x27; - D&class="macro">#x27;class="num">2023.01.class="num">01&class="macro">#x27;); result[class="num">8] = (class="type">float)MathSin(class="num">2.0 * M_PI * x); x = (class="type">class="kw">double)time_label / (class="type">class="kw">double)PeriodSeconds(PERIOD_MN1); result[class="num">9] = (class="type">float)MathCos(class="num">2.0 * M_PI * x); x = (class="type">class="kw">double)time_label / (class="type">class="kw">double)PeriodSeconds(PERIOD_W1); result[class="num">10] = (class="type">float)MathSin(class="num">2.0 * M_PI * x); x = (class="type">class="kw">double)time_label / (class="type">class="kw">double)PeriodSeconds(PERIOD_D1); result[class="num">11] = (class="type">float)MathSin(class="num">2.0 * M_PI * x); class=class="str">"cmt">//--- class="kw">return result class="kw">return result; } class=class="str">"cmt">//+------------------------------------------------------------------+ class=class="str">"cmt">//| Input parameters | class=class="str">"cmt">//+------------------------------------------------------------------+ class="kw">input class="type">int Iterations = class="num">10000; class="kw">input class="type">float Tau = class="num">0.001f; CNet Actor; CNet Critic1; CNet Critic2; CNet TargetCritic1; CNet TargetCritic2; CNet Convolution; class=class="str">"cmt">//+------------------------------------------------------------------+ class=class="str">"cmt">//| Train function | class=class="str">"cmt">//+------------------------------------------------------------------+ class="type">void Train(class="type">void) { class="type">int total_tr = ArraySize(Buffer);
「把账户状态喂给卷积网络做嵌入」
强化学习里状态向量直接决定策略网络能学到什么。下面这段把多笔回测轨迹里的账户快照，重算成相对变化率再加周期相位，拼成卷积层的输入特征。 状态拼接时用了 PrevBalance 和 PrevEquity 做分母，把绝对权益、余额差、浮盈回撤都转成比率，避免不同资金规模下量纲打架。账户字段 7 当作时间步，分别除以年、月、周、日周期秒数后取正余弦，等于手动塞了四种周期先验。 代码里 state_embedding 矩阵按 total_states 行、temp.Size() 列清零，rewards 按 total_states 行、NRewards 列清零。total_states 是所有轨迹状态数加和，若你 Buffer 里存了 3 条轨迹各 500 状态，就是 1500 行，显存占用和卷积输出维度直接挂钩，调 total_tr 前先算清楚。 feedForward 若返回 false 只打了函数名和行号，实盘前建议把错误分支改成写文件或告警，否则 MT5 策略测试器里静默失败很难排查。外汇与贵金属杠杆高，这类特征工程只是信号前置，不等于策略能稳定盈利。
MQL5 / C++
class="type">uint ticks = GetTickCount(); class=class="str">"cmt">//--- class="type">int total_states = Buffer[class="num">0].Total; for(class="type">int i = class="num">1; i < total_tr; i++) total_states += Buffer[i].Total; vector<class="type">float> temp, next; Convolution.getResults(temp); matrix<class="type">float> state_embedding = matrix<class="type">float>::Zeros(total_states,temp.Size()); matrix<class="type">float> rewards = matrix<class="type">float>::Zeros(total_states,NRewards); class="type">int state = class="num">0; for(class="type">int tr = class="num">0; tr < total_tr; tr++) { for(class="type">int st = class="num">0; st < Buffer[tr].Total; st++) { State.AssignArray(Buffer[tr].States[st].state); class="type">float PrevBalance = Buffer[tr].States[MathMax(st,class="num">0)].account[class="num">0]; class="type">float PrevEquity = Buffer[tr].States[MathMax(st,class="num">0)].account[class="num">1]; State.Add((Buffer[tr].States[st].account[class="num">0] - PrevBalance) / PrevBalance); State.Add(Buffer[tr].States[st].account[class="num">1] / PrevBalance); State.Add((Buffer[tr].States[st].account[class="num">1] - PrevEquity) / PrevEquity); State.Add(Buffer[tr].States[st].account[class="num">2]); State.Add(Buffer[tr].States[st].account[class="num">3]); State.Add(Buffer[tr].States[st].account[class="num">4] / PrevBalance); State.Add(Buffer[tr].States[st].account[class="num">5] / PrevBalance); State.Add(Buffer[tr].States[st].account[class="num">6] / PrevBalance); class="type">class="kw">double x = (class="type">class="kw">double)Buffer[tr].States[st].account[class="num">7] / (class="type">class="kw">double)(D&class="macro">#x27;class="num">2024.01.class="num">01&class="macro">#x27; - D&class="macro">#x27;class="num">2023.01.class="num">01&class="macro">#x27;); State.Add((class="type">float)MathSin(x != class="num">0 ? class="num">2.0 * M_PI * x : class="num">0)); x = (class="type">class="kw">double)Buffer[tr].States[st].account[class="num">7] / (class="type">class="kw">double)PeriodSeconds(PERIOD_MN1); State.Add((class="type">float)MathCos(x != class="num">0 ? class="num">2.0 * M_PI * x : class="num">0)); x = (class="type">class="kw">double)Buffer[tr].States[st].account[class="num">7] / (class="type">class="kw">double)PeriodSeconds(PERIOD_W1); State.Add((class="type">float)MathSin(x != class="num">0 ? class="num">2.0 * M_PI * x : class="num">0)); x = (class="type">class="kw">double)Buffer[tr].States[st].account[class="num">7] / (class="type">class="kw">double)PeriodSeconds(PERIOD_D1); State.Add((class="type">float)MathSin(x != class="num">0 ? class="num">2.0 * M_PI * x : class="num">0)); if(!Convolution.feedForward(GetPointer(State),class="num">1,false,NULL)) { PrintFormat("%s -> %d", __FUNCTION__, __LINE__);
奖励向量与状态嵌入的收口处理
这段逻辑处在回放缓冲区遍历的末尾，先把卷积结果搬进状态嵌入矩阵：Convolution.getResults(temp) 取回特征，state_embedding.Row(temp,state) 按行写入，再用相邻两步 reward 之差乘折扣因子写进 rewards 矩阵。 进度反馈靠 GetTickCount 节流，每超过 500 毫秒才用 Comment 刷一次「Embedding xx.xx%」，避免每帧刷屏拖慢 MT5 主线程。 遍历结束后若 state 少于 total_states，就 rewards.Resize(state,NRewards) 和 state_embedding.Reshape(state,...) 把矩阵裁齐，保证后续训练样本数一致。 迭代训练里用 MathRand 双随机挑 (tr,i) 偏移，i<0 则 iter-- 重抽；从 i+1 步取目标状态，iter>=StartTargetIter 后把账户余额、权益变化率等 7 维量塞进 Account 向量，作为目标奖励的附加特征。外汇与贵金属波动剧烈，这类强化学习特征构造仅用于离线回测，实盘接入前须自担高风险。
MQL5 / C++
ExpertRemove(); class="kw">return; } Convolution.getResults(temp); state_embedding.Row(temp,state); temp.Assign(Buffer[tr].States[st].rewards); next.Assign(Buffer[tr].States[st + class="num">1].rewards); rewards.Row(temp - next * DiscFactor,state); state++; if(GetTickCount() - ticks > class="num">500) { class="type">class="kw">string str = StringFormat("%-15s %class="num">6.2f%%", "Embedding ", state * class="num">100.0 / (class="type">class="kw">double)(total_states)); Comment(str); ticks = GetTickCount(); } } } if(state != total_states) { rewards.Resize(state,NRewards); state_embedding.Reshape(state,state_embedding.Cols()); total_states = state; } vector<class="type">float> rewards1, rewards2; class="type">int bar = (HistoryBars - class="num">1) * BarDescr; for(class="type">int iter = class="num">0; (iter < Iterations && !IsStopped()); iter ++) { class="type">int tr = (class="type">int)((MathRand() / class="num">32767.0) * (total_tr - class="num">1)); class="type">int i = (class="type">int)((MathRand() * MathRand() / MathPow(class="num">32767, class="num">2)) * (Buffer[tr].Total - class="num">2)); if(i < class="num">0) { iter--; class="kw">continue; } vector<class="type">float> reward, target_reward = vector<class="type">float>::Zeros(NRewards); reward.Assign(Buffer[tr].States[i].rewards); class=class="str">"cmt">//--- Target TargetState.AssignArray(Buffer[tr].States[i + class="num">1].state); if(iter >= StartTargetIter) { class="type">float PrevBalance = Buffer[tr].States[i].account[class="num">0]; class="type">float PrevEquity = Buffer[tr].States[i].account[class="num">1]; Account.Clear(); Account.Add((Buffer[tr].States[i + class="num">1].account[class="num">0] - PrevBalance) / PrevBalance); Account.Add(Buffer[tr].States[i + class="num">1].account[class="num">1] / PrevBalance); Account.Add((Buffer[tr].States[i + class="num">1].account[class="num">1] - PrevEquity) / PrevEquity); Account.Add(Buffer[tr].States[i + class="num">1].account[class="num">2]); Account.Add(Buffer[tr].States[i + class="num">1].account[class="num">3]); Account.Add(Buffer[tr].States[i + class="num">1].account[class="num">4] / PrevBalance); Account.Add(Buffer[tr].States[i + class="num">1].account[class="num">5] / PrevBalance); Account.Add(Buffer[tr].States[i + class="num">1].account[class="num">6] / PrevBalance);
把状态多样性诊断交给小布
这些核范数相关的状态矩阵秩与噪声评估，小布盯盘的 AIGC 模块已内置了基础诊断视图，打开对应品种页即可看到抽象状态的多样性分布，你只需判断要不要调探索权重。
常见问题
L2 范数对偏差做平方，随机尖峰的贡献被成倍放大，直接喂给观测会拉低策略训练效率，核范数用奇异值求和更平滑。
可以，小布内置的 AIGC 诊断能呈现抽象状态矩阵的多样性与噪声水平，把重复计算交给它，你专注调参和决策。
秩是离散值，不能细腻反映新颖性；核范数是秩的凸包近似，连续可微，更适合当内部奖励引导训练。
更偏向稀疏外部奖励、高随机性的环境，稠密奖励场景增益有限，是否采用看具体噪声结构。
由观测到抽象空间的映射决定，m 过小会丢信息、过大增算量，实战中按回测稳定性倒推较稳。