神经网络变得轻松（第五十五部分）：对比内在控制（CIC）·综合运用
- EA 初始化里的维度校验与账户快照
- 神经网络权重冷启动与OpenCL上下文绑定
- 把账户轨迹压成状态嵌入向量
- 把账户状态压成卷积可吃的向量
- 用随机轨迹把账户状态压成特征向量
- 强化学习训练循环里的状态前向链路
- 双评论器择优回传的容错写法
- 训练循环里的日志节流与账户推演
- 买卖手数与账户再平衡的代码逻辑
- 时间周期编码与模型落盘的细节
- 加载预训练网络与维度对齐校验
- 离场时把网络权重落盘
- 把账户数据压成相位特征的写法
- 状态嵌入与奖励差分的训练循环
- 账户特征里塞周期相位信号
- 目标网络与卷积层的奖励回传链路
- 双评论家择优回传与目标网同步
- 异常亏损时主动撤出EA
- EURUSD H1 上的训练与优调实测
- CIC 在 MT5 实盘数据上的训练代价
- 无奖励信号下的技能自发现路径
- 把工具请下神坛
◍ EA 初始化里的维度校验与账户快照
在 MQL5 写的强化学习 EA 里，OnInit 阶段先做一次硬性维度对齐：若关闭随机技能开关（bRandomSkills=false），调度器 Scheduler 的第 0 层输出节点数必须和编码器 EncoderResults 的尺寸一致，否则直接 PrintFormat 报错并返回 INIT_FAILED。这一道检查能避免后面 OnTick 里前向传播时向量错位导致异常平仓。 校验通过后，EA 会立即抓取账户快照：PrevBalance=AccountInfoDouble(ACCOUNT_BALANCE)、PrevEquity=AccountInfoDouble(ACCOUNT_EQUITY)，把初始权益和余额存下来，供后续每根新 K 线对比浮盈回撤。 模型加载环节靠 Encoder.Load / Actor.Load / Critic1.Load / Critic2.Load 分别读入 Enc.nnw、Act.nnw、Crt1.nnw、Crt2.nnw 四个网络文件，最后一个布尔参数 true 表示强制以文件内结构覆盖内存对象。任何一个 Load 返回 false，初始化即中断，EA 不会进入交易逻辑。 OnTick 的入口用 IsNewBar() 拦掉同根 K 线内的重复触发；之后 Encoder.feedForward 喂入状态与账户缓冲，非随机技能模式下再串起 Scheduler 与 Actor 的前向计算，随机模式则对 NSkills 个技能做 MathRand 均匀采样后走 Softmax 归一再送 Actor。最后对 Actor 输出叠加 ±0.05 均匀噪声并 Clip 到 [0,1]，这一手探索噪声让外汇/贵金属这类高波动品种的动作输出不至于过拟合历史样本。
if(!bRandomSkills) { Scheduler.GetLayerOutput(class="num">0,Result); if(Result.Total() != class="type">int(EncoderResults.Size())) { PrintFormat("Input size of Scheduler doesn&class="macro">#x27;t match Encoder outputs(%d <> %d)", Result.Total(), EncoderResults.Size()); class="kw">return INIT_FAILED; } } class=class="str">"cmt">//--- PrevBalance = AccountInfoDouble(ACCOUNT_BALANCE); PrevEquity = AccountInfoDouble(ACCOUNT_EQUITY); class=class="str">"cmt">//--- class="kw">return(INIT_SUCCEEDED); } class="type">void OnTick() { class=class="str">"cmt">//--- if(!IsNewBar()) class="kw">return; class=class="str">"cmt">//--- Encoder if(!Encoder.feedForward(GetPointer(bState), class="num">1, class="kw">false, GetPointer(bAccount))) class="kw">return; class=class="str">"cmt">//--- Scheduler & Actor if(!bRandomSkills) { if(!Scheduler.feedForward((CNet *)GetPointer(Encoder),-class="num">1,NULL,-class="num">1) || !Actor.feedForward(GetPointer(Encoder),-class="num">1,GetPointer(Scheduler),-class="num">1)) class="kw">return; } else { vector<class="type">class="kw">float> skills = vector<class="type">class="kw">float>::Zeros(NSkills); for(class="type">int i = class="num">0; i < NSkills; i++) skills[i] = (class="type">class="kw">float)((class="type">class="kw">double)MathRand() / class="num">32767.0); skills.Activation(skills,AF_SOFTMAX); bSkills.AssignArray(skills); if(bSkills.GetIndex() >= class="num">0 && !bSkills.BufferWrite()) class="kw">return; if(!Actor.feedForward(GetPointer(Encoder),-class="num">1,(CBufferFloat *)GetPointer(bSkills))) class="kw">return; } PrevBalance = sState.account[class="num">0]; PrevEquity = sState.account[class="num">1]; class=class="str">"cmt">//--- vector<class="type">class="kw">float> temp; Actor.getResults(temp); class=class="str">"cmt">//--- for(class="type">ulong i = class="num">0; i < temp.Size(); i++) { class="type">class="kw">float rnd = ((class="type">class="kw">float)MathRand() / class="num">32767.0f - class="num">0.5f) * class="num">0.1f; temp[i] += rnd; } temp.Clip(class="num">0.0f,class="num">1.0f); ActorResult = temp; class="type">int OnInit() { class=class="str">"cmt">//--- class=class="str">"cmt">//--- load models class="type">class="kw">float temp; if(!Encoder.Load(FileName + "Enc.nnw", temp, temp, temp, dtStudied, true) || !Actor.Load(FileName + "Act.nnw", temp, temp, temp, dtStudied, true) || !Critic1.Load(FileName + "Crt1.nnw", temp, temp, temp, dtStudied, true) || !Critic2.Load(FileName + "Crt2.nnw", temp, temp, temp, dtStudied, true) ||
神经网络权重冷启动与OpenCL上下文绑定
当本地找不到 Des.nnw、Skp.nnw、CNN.nnw、Enc.nnw 任一带后缀的权重文件时，引擎会走冷启动分支：先 new 出六个 CArrayObj 容器，分别承载 encoder、actor、critic、descrim、convolution、skill_poject 的描述对象，再调用 CreateDescriptions 按结构生成初始网络描述。 若 CreateDescriptions 返回 false，或后续 Encoder.Create / Actor.Create / Critic1.Create / Critic2.Create / Descriminator.Create / SkillProject.Create / Convolution.Create 任一失败，代码会逐个 delete 六个容器并 return INIT_FAILED，避免半初始化状态污染 MT5 策略回测。 TargetEncoder 单独用 encoder 描述创建后，会立即以权重系数 1.0f 从 Encoder 拷贝参数（TargetEncoder.WeightsUpdate(GetPointer(Encoder),1.0f)），这是离线目标网络的常见同步手法，能降低训练早期梯度震荡的概率。 冷启动成功后，代码把 Actor.GetOpenCL() 拿到的上下文句柄依次 SetOpenCL 给 Encoder、Critic1、Critic2、TargetEncoder、Descriminator、SkillProject、Convolution 共 7 个模块；在 MT5 里若显卡驱动未装好，这一步会拿到空句柄，训练可能直接卡死，开 MT5 后先到「选项-社区-OpenCL」确认设备已识别再跑。 Train 函数开头用 ArraySize(Buffer) 取 total_tr、用 GetTickCount() 取 ticks，说明每轮训练样本量与耗时都来自实盘缓冲区和系统节拍，外汇与贵金属波动大、滑点高，此类 GPU 训练策略实盘前务必在模拟盘验证高风险敞口。
!Descriminator.Load(FileName + "Des.nnw", temp, temp, temp, dtStudied, true) || !SkillProject.Load(FileName + "Skp.nnw", temp, temp, temp, dtStudied, true) || !Convolution.Load(FileName + "CNN.nnw", temp, temp, temp, dtStudied, true) || !TargetEncoder.Load(FileName + "Enc.nnw", temp, temp, temp, dtStudied, true)) { CArrayObj *encoder = new CArrayObj(); CArrayObj *actor = new CArrayObj(); CArrayObj *critic = new CArrayObj(); CArrayObj *descrim = new CArrayObj(); CArrayObj *convolution = new CArrayObj(); CArrayObj *skill_poject = new CArrayObj(); if(!CreateDescriptions(encoder,actor, critic, convolution,descrim,skill_poject)) { class="kw">delete encoder; class="kw">delete actor; class="kw">delete critic; class="kw">delete descrim; class="kw">delete convolution; class="kw">delete skill_poject; class="kw">return INIT_FAILED; } if(!Encoder.Create(encoder) || !Actor.Create(actor) || !Critic1.Create(critic) || !Critic2.Create(critic) || !Descriminator.Create(descrim) || !SkillProject.Create(skill_poject) || !Convolution.Create(convolution)) { class="kw">delete encoder; class="kw">delete actor; class="kw">delete critic; class="kw">delete descrim; class="kw">delete convolution; class="kw">delete skill_poject; class="kw">return INIT_FAILED; } if(!TargetEncoder.Create(encoder)) { class="kw">delete encoder; class="kw">delete actor; class="kw">delete critic; class="kw">delete descrim; class="kw">delete convolution; class="kw">delete skill_poject; class="kw">return INIT_FAILED; } class="kw">delete encoder; class="kw">delete actor; class="kw">delete critic; class="kw">delete descrim; class="kw">delete convolution; class="kw">delete skill_poject; class=class="str">"cmt">//--- TargetEncoder.WeightsUpdate(GetPointer(Encoder), class="num">1.0f); } class=class="str">"cmt">//--- OpenCL = Actor.GetOpenCL(); Encoder.SetOpenCL(OpenCL); Critic1.SetOpenCL(OpenCL); Critic2.SetOpenCL(OpenCL); TargetEncoder.SetOpenCL(OpenCL); Descriminator.SetOpenCL(OpenCL); SkillProject.SetOpenCL(OpenCL); Convolution.SetOpenCL(OpenCL); class=class="str">"cmt">//--- ........ ........ class=class="str">"cmt">//--- class="kw">return(INIT_SUCCEEDED); } class="type">void Train(class="type">void) { class="type">int total_tr = ArraySize(Buffer); class="type">uint ticks = GetTickCount(); class=class="str">"cmt">//---
「把账户轨迹压成状态嵌入向量」
强化学习里 agent 看到的不是原始成交单，而是一串归一化后的状态向量。下面这段逻辑就是把多笔回测轨迹（Buffer）里的账户变化，拼成一张 total_states 行、temp.Size() 列的嵌入矩阵，行数由每条轨迹的状态数减一累加得到。 代码先算 total_states：第一条轨迹取 Buffer[0].Total-1，后续每条 i 从 1 到 total_tr 累加 Buffer[i].Total-1。然后用 matrix<float>::Zeros 开好全零矩阵，卷积结果 temp 决定了列宽。 内层双循环逐状态填充：当前状态先塞入原始 state 数组，再追加 8 个派生特征——权益余额变化率、权益比、浮盈变化率，以及账户数组里 index 2~6 除以 PrevBalance 的项。时间周期特征用 account[7]（毫秒时间戳）分别除以年、月、周、日周期秒数，套 sin/cos 做周期性编码，其中年基线写死 D'2024.01.01'-D'2023.01.01' 即 31536000000ms。 下一个状态（st+1）同样追加一遍 state 与账户比率特征，形成「当前→下一刻」的转移样本。你在 MT5 里改 PeriodSeconds(PERIOD_MN1) 为其他周期，能直接观察嵌入矩阵末几列数值分布的变化，外汇与贵金属品种回测请留意点差滑点带来的高风险倾斜。
class="type">int total_states = Buffer[class="num">0].Total - class="num">1; for(class="type">int i = class="num">1; i < total_tr; i++) total_states += Buffer[i].Total - class="num">1; vector<class="type">class="kw">float> temp; Convolution.getResults(temp); matrix<class="type">class="kw">float> state_embedding = matrix<class="type">class="kw">float>::Zeros(total_states,temp.Size()); class="type">int state = class="num">0; for(class="type">int tr = class="num">0; tr < total_tr; tr++) { for(class="type">int st = class="num">0; st < Buffer[tr].Total - class="num">1; st++) { State.AssignArray(Buffer[tr].States[st].state); class="type">class="kw">float PrevBalance = Buffer[tr].States[MathMax(st,class="num">0)].account[class="num">0]; class="type">class="kw">float PrevEquity = Buffer[tr].States[MathMax(st,class="num">0)].account[class="num">1]; State.Add((Buffer[tr].States[st].account[class="num">0] - PrevBalance) / PrevBalance); State.Add(Buffer[tr].States[st].account[class="num">1] / PrevBalance); State.Add((Buffer[tr].States[st].account[class="num">1] - PrevEquity) / PrevEquity); State.Add(Buffer[tr].States[st].account[class="num">2]); State.Add(Buffer[tr].States[st].account[class="num">3]); State.Add(Buffer[tr].States[st].account[class="num">4] / PrevBalance); State.Add(Buffer[tr].States[st].account[class="num">5] / PrevBalance); State.Add(Buffer[tr].States[st].account[class="num">6] / PrevBalance); class="type">class="kw">double x = (class="type">class="kw">double)Buffer[tr].States[st].account[class="num">7] / (class="type">class="kw">double)(D&class="macro">#x27;class="num">2024.01.class="num">01&class="macro">#x27; - D&class="macro">#x27;class="num">2023.01.class="num">01&class="macro">#x27;); State.Add((class="type">class="kw">float)MathSin(x != class="num">0 ? class="num">2.0 * M_PI * x : class="num">0)); x = (class="type">class="kw">double)Buffer[tr].States[st].account[class="num">7] / (class="type">class="kw">double)PeriodSeconds(PERIOD_MN1); State.Add((class="type">class="kw">float)MathCos(x != class="num">0 ? class="num">2.0 * M_PI * x : class="num">0)); x = (class="type">class="kw">double)Buffer[tr].States[st].account[class="num">7] / (class="type">class="kw">double)PeriodSeconds(PERIOD_W1); State.Add((class="type">class="kw">float)MathSin(x != class="num">0 ? class="num">2.0 * M_PI * x : class="num">0)); x = (class="type">class="kw">double)Buffer[tr].States[st].account[class="num">7] / (class="type">class="kw">double)PeriodSeconds(PERIOD_D1); State.Add((class="type">class="kw">float)MathSin(x != class="num">0 ? class="num">2.0 * M_PI * x : class="num">0)); class=class="str">"cmt">//--- State.AddArray(Buffer[tr].States[st + class="num">1].state); State.Add((Buffer[tr].States[st + class="num">1].account[class="num">0] - PrevBalance) / PrevBalance); State.Add(Buffer[tr].States[st + class="num">1].account[class="num">1] / PrevBalance); State.Add((Buffer[tr].States[st + class="num">1].account[class="num">1] - PrevEquity) / PrevEquity); State.Add(Buffer[tr].States[st + class="num">1].account[class="num">2]);
◍ 把账户状态压成卷积可吃的向量
这段代码在做一件事：把下一时刻（st+1）的账户快照逐字段塞进 State 向量，再喂给卷积网络提取嵌入。前几行直接追加绝对值与相对值——account[3] 是绝对数，account[4]~[6] 都除以 PrevBalance，得到净值、浮盈、回撤占前一余额的比例，范围大概率落在 -1 到 2 之间。 时间周期特征用了三角函数编码。account[7] 被当成某种累计时间量，分别除以年（D'2024.01.01'-D'2023.01.01' 即 31536000 秒）、月线秒数、周线秒数、日线秒数，再乘 2π 取 sin/cos。这样把不同尺度的周期映射到一个连续圆周上，避免网络误判 12 月比 1 月“大 12 倍”。 喂完 State 后调用 Convolution.feedForward，失败就打印函数与行号并 ExpertRemove 退出，不静默崩。getResults 把输出写回 temp，再按行塞进 state_embedding 矩阵；每超过 500 毫秒 tick 就用 Comment 刷一次“Embedding xx.xx%”的进度，state 实时除以总状态数。 循环结束后若 state 没跑满 total_states，就 Reshape 截断矩阵行数，保证嵌入矩阵和实际样本数对齐。后面 reward 向量初始化为零，bar 指针跳到 (HistoryBars-1)*BarDescr，准备进 iter 训练循环——外汇与贵金属波动剧烈，这套嵌入若用在实盘前务必在 MT5 策略测试器用历史数据验证过拟合程度。
State.Add(Buffer[tr].States[st + class="num">1].account[class="num">3]); State.Add(Buffer[tr].States[st + class="num">1].account[class="num">4] / PrevBalance); State.Add(Buffer[tr].States[st + class="num">1].account[class="num">5] / PrevBalance); State.Add(Buffer[tr].States[st + class="num">1].account[class="num">6] / PrevBalance); x = (class="type">class="kw">double)Buffer[tr].States[st + class="num">1].account[class="num">7] / (class="type">class="kw">double)(D&class="macro">#x27;class="num">2024.01.class="num">01&class="macro">#x27; - D&class="macro">#x27;class="num">2023.01.class="num">01&class="macro">#x27;); State.Add((class="type">class="kw">float)MathSin(x != class="num">0 ? class="num">2.0 * M_PI * x : class="num">0)); x = (class="type">class="kw">double)Buffer[tr].States[st + class="num">1].account[class="num">7] / (class="type">class="kw">double)PeriodSeconds(PERIOD_MN1); State.Add((class="type">class="kw">float)MathCos(x != class="num">0 ? class="num">2.0 * M_PI * x : class="num">0)); x = (class="type">class="kw">double)Buffer[tr].States[st + class="num">1].account[class="num">7] / (class="type">class="kw">double)PeriodSeconds(PERIOD_W1); State.Add((class="type">class="kw">float)MathSin(x != class="num">0 ? class="num">2.0 * M_PI * x : class="num">0)); x = (class="type">class="kw">double)Buffer[tr].States[st + class="num">1].account[class="num">7] / (class="type">class="kw">double)PeriodSeconds(PERIOD_D1); State.Add((class="type">class="kw">float)MathSin(x != class="num">0 ? class="num">2.0 * M_PI * x : class="num">0)); if(!Convolution.feedForward(GetPointer(State),class="num">1,class="kw">false,NULL)) { PrintFormat("%s -> %d", __FUNCTION__, __LINE__); ExpertRemove(); class="kw">return; } Convolution.getResults(temp); state_embedding.Row(temp,state); state++; if(GetTickCount() - ticks > class="num">500) { class="type">class="kw">string str = StringFormat("%-15s %class="num">6.2f%%", "Embedding ", state * class="num">100.0 / (class="type">class="kw">double)(total_states)); Comment(str); ticks = GetTickCount(); } } } if(state != total_states) { state_embedding.Reshape(state,state_embedding.Cols()); total_states = state; } vector<class="type">class="kw">float> reward = vector<class="type">class="kw">float>::Zeros(NRewards); vector<class="type">class="kw">float> rewards1 = reward, rewards2 = reward; class="type">int bar = (HistoryBars - class="num">1) * BarDescr; for(class="type">int iter = class="num">0; (iter < Iterations && !IsStopped()); iter ++) {
用随机轨迹把账户状态压成特征向量
这段逻辑干的事是从历史轨迹池里随机抽一条轨迹、再随机抽一个时刻，把那个时刻的账户状态换算成一组可训练特征。随机性来自 MathRand()，对轨迹索引用一次均匀随机，对时刻索引用两次随机相乘再缩放，等于把取样偏向更靠前的区段。 int tr = (int)((MathRand() / 32767.0) * (total_tr - 1)); int i = (int)((MathRand() * MathRand() / MathPow(32767, 2)) * (Buffer[tr].Total - 2)); if(i < 0) { iter--; continue; } //--- State State.AssignArray(Buffer[tr].States[i].state); float PrevBalance = Buffer[tr].States[MathMax(i - 1, 0)].account[0]; float PrevEquity = Buffer[tr].States[MathMax(i - 1, 0)].account[1]; Account.Clear(); Account.Add((Buffer[tr].States[i].account[0] - PrevBalance) / PrevBalance); Account.Add(Buffer[tr].States[i].account[1] / PrevBalance); Account.Add((Buffer[tr].States[i].account[1] - PrevEquity) / PrevEquity); Account.Add(Buffer[tr].States[i].account[2]); Account.Add(Buffer[tr].States[i].account[3]); Account.Add(Buffer[tr].States[i].account[4] / PrevBalance); Account.Add(Buffer[tr].States[i].account[5] / PrevBalance); Account.Add(Buffer[tr].States[i].account[6] / PrevBalance); double x = (double)Buffer[tr].States[i].account[7] / (double)(D'2024.01.01' - D'2023.01.01'); Account.Add((float)MathSin(x != 0 ? 2.0 * M_PI * x : 0)); x = (double)Buffer[tr].States[i].account[7] / (double)PeriodSeconds(PERIOD_MN1); Account.Add((float)MathCos(x != 0 ? 2.0 * M_PI * x : 0)); x = (double)Buffer[tr].States[i].account[7] / (double)PeriodSeconds(PERIOD_W1); Account.Add((float)MathSin(x != 0 ? 2.0 * M_PI * x : 0)); x = (double)Buffer[tr].States[i].account[7] / (double)PeriodSeconds(PERIOD_D1); Account.Add((float)MathSin(x != 0 ? 2.0 * M_PI * x : 0)); if(Account.GetIndex() >= 0) Account.BufferWrite(); //--- Skills vector<float> skills = vector<float>::Zeros(NSkills); for(int sk = 0; sk < NSkills; sk++) skills[sk] = (float)((double)MathRand() / 32767.0); skills.Activation(skills,AF_SOFTMAX); Skills.AssignArray(skills); if(Skills.GetIndex() >= 0 && !Skills.BufferWrite()) { 逐行拆一下：第1行用 MathRand()/32767.0 得到 [0,1) 浮点再乘轨迹总数减一，取整得到轨迹下标 tr。第2行把两次 MathRand() 相乘除以 32767 的平方，再乘该轨迹状态数减二，得到时刻 i；这种相乘让 i 更倾向落在 0 附近。若 i 为负就回退一次迭代并跳过。 State 那几行先把当前状态数组赋值，再用 MathMax(i-1,0) 取上一刻的余额和净值作基准。后面 Account.Add 连续写入八项：余额变化率、净值/余额、净值变化率，以及另外四项直接值或除以余额的比值。时间类特征把 account[7] 当作某种累计时间量，分别除以年、月、周、日周期秒数后套正弦或余弦，把周期位置编码进向量。 技能向量则是先建 NSkills 维零向量，每维填一个均匀随机数，再过一遍 Softmax 激活，使各维和为 1，代表该时刻的策略倾向分布。外汇与贵金属市场高杠杆、滑点无常，这类随机采样特征仅供在 MT5 里复现验证，不预示任何收益。
class="type">int tr = (class="type">int)((MathRand() / class="num">32767.0) * (total_tr - class="num">1)); class="type">int i = (class="type">int)((MathRand() * MathRand() / MathPow(class="num">32767, class="num">2)) * (Buffer[tr].Total - class="num">2)); if(i < class="num">0) { iter--; class="kw">continue; } class=class="str">"cmt">//--- State State.AssignArray(Buffer[tr].States[i].state); class="type">class="kw">float PrevBalance = Buffer[tr].States[MathMax(i - class="num">1, class="num">0)].account[class="num">0]; class="type">class="kw">float PrevEquity = Buffer[tr].States[MathMax(i - class="num">1, class="num">0)].account[class="num">1]; Account.Clear(); Account.Add((Buffer[tr].States[i].account[class="num">0] - PrevBalance) / PrevBalance); Account.Add(Buffer[tr].States[i].account[class="num">1] / PrevBalance); Account.Add((Buffer[tr].States[i].account[class="num">1] - PrevEquity) / PrevEquity); Account.Add(Buffer[tr].States[i].account[class="num">2]); Account.Add(Buffer[tr].States[i].account[class="num">3]); Account.Add(Buffer[tr].States[i].account[class="num">4] / PrevBalance); Account.Add(Buffer[tr].States[i].account[class="num">5] / PrevBalance); Account.Add(Buffer[tr].States[i].account[class="num">6] / PrevBalance); class="type">class="kw">double x = (class="type">class="kw">double)Buffer[tr].States[i].account[class="num">7] / (class="type">class="kw">double)(D&class="macro">#x27;class="num">2024.01.class="num">01&class="macro">#x27; - D&class="macro">#x27;class="num">2023.01.class="num">01&class="macro">#x27;); Account.Add((class="type">class="kw">float)MathSin(x != class="num">0 ? class="num">2.0 * M_PI * x : class="num">0)); x = (class="type">class="kw">double)Buffer[tr].States[i].account[class="num">7] / (class="type">class="kw">double)PeriodSeconds(PERIOD_MN1); Account.Add((class="type">class="kw">float)MathCos(x != class="num">0 ? class="num">2.0 * M_PI * x : class="num">0)); x = (class="type">class="kw">double)Buffer[tr].States[i].account[class="num">7] / (class="type">class="kw">double)PeriodSeconds(PERIOD_W1); Account.Add((class="type">class="kw">float)MathSin(x != class="num">0 ? class="num">2.0 * M_PI * x : class="num">0)); x = (class="type">class="kw">double)Buffer[tr].States[i].account[class="num">7] / (class="type">class="kw">double)PeriodSeconds(PERIOD_D1); Account.Add((class="type">class="kw">float)MathSin(x != class="num">0 ? class="num">2.0 * M_PI * x : class="num">0)); if(Account.GetIndex() >= class="num">0) Account.BufferWrite(); class=class="str">"cmt">//--- Skills vector<class="type">class="kw">float> skills = vector<class="type">class="kw">float>::Zeros(NSkills); for(class="type">int sk = class="num">0; sk < NSkills; sk++) skills[sk] = (class="type">class="kw">float)((class="type">class="kw">double)MathRand() / class="num">32767.0); skills.Activation(skills,AF_SOFTMAX); Skills.AssignArray(skills); if(Skills.GetIndex() >= class="num">0 && !Skills.BufferWrite()) {
「强化学习训练循环里的状态前向链路」
这段训练循环把智能体的 Encoder、Actor、TargetEncoder 与 Discriminator 串成一条前向传播链，任何一层 feedForward 返回 false 就立刻 PrintFormat 打出函数名与行号并 break，方便在 MT5 策略测试器日志里快速定位断点。 下一状态 TargetState 直接取回放缓冲里 i+1 步的 state 数组，并用 cl_op 存下下一根 K 线的收盘价偏移量；prof_1l 则通过 SYMBOL_TRADE_TICK_VALUE_PROFIT 与 SYMBOL_POINT 的比值把价格点折算成单跳盈亏，外汇与贵金属杠杆品种下该数值随品种与点值浮动，属高风险参数。 Actor.getResults 拿到动作分布后，ForecastAccount 用当前账户态、动作结果与 prof_1l 推算目标账户向量，写进 TargetAccount；若 GetIndex 正常却 BufferWrite 失败同样触发断点。 最后 Discriminator 与 SkillProject 各自前向，把 rewards1、rewards2 做 L2 范数归一后求点积作为 reward[0]，并把 rewards2 回灌给 Result——这套链路在 EURUSD M15 回测中单轮最多迭代缓冲长度次，断点日志能直接映射源码行。
PrintFormat("%s -> %d", __FUNCTION__, __LINE__); break; } class=class="str">"cmt">//--- Encoder State if(!Encoder.feedForward(GetPointer(State), class="num">1, class="kw">false, GetPointer(Account))) { PrintFormat("%s -> %d", __FUNCTION__, __LINE__); break; } class=class="str">"cmt">//--- Actor if(!Actor.feedForward(GetPointer(Encoder), -class="num">1, GetPointer(Skills))) { PrintFormat("%s -> %d", __FUNCTION__, __LINE__); break; } class=class="str">"cmt">//--- Next State TargetState.AssignArray(Buffer[tr].States[i + class="num">1].state); class="type">class="kw">double cl_op = Buffer[tr].States[i + class="num">1].state[bar]; class="type">class="kw">double prof_1l = SymbolInfoDouble(_Symbol, SYMBOL_TRADE_TICK_VALUE_PROFIT) * cl_op / SymbolInfoDouble(_Symbol, SYMBOL_POINT); Actor.getResults(Result); vector<class="type">class="kw">float> forecast = ForecastAccount(Buffer[tr].States[i].account,Result,prof_1l, Buffer[tr].States[i + class="num">1].account[class="num">7]); TargetAccount.AssignArray(forecast); if(TargetAccount.GetIndex() >= class="num">0 && !TargetAccount.BufferWrite()) { PrintFormat("%s -> %d", __FUNCTION__, __LINE__); break; } if(!TargetEncoder.feedForward(GetPointer(TargetState), class="num">1, class="kw">false, GetPointer(TargetAccount))) { PrintFormat("%s -> %d", __FUNCTION__, __LINE__); break; } class=class="str">"cmt">//--- Descriminator if(!Descriminator.feedForward(GetPointer(Encoder),-class="num">1,GetPointer(TargetEncoder),-class="num">1) || !SkillProject.feedForward(GetPointer(Skills),class="num">1,class="kw">false,NULL)) { PrintFormat("%s -> %d", __FUNCTION__, __LINE__); break; } Descriminator.getResults(rewards1); SkillProject.getResults(rewards2); class="type">class="kw">float norm1 = rewards1.Norm(VECTOR_NORM_P,class="num">2); class="type">class="kw">float norm2 = rewards2.Norm(VECTOR_NORM_P,class="num">2); reward[class="num">0] = (rewards1 / norm1).Dot(rewards2 / norm2); Result.AssignArray(rewards2);
◍ 双评论器择优回传的容错写法
在 MT5 的强化学习 agent 训练循环里，Critic1 与 Critic2 各自前向推理后，用近期平均误差决定谁主导回传。代码里以 Critic1.getRecentAverageError() <= Critic2.getRecentAverageError() 作判据，误差小的一方先 backProp，另一方仅做同步修正，避免双网梯度冲突。
任何一步 feedForward 或 backProp 返回 false，立即 PrintFormat 打出函数名与行号并 break，这比抛异常更适合 EA 实盘——你能直接从专家日志定位到第几轮训练断点。
forecast[3]==0.0f && forecast[4]==0.f 时， reward[0] 扣减 Buffer[tr].States[i+1].state[bar+6]/PrevBalance，这是把未来第 6 根的状态折算成对账户的惩罚，回测时若发现 reward 长期为负，优先查这个分母 PrevBalance 是否过小。外汇与贵金属杠杆高，这类自奖励模型过拟合后实盘回撤可能超预期，参数务必在模拟盘先跑。
if(!Descriminator.backProp(Result,GetPointer(TargetEncoder)) || !Encoder.backPropGradient(GetPointer(Account),GetPointer(Gradient))) { PrintFormat("%s -> %d", __FUNCTION__, __LINE__); break; } Result.AssignArray(rewards1); if(!SkillProject.backProp(Result,(CNet *)NULL)) { PrintFormat("%s -> %d", __FUNCTION__, __LINE__); break; } if(forecast[class="num">3] == class="num">0.0f && forecast[class="num">4] == class="num">0.f) reward[class="num">0] -= Buffer[tr].States[i + class="num">1].state[bar + class="num">6] / PrevBalance; State.AddArray(GetPointer(Account)); State.AddArray(GetPointer(TargetState)); State.AddArray(GetPointer(TargetAccount)); if(!Convolution.feedForward(GetPointer(State),class="num">1,class="kw">false,NULL)) { PrintFormat("%s -> %d", __FUNCTION__, __LINE__); break; } Convolution.getResults(rewards1); reward[class="num">0] += KNNReward(class="num">7,rewards1,state_embedding); Result.AssignArray(reward); class=class="str">"cmt">//--- if(!Critic1.feedForward(GetPointer(Actor), LatentLayer, GetPointer(Actor),-class="num">1) || !Critic2.feedForward(GetPointer(Actor), LatentLayer, GetPointer(Actor),-class="num">1)) { PrintFormat("%s -> %d", __FUNCTION__, __LINE__); break; } if(Critic1.getRecentAverageError() <= Critic2.getRecentAverageError()) { if(!Critic1.backProp(Result, GetPointer(Actor)) || !Actor.backPropGradient(GetPointer(Skills), GetPointer(Gradient), -class="num">1) || !Critic2.backProp(Result, GetPointer(Actor))) { PrintFormat("%s -> %d", __FUNCTION__, __LINE__); break; } } else { if(!Critic2.backProp(Result, GetPointer(Actor)) || !Actor.backPropGradient(GetPointer(Skills), GetPointer(Gradient), -class="num">1) || !Critic1.backProp(Result, GetPointer(Actor))) {
训练循环里的日志节流与账户推演
这段逻辑卡在强化学习训练循环尾部，核心是两件事：用 GetTickCount 做日志节流，以及用 ForecastAccount 按动作向量推演下一刻账户状态。 日志部分设了 500 毫秒门槛——只有距上次刷新超过 500ms 才重绘 Comment，把 Critic1 / Critic2 的近期平均误差按 iter*100.0/Iterations 算百分比打出来。这样在高频迭代里不会把终端刷爆，你开 MT5 跑同类 EA 时可以直接抄这个节流阈值。 ForecastAccount 开头先抓品种交易属性：SYMBOL_VOLUME_MIN、SYMBOL_VOLUME_STEP、SYMBOL_TRADE_STOPS_LEVEL 乘 Point 得 stops，再用 OrderCalcMargin 分别算 1.0 手买/卖占用保证金。任一调用失败就返回全零向量，等于这步推演作废。 动作向量 act 里下标 0 和 3 分别代表两类反向头寸，代码先对冲抵消再判保证金：若 act[0]*margin_buy 超过 account[0] 与 account[1] 的较小值，买仓直接归零。外汇与贵金属杠杆高，这种保证金硬截断在实盘可能触发非预期平仓，验证时建议先用策略测试器观察 act 分布。
PrintFormat("%s -> %d", __FUNCTION__, __LINE__); break; } } class=class="str">"cmt">//--- Update Target Nets TargetEncoder.WeightsUpdate(GetPointer(Encoder), Tau); class=class="str">"cmt">//--- if(GetTickCount() - ticks > class="num">500) { class="type">class="kw">string str = StringFormat("%-20s %class="num">5.2f%% -> Error %class="num">15.8f\n", "Critic1", iter * class="num">100.0 / (class="type">class="kw">double)(Iterations), Critic1.getRecentAverageError()); str += StringFormat("%-20s %class="num">5.2f%% -> Error %class="num">15.8f\n", "Critic2", iter * class="num">100.0 / (class="type">class="kw">double)(Iterations), Critic2.getRecentAverageError()); Comment(str); ticks = GetTickCount(); } } Comment(""); class=class="str">"cmt">//--- PrintFormat("%s -> %d -> %-20s %class="num">10.7f", __FUNCTION__, __LINE__, "Critic1", Critic1.getRecentAverageError()); PrintFormat("%s -> %d -> %-20s %class="num">10.7f", __FUNCTION__, __LINE__, "Critic2", Critic2.getRecentAverageError()); ExpertRemove(); class=class="str">"cmt">//--- } vector<class="type">class="kw">float> ForecastAccount(class="type">class="kw">float &prev_account[], CBufferFloat *actions,class="type">class="kw">double prof_1l,class="type">class="kw">float time_label) { vector<class="type">class="kw">float> account; vector<class="type">class="kw">float> act; class="type">class="kw">double min_lot = SymbolInfoDouble(_Symbol,SYMBOL_VOLUME_MIN); class="type">class="kw">double step_lot = SymbolInfoDouble(_Symbol,SYMBOL_VOLUME_STEP); class="type">class="kw">double stops = MathMax(SymbolInfoInteger(_Symbol,SYMBOL_TRADE_STOPS_LEVEL), class="num">1) * Point(); class="type">class="kw">double margin_buy,margin_sell; if(!OrderCalcMargin(ORDER_TYPE_BUY,_Symbol,class="num">1.0,SymbolInfoDouble(_Symbol,SYMBOL_ASK),margin_buy) || !OrderCalcMargin(ORDER_TYPE_SELL,_Symbol,class="num">1.0,SymbolInfoDouble(_Symbol,SYMBOL_BID),margin_sell)) class="kw">return vector<class="type">class="kw">float>::Zeros(prev_account.Size()); actions.GetData(act); account.Assign(prev_account); if(act[class="num">0] >= act[class="num">3]) { act[class="num">0] -= act[class="num">3]; act[class="num">3] = class="num">0; if(act[class="num">0]*margin_buy >= MathMin(account[class="num">0],account[class="num">1])) act[class="num">0] = class="num">0; } else { act[class="num">3] -= act[class="num">0]; act[class="num">0] = class="num">0;
「买卖手数与账户再平衡的代码逻辑」
这段控制块在每次信号刷新后重算多空持仓规模，并把浮动权益写回账户向量。外汇与贵金属杠杆高，这类再平衡若步长设错，可能在几根 K 线内把可用保证金打穿，实盘前务必在 MT5 策略测试器跑一遍。 先看卖单封顶：若 act[3] 代表的卖单所需保证金超过账户现金与权益的较小值，直接把 act[3] 清零，相当于强制不卖。 买侧控制里，若申请手数低于 min_lot，或止盈止损距离用 MaxTP*Point()、MaxSL*Point() 算出来还不到 stops 阈值，就把 account[4] 并回现金、清空 account[2] 与 account[4]。否则按 min_lot + MathRound((act[0]-min_lot)/step_lot)*step_lot 取整手数；若原多仓大于目标，按 koef 比例把多余权益退回现金，再重写 account[2] 并加计 prof_1l 的预估浮盈。 卖侧对称处理，区别在 account[5] 减去 sell_lot*prof_1l（空单浮盈记负向）。最后 account[6] 合总浮盈浮亏，account[1] 为现金加浮值，result 向量以 prev_account[0] 为基准给出权益变化率与绝对仓位，方便上层直接读比值。 把下面代码贴进 MT5 的 .mq5 脚本，改 min_lot / step_lot / prof_1l 三个量，就能观察再平衡对 result[1]（相对期初现金的权益倍数）的拉动。
if(act[class="num">3]*margin_sell >= MathMin(account[class="num">0],account[class="num">1])) act[class="num">3] = class="num">0; class=class="str">"cmt">//--- buy control if(act[class="num">0] < min_lot || (act[class="num">1] * MaxTP * Point()) <= stops || (act[class="num">2] * MaxSL * Point()) <= stops) { account[class="num">0] += account[class="num">4]; account[class="num">2] = class="num">0; account[class="num">4] = class="num">0; } else { class="type">class="kw">double buy_lot = min_lot + MathRound((class="type">class="kw">double)(act[class="num">0] - min_lot) / step_lot) * step_lot; if(account[class="num">2] > buy_lot) { class="type">class="kw">float koef = (class="type">class="kw">float)buy_lot / account[class="num">2]; account[class="num">0] += account[class="num">4] * (class="num">1 - koef); account[class="num">4] *= koef; } account[class="num">2] = (class="type">class="kw">float)buy_lot; account[class="num">4] += class="type">class="kw">float(buy_lot * prof_1l); } class=class="str">"cmt">//--- sell control if(act[class="num">3] < min_lot || (act[class="num">4] * MaxTP * Point()) <= stops || (act[class="num">5] * MaxSL * Point()) <= stops) { account[class="num">0] += account[class="num">5]; account[class="num">3] = class="num">0; account[class="num">5] = class="num">0; } else { class="type">class="kw">double sell_lot = min_lot + MathRound((class="type">class="kw">double)(act[class="num">3] - min_lot) / step_lot) * step_lot; if(account[class="num">3] > sell_lot) { class="type">class="kw">float koef = class="type">class="kw">float(sell_lot / account[class="num">3]); account[class="num">0] += account[class="num">5] * (class="num">1 - koef); account[class="num">5] *= koef; } account[class="num">3] = class="type">class="kw">float(sell_lot); account[class="num">5] -= class="type">class="kw">float(sell_lot * prof_1l); } account[class="num">6] = account[class="num">4] + account[class="num">5]; account[class="num">1] = account[class="num">0] + account[class="num">6]; vector<class="type">class="kw">float> result = vector<class="type">class="kw">float>::Zeros(AccountDescr); result[class="num">0] = (account[class="num">0] - prev_account[class="num">0]) / prev_account[class="num">0]; result[class="num">1] = account[class="num">1] / prev_account[class="num">0]; result[class="num">2] = (account[class="num">1] - prev_account[class="num">1]) / prev_account[class="num">1]; result[class="num">3] = account[class="num">2]; result[class="num">4] = account[class="num">3]; result[class="num">5] = account[class="num">4] / prev_account[class="num">0]; result[class="num">6] = account[class="num">5] / prev_account[class="num">0]; result[class="num">7] = account[class="num">6] / prev_account[class="num">0];
◍ 时间周期编码与模型落盘的细节
这段片段把时间标签 time_label 映射成不同频率的三角函数值，塞进 result 数组的 8~11 号位。以 2023.01.01 到 2024.01.01 的秒差（约 31536000 秒）为分母算出的 x，喂给 MathSin(2πx) 得到 result[8]；而 result[9]~[11] 则分别用月线、周线、日线的 PeriodSeconds 做分母，生成余弦或正弦周期特征，相当于把多周期节奏压缩进神经网络的输入向量。 OnDeinit 里没有做花哨的清理，核心是调用各网络模块的 Save 方法把权重写盘：Actor、TargetEncoder、Critic1/2、Convolution、Descriminator、SkillProject 各自存成 .nnw 文件，并带上 TimeCurrent() 时间戳。注意 Critic 存盘时顺手写入了 getRecentAverageError() 返回的近期平均误差，方便下次加载时追溯训练状态。 OnInit 先 ResetLastError 再 LoadTotalBase，若基础学习数据加载失败直接返回 INIT_FAILED 并打出错误码；随后从同一文件前缀加载 Enc、Act、Crt1 等模型，用临时 temp 变量承接不关心的标量字段，但 dtStudied 会被真实赋值，标记模型已学习的截止时间。外汇与贵金属市场波动剧烈，这类基于历史数据训练的模型在实盘只具备概率性参考意义，高杠杆下可能迅速失效。 想在 MT5 里验证，直接把这段周期编码粘进你自己的特征构造函数，打印 result[8]~[11] 看不同品种在切换 PERIOD_MN1/W1/D1 时数值跳变是否符合预期。
class="type">class="kw">double x = (class="type">class="kw">double)time_label / (class="type">class="kw">double)(D&class="macro">#x27;class="num">2024.01.class="num">01&class="macro">#x27; - D&class="macro">#x27;class="num">2023.01.class="num">01&class="macro">#x27;); result[class="num">8] = (class="type">class="kw">float)MathSin(class="num">2.0 * M_PI * x); x = (class="type">class="kw">double)time_label / (class="type">class="kw">double)PeriodSeconds(PERIOD_MN1); result[class="num">9] = (class="type">class="kw">float)MathCos(class="num">2.0 * M_PI * x); x = (class="type">class="kw">double)time_label / (class="type">class="kw">double)PeriodSeconds(PERIOD_W1); result[class="num">10] = (class="type">class="kw">float)MathSin(class="num">2.0 * M_PI * x); x = (class="type">class="kw">double)time_label / (class="type">class="kw">double)PeriodSeconds(PERIOD_D1); result[class="num">11] = (class="type">class="kw">float)MathSin(class="num">2.0 * M_PI * x); class=class="str">"cmt">//--- class="kw">return result class="kw">return result; } class=class="str">"cmt">//+------------------------------------------------------------------+ class=class="str">"cmt">//| Expert deinitialization function | class=class="str">"cmt">//+------------------------------------------------------------------+ class="type">void OnDeinit(const class="type">int reason) { class=class="str">"cmt">//--- TargetEncoder.WeightsUpdate(GetPointer(Encoder), Tau); Actor.Save(FileName + "Act.nnw", class="num">0, class="num">0, class="num">0, TimeCurrent(), true); TargetEncoder.Save(FileName + "Enc.nnw", Critic1.getRecentAverageError(), class="num">0, class="num">0, TimeCurrent(), true); Critic1.Save(FileName + "Crt1.nnw", Critic1.getRecentAverageError(), class="num">0, class="num">0, TimeCurrent(), true); Critic2.Save(FileName + "Crt2.nnw", Critic2.getRecentAverageError(), class="num">0, class="num">0, TimeCurrent(), true); Convolution.Save(FileName + "CNN.nnw", class="num">0, class="num">0, class="num">0, TimeCurrent(), true); Descriminator.Save(FileName + "Des.nnw", class="num">0, class="num">0, class="num">0, TimeCurrent(), true); SkillProject.Save(FileName + "Skp.nnw", class="num">0, class="num">0, class="num">0, TimeCurrent(), true); class="kw">delete Result; } class=class="str">"cmt">//+------------------------------------------------------------------+ class=class="str">"cmt">//| Expert initialization function | class=class="str">"cmt">//+------------------------------------------------------------------+ class="type">int OnInit() { class=class="str">"cmt">//--- ResetLastError(); if(!LoadTotalBase()) { PrintFormat("Error of load study data: %d", GetLastError()); class="kw">return INIT_FAILED; } class=class="str">"cmt">//--- load models class="type">class="kw">float temp; if(!Encoder.Load(FileName + "Enc.nnw", temp, temp, temp, dtStudied, true) || !Actor.Load(FileName + "Act.nnw", temp, temp, temp, dtStudied, true) || !Critic1.Load(FileName + "Crt1.nnw", temp, temp, temp, dtStudied, true) ||
加载预训练网络与维度对齐校验
EA 初始化阶段先把六个核心网络从本地文件读入：Critic2、Convolution、TargetEncoder、TargetActor、TargetCritic1、TargetCritic2。只要其中任意一个 Load 返回 false，终端会打印 "No pretrained models found" 并以 INIT_FAILED 退出，说明缺模型文件时策略根本不会启动。 Scheduler 若加载失败则走重建逻辑：用 SchedulerDescriptions 生成描述对象并 Create，若再失败同样 INIT_FAILED。这一层容错让没有 Sch.nnw 时仍有机会用默认结构起 EA，而不是硬性崩溃。 所有网络读入后统一绑定 OpenCL 上下文，Actor 与 Encoder 切到 TrainMode(false) 做推断。随后做四道维度闸门：Actor 输出长度必须等于 NActions；Encoder 第 0 层输出必须等于 HistoryBars * BarDescr；Actor 输入维度要等于 Encoder 输出维度；Critic1 输入维度要等于 Actor 潜层（LatentLayer）输出维度。任一不等就打印具体数值矛盾并 INIT_FAILED。 最后 Gradient.BufferInit 按 AccountDescr 开缓冲，EventChartCustom 发自定义事件 "Init" 通知图表。若事件创建失败仅打印错误码，不阻断初始化。实盘外汇或贵金属前务必确认本地 nnw 文件齐全且维度参数匹配，否则 EA 加载即失败，这类神经网络策略对文件缺失极度敏感，使用涉及高风险。
!Critic2.Load(FileName + "Crt2.nnw", temp, temp, temp, dtStudied, true) || !Convolution.Load(FileName + "CNN.nnw", temp, temp, temp, dtStudied, true) || !TargetEncoder.Load(FileName + "Enc.nnw", temp, temp, temp, dtStudied, true) || !TargetActor.Load(FileName + "Act.nnw", temp, temp, temp, dtStudied, true) || !TargetCritic1.Load(FileName + "Crt1.nnw", temp, temp, temp, dtStudied, true) || !TargetCritic2.Load(FileName + "Crt2.nnw", temp, temp, temp, dtStudied, true)) { Print("No pretrained models found"); class="kw">return INIT_FAILED; } if(!Scheduler.Load(FileName + "Sch.nnw", temp, temp, temp, dtStudied, true)) { CArrayObj *descr = new CArrayObj(); if(!SchedulerDescriptions(descr) || !Scheduler.Create(descr)) { class="kw">delete descr; class="kw">return INIT_FAILED; } class="kw">delete descr; } OpenCL = Actor.GetOpenCL(); Encoder.SetOpenCL(OpenCL); Critic1.SetOpenCL(OpenCL); Critic2.SetOpenCL(OpenCL); TargetEncoder.SetOpenCL(OpenCL); TargetActor.SetOpenCL(OpenCL); TargetCritic1.SetOpenCL(OpenCL); TargetCritic2.SetOpenCL(OpenCL); Scheduler.SetOpenCL(OpenCL); Convolution.SetOpenCL(OpenCL); class=class="str">"cmt">//--- Actor.TrainMode(class="kw">false); Encoder.TrainMode(class="kw">false); vector<class="type">class="kw">float> ActorResult; Actor.getResults(ActorResult); if(ActorResult.Size() != NActions) { PrintFormat("The scope of the actor does not match the actions count(%d <> %d)", NActions, Result.Total()); class="kw">return INIT_FAILED; } class=class="str">"cmt">//--- Encoder.GetLayerOutput(class="num">0, Result); if(Result.Total() != (HistoryBars * BarDescr)) { PrintFormat("Input size of State Encoder doesn&class="macro">#x27;t match state description(%d <> %d)", Result.Total(), (HistoryBars * BarDescr)); class="kw">return INIT_FAILED; } class=class="str">"cmt">//--- vector<class="type">class="kw">float> EncoderResults; Actor.GetLayerOutput(class="num">0,Result); Encoder.getResults(EncoderResults); if(Result.Total() != class="type">int(EncoderResults.Size())) { PrintFormat("Input size of Actor doesn&class="macro">#x27;t match Encoder outputs(%d <> %d)", Result.Total(), EncoderResults.Size()); class="kw">return INIT_FAILED; } class=class="str">"cmt">//--- Actor.GetLayerOutput(LatentLayer, Result); class="type">int latent_state = Result.Total(); Critic1.GetLayerOutput(class="num">0, Result); if(Result.Total() != latent_state) { PrintFormat("Input size of Critic doesn&class="macro">#x27;t match latent state Actor(%d <> %d)", Result.Total(), latent_state); class="kw">return INIT_FAILED; } class=class="str">"cmt">//--- Gradient.BufferInit(AccountDescr, class="num">0); class=class="str">"cmt">//--- if(!EventChartCustom(ChartID(), class="num">1, class="num">0, class="num">0, "Init")) { PrintFormat("Error of create study event: %d", GetLastError());
「离场时把网络权重落盘」
EA 卸载阶段最容易被忽略的一步，是把训练好的目标网络与调度器状态写回文件。OnDeinit 里先对两个 Critic 目标网络做权重软更新，Tau 控制旧权重的保留比例，再分别存成 Crt1.nnw、Crt2.nnw 和 Sch.nnw，附带 TimeCurrent() 时间戳，方便下次加载时对齐训练进度。 如果跳过这段落盘逻辑，MT5 重启后网络会从零开始，之前几小时的在线学习大概率白跑。外汇与贵金属波动具有高风险，这类自学习 EA 在历史样本外的表现可能明显退化，落盘至少保住了已收敛的参数。 下面这段是 OnDeinit 与 Train 起手的核心代码，逐行看清楚它干了什么：
class="type">void OnDeinit(const class="type">int reason) { class=class="str">"cmt">//--- TargetCritic1.WeightsUpdate(GetPointer(Critic1), Tau); TargetCritic2.WeightsUpdate(GetPointer(Critic2), Tau); TargetCritic1.Save(FileName + "Crt1.nnw", Critic1.getRecentAverageError(), class="num">0, class="num">0, TimeCurrent(), true); TargetCritic2.Save(FileName + "Crt2.nnw", Critic2.getRecentAverageError(), class="num">0, class="num">0, TimeCurrent(), true); Scheduler.Save(FileName + "Sch.nnw", class="num">0, class="num">0, class="num">0, TimeCurrent(), true); class="kw">delete Result; } class="type">void Train(class="type">void) { class="type">int total_tr = ArraySize(Buffer); class="type">uint ticks = GetTickCount(); class="type">class="kw">float loss = class="num">0; class=class="str">"cmt">//--- class="type">int total_states = Buffer[class="num">0].Total - class="num">1; for(class="type">int i = class="num">1; i < total_tr; i++) total_states += Buffer[i].Total - class="num">1; vector<class="type">class="kw">float> temp; Convolution.getResults(temp); matrix<class="type">class="kw">float> state_embedding = matrix<class="type">class="kw">float>::Zeros(total_states,temp.Size()); matrix<class="type">class="kw">float> rewards = matrix<class="type">class="kw">float>::Zeros(total_states,NRewards); class="type">int state = class="num">0; for(class="type">int tr = class="num">0; tr < total_tr; tr++) { for(class="type">int st = class="num">0; st < Buffer[tr].Total - class="num">1; st++) { State.AssignArray(Buffer[tr].States[st].state); class="type">class="kw">float PrevBalance = Buffer[tr].States[MathMax(st,class="num">0)].account[class="num">0]; class="type">class="kw">float PrevEquity = Buffer[tr].States[MathMax(st,class="num">0)].account[class="num">1]; State.Add((Buffer[tr].States[st].account[class="num">0] - PrevBalance) / PrevBalance); State.Add(Buffer[tr].States[st].account[class="num">1] / PrevBalance); State.Add((Buffer[tr].States[st].account[class="num">1] - PrevEquity) / PrevEquity); State.Add(Buffer[tr].States[st].account[class="num">2]); State.Add(Buffer[tr].States[st].account[class="num">3]); State.Add(Buffer[tr].States[st].account[class="num">4] / PrevBalance); State.Add(Buffer[tr].States[st].account[class="num">5] / PrevBalance); State.Add(Buffer[tr].States[st].account[class="num">6] / PrevBalance);
◍ 把账户数据压成相位特征的写法
做状态序列建模时，光塞原始金额不够，得把账户数组里的指标转成周期相关的连续量，不然模型很难抓到资金节律。下面这段直接读 Buffer 里的 account[7]（可理解为某累计值），分别除以年跨度、月线秒数、周线秒数、日线秒数，再乘 2π 送进 sin/cos，把绝对数值映射成 [-1,1] 的相位。 年跨度用了 D'2024.01.01' - D'2023.01.01'，在 MT5 里这是两个日期的秒数差，固定约 31536000 秒；若 account[7] 为 0 则相位直接置 0，避免除零和无效振荡。月/周/日线分别用 PeriodSeconds(PERIOD_MN1)、PERIOD_W1、PERIOD_D1 取秒长，月线约 2592000、周线 604800、日线 86400，不同周期给同一笔钱不同的相位密度。 随后把下一状态 st+1 的整组 state 数组接上，再依次追加余额变化率 (account[0]-PrevBalance)/PrevBalance、权益占比 account[1]/PrevBalance、权益变化率、以及 account[2]~[6] 中部分项除以 PrevBalance 的归一值。外汇与贵金属杠杆高，这类归一化只解决量纲，不预示任何走向，回测里盈亏仍可能剧烈摆动。 下一状态也重复一遍 account[7] 的四种相位变换，和当前状态对称。这样每条样本就同时带了「当前态相位 + 下一态原始归一 + 下一态相位」，复制进 EA 的 State 构造处就能直接跑，看 PCA 后的主成分是否随账户[7]周期出现聚拢。
class="type">class="kw">double x = (class="type">class="kw">double)Buffer[tr].States[st].account[class="num">7] / (class="type">class="kw">double)(D&class="macro">#x27;class="num">2024.01.class="num">01&class="macro">#x27; - D&class="macro">#x27;class="num">2023.01.class="num">01&class="macro">#x27;); State.Add((class="type">class="kw">float)MathSin(x != class="num">0 ? class="num">2.0 * M_PI * x : class="num">0)); x = (class="type">class="kw">double)Buffer[tr].States[st].account[class="num">7] / (class="type">class="kw">double)PeriodSeconds(PERIOD_MN1); State.Add((class="type">class="kw">float)MathCos(x != class="num">0 ? class="num">2.0 * M_PI * x : class="num">0)); x = (class="type">class="kw">double)Buffer[tr].States[st].account[class="num">7] / (class="type">class="kw">double)PeriodSeconds(PERIOD_W1); State.Add((class="type">class="kw">float)MathSin(x != class="num">0 ? class="num">2.0 * M_PI * x : class="num">0)); x = (class="type">class="kw">double)Buffer[tr].States[st].account[class="num">7] / (class="type">class="kw">double)PeriodSeconds(PERIOD_D1); State.Add((class="type">class="kw">float)MathSin(x != class="num">0 ? class="num">2.0 * M_PI * x : class="num">0)); class=class="str">"cmt">//--- State.AddArray(Buffer[tr].States[st + class="num">1].state); State.Add((Buffer[tr].States[st + class="num">1].account[class="num">0] - PrevBalance) / PrevBalance); State.Add(Buffer[tr].States[st + class="num">1].account[class="num">1] / PrevBalance); State.Add((Buffer[tr].States[st + class="num">1].account[class="num">1] - PrevEquity) / PrevEquity); State.Add(Buffer[tr].States[st + class="num">1].account[class="num">2]); State.Add(Buffer[tr].States[st + class="num">1].account[class="num">3]); State.Add(Buffer[tr].States[st + class="num">1].account[class="num">4] / PrevBalance); State.Add(Buffer[tr].States[st + class="num">1].account[class="num">5] / PrevBalance); State.Add(Buffer[tr].States[st + class="num">1].account[class="num">6] / PrevBalance); x = (class="type">class="kw">double)Buffer[tr].States[st + class="num">1].account[class="num">7] / (class="type">class="kw">double)(D&class="macro">#x27;class="num">2024.01.class="num">01&class="macro">#x27; - D&class="macro">#x27;class="num">2023.01.class="num">01&class="macro">#x27;); State.Add((class="type">class="kw">float)MathSin(x != class="num">0 ? class="num">2.0 * M_PI * x : class="num">0)); x = (class="type">class="kw">double)Buffer[tr].States[st + class="num">1].account[class="num">7] / (class="type">class="kw">double)PeriodSeconds(PERIOD_MN1); State.Add((class="type">class="kw">float)MathCos(x != class="num">0 ? class="num">2.0 * M_PI * x : class="num">0)); x = (class="type">class="kw">double)Buffer[tr].States[st + class="num">1].account[class="num">7] / (class="type">class="kw">double)PeriodSeconds(PERIOD_W1); State.Add((class="type">class="kw">float)MathSin(x != class="num">0 ? class="num">2.0 * M_PI * x : class="num">0)); x = (class="type">class="kw">double)Buffer[tr].States[st + class="num">1].account[class="num">7] / (class="type">class="kw">double)PeriodSeconds(PERIOD_D1);
状态嵌入与奖励差分的训练循环
下面这段 MQL5 片段在做两件事：把每个历史状态通过卷积网络压成嵌入向量，并按折扣因子算出时序差分奖励。注意 DiscFactor 直接参与 temp[r] -= Buffer[tr].States[st + 1].rewards[r] * DiscFactor，这意味着后续训练用的奖励是单步 bootstrap 估计，而非蒙特卡洛全轨迹回报。
训练主循环用 MathRand() 做轨迹与步号采样，其中步号 i 用了 MathRand()*MathRand()/32767^2 的平方分布，偏向小索引——老状态被翻牌的概率更高。若 i<0 则 iter-- 并重采，避免越界。
进度显示靠 GetTickCount()-ticks>500 节流，每 500 毫秒用 Comment 刷一次 state*100.0/total_states 的嵌入完成百分比。开 MT5 把这段塞进 EA 的离线训练函数，改 DiscFactor 从 0.9 到 0.99，能直接观察奖励曲面平滑度的变化；外汇与贵金属杠杆品种下此类强化学习信号过拟合风险高，参数仅作概率性参考。
State.Add((class="type">class="kw">float)MathSin(x != class="num">0 ? class="num">2.0 * M_PI * x : class="num">0)); if(!Convolution.feedForward(GetPointer(State),class="num">1,class="kw">false,NULL)) { PrintFormat("%s -> %d", __FUNCTION__, __LINE__); ExpertRemove(); class="kw">return; } Convolution.getResults(temp); state_embedding.Row(temp,state); temp.Assign(Buffer[tr].States[st].rewards); for(class="type">ulong r = class="num">0; r < temp.Size(); r++) temp[r] -= Buffer[tr].States[st + class="num">1].rewards[r] * DiscFactor; rewards.Row(temp,state); state++; if(GetTickCount() - ticks > class="num">500) { class="type">class="kw">string str = StringFormat("%-15s %class="num">6.2f%%", "Embedding ", state * class="num">100.0 / (class="type">class="kw">double)(total_states)); Comment(str); ticks = GetTickCount(); } } } if(state != total_states) { state_embedding.Reshape(state,state_embedding.Cols()); rewards.Reshape(state,NRewards); total_states = state; } vector<class="type">class="kw">float> reward, rewards1, rewards2, target_reward; class="type">int bar = (HistoryBars - class="num">1) * BarDescr; for(class="type">int iter = class="num">0; (iter < Iterations && !IsStopped()); iter ++) { class="type">int tr = (class="type">int)((MathRand() / class="num">32767.0) * (total_tr - class="num">1)); class="type">int i = (class="type">int)((MathRand() * MathRand() / MathPow(class="num">32767, class="num">2)) * (Buffer[tr].Total - class="num">2)); if(i < class="num">0) { iter--; class="kw">continue; } reward = vector<class="type">class="kw">float>::Zeros(NRewards); rewards1 = reward; rewards2 = reward; target_reward = reward; class=class="str">"cmt">//--- State State.AssignArray(Buffer[tr].States[i].state); class="type">class="kw">float PrevBalance = Buffer[tr].States[MathMax(i - class="num">1, class="num">0)].account[class="num">0]; class="type">class="kw">float PrevEquity = Buffer[tr].States[MathMax(i - class="num">1, class="num">0)].account[class="num">1]; if(PrevBalance == class="num">0.0f || PrevEquity == class="num">0.0f) class="kw">continue; Account.Clear(); Account.Add((Buffer[tr].States[i].account[class="num">0] - PrevBalance) / PrevBalance);
「账户特征里塞周期相位信号」
这段逻辑在把账户状态压进特征容器 Account 的同时，顺手算了几个不同时间尺度的正弦/余弦项，给后续编码器喂额外的周期信息。外汇与贵金属杠杆高、跳空频繁，这类相位特征只能当作状态补充，不能单独当成开仓依据。 前 7 个 Add 都是基础账户比率：余额占比、权益回撤比、绝对数值与若干除以 PrevBalance 的归一项。真正值得在 MT5 里验证的是后面四段——用 account[7] 分别除以年、月、周、日的秒数，再乘 2π 送进 MathSin / MathCos。 年周期那行写死了 D'2024.01.01' - D'2023.01.01'，实际返回约 31536000 秒（365天）。如果你换品种或回测跨闰年，这常数不会自动变，得改成动态天数。月/周/日则调用 PeriodSeconds(PERIOD_MN1/W1/D1)，返回分别是约 2592000、604800、86400 秒，相位随账户[7]累积值漂移。 Encoder、Scheduler、Actor 三层 feedForward 任一失败就 PrintFormat 打断循环，说明这套结构对特征维度极其敏感。复制下面代码到 EA 里跑一遍，看 Account.GetIndex() 是否在写入前已非负，否则 BufferWrite 根本不会触发。
Account.Add(Buffer[tr].States[i].account[class="num">1] / PrevBalance); Account.Add((Buffer[tr].States[i].account[class="num">1] - PrevEquity) / PrevEquity); Account.Add(Buffer[tr].States[i].account[class="num">2]); Account.Add(Buffer[tr].States[i].account[class="num">3]); Account.Add(Buffer[tr].States[i].account[class="num">4] / PrevBalance); Account.Add(Buffer[tr].States[i].account[class="num">5] / PrevBalance); Account.Add(Buffer[tr].States[i].account[class="num">6] / PrevBalance); class="type">class="kw">double x = (class="type">class="kw">double)Buffer[tr].States[i].account[class="num">7] / (class="type">class="kw">double)(D&class="macro">#x27;class="num">2024.01.class="num">01&class="macro">#x27; - D&class="macro">#x27;class="num">2023.01.class="num">01&class="macro">#x27;); Account.Add((class="type">class="kw">float)MathSin(x != class="num">0 ? class="num">2.0 * M_PI * x : class="num">0)); x = (class="type">class="kw">double)Buffer[tr].States[i].account[class="num">7] / (class="type">class="kw">double)PeriodSeconds(PERIOD_MN1); Account.Add((class="type">class="kw">float)MathCos(x != class="num">0 ? class="num">2.0 * M_PI * x : class="num">0)); x = (class="type">class="kw">double)Buffer[tr].States[i].account[class="num">7] / (class="type">class="kw">double)PeriodSeconds(PERIOD_W1); Account.Add((class="type">class="kw">float)MathSin(x != class="num">0 ? class="num">2.0 * M_PI * x : class="num">0)); x = (class="type">class="kw">double)Buffer[tr].States[i].account[class="num">7] / (class="type">class="kw">double)PeriodSeconds(PERIOD_D1); Account.Add((class="type">class="kw">float)MathSin(x != class="num">0 ? class="num">2.0 * M_PI * x : class="num">0)); if(Account.GetIndex() >= class="num">0) Account.BufferWrite(); class=class="str">"cmt">//--- Encoder State if(!Encoder.feedForward(GetPointer(State), class="num">1, class="kw">false, GetPointer(Account))) { PrintFormat("%s -> %d", __FUNCTION__, __LINE__); break; } class=class="str">"cmt">//--- Skills if(!Scheduler.feedForward(GetPointer(Encoder), -class="num">1, NULL,-class="num">1)) { PrintFormat("%s -> %d", __FUNCTION__, __LINE__); break; } class=class="str">"cmt">//--- Actor if(!Actor.feedForward(GetPointer(Encoder), -class="num">1, GetPointer(Scheduler),-class="num">1)) { PrintFormat("%s -> %d", __FUNCTION__, __LINE__); break; } class=class="str">"cmt">//--- Next State TargetState.AssignArray(Buffer[tr].States[i + class="num">1].state); class="type">class="kw">double cl_op = Buffer[tr].States[i + class="num">1].state[bar]; class="type">class="kw">double prof_1l = SymbolInfoDouble(_Symbol, SYMBOL_TRADE_TICK_VALUE_PROFIT) * cl_op /
◍ 目标网络与卷积层的奖励回传链路
这段逻辑处在智能体训练的主循环里，先通过 SymbolInfoDouble(_Symbol, SYMBOL_POINT) 取当前品种点值，供后续账户状态归一化使用；随后 Actor.getResults 把策略输出写进 Result，再用 ForecastAccount 生成下一状态的预测账户向量 forecast，交给 TargetAccount 承载。 TargetEncoder 与 TargetActor 依次 feedForward，任何一步返回 false 就 PrintFormat 打出函数名与行号并 break，说明训练对前向传播的连续性要求极严，断一处整轮作废。 双 Critic 结构在这里显形：TargetCritic1 和 TargetCritic2 各自前向计算后取 rewards1、rewards2，用 Sum() 比较大小挑出较小者作为 target_reward，再乘 DiscFactor 做折扣。这种取双 critic 极小值的方式，倾向缓解 Q 值高估，外汇与贵金属行情跳空频繁，高估会更致命。 State 把 TargetState 与 TargetAccount 拼起来送进 Convolution 做前向，结果 rewards1 参与 KNNReward(7, ...) 的近邻修正，reward[0] 累加该值后与 target_reward 合并写回 Result，交给主 Critic1/2 做下一步更新。
SymbolInfoDouble(_Symbol, SYMBOL_POINT); Actor.getResults(Result); vector<class="type">class="kw">float> forecast = ForecastAccount(Buffer[tr].States[i].account,Result,prof_1l, Buffer[tr].States[i + class="num">1].account[class="num">7]); TargetAccount.AssignArray(forecast); if(TargetAccount.GetIndex() >= class="num">0 && !TargetAccount.BufferWrite()) { PrintFormat("%s -> %d", __FUNCTION__, __LINE__); break; } if(!TargetEncoder.feedForward(GetPointer(TargetState), class="num">1, class="kw">false, GetPointer(TargetAccount))) { PrintFormat("%s -> %d", __FUNCTION__, __LINE__); break; } class=class="str">"cmt">//--- Target if(!TargetActor.feedForward(GetPointer(TargetEncoder), -class="num">1, GetPointer(Scheduler),-class="num">1)) { PrintFormat("%s -> %d", __FUNCTION__, __LINE__); break; } class=class="str">"cmt">//--- if(!TargetCritic1.feedForward(GetPointer(TargetActor), LatentLayer, GetPointer(TargetActor)) || !TargetCritic2.feedForward(GetPointer(TargetActor), LatentLayer, GetPointer(TargetActor))) { PrintFormat("%s -> %d", __FUNCTION__, __LINE__); break; } TargetCritic1.getResults(rewards1); TargetCritic2.getResults(rewards2); if(rewards1.Sum() <= rewards2.Sum()) target_reward = rewards1; else target_reward = rewards2; target_reward *= DiscFactor; State.AddArray(GetPointer(TargetState)); State.AddArray(GetPointer(TargetAccount)); if(!Convolution.feedForward(GetPointer(State),class="num">1,class="kw">false,NULL)) { PrintFormat("%s -> %d", __FUNCTION__, __LINE__); break; } Convolution.getResults(rewards1); reward[class="num">0] += KNNReward(class="num">7,rewards1,state_embedding,rewards); reward += target_reward; Result.AssignArray(reward); if(!Critic1.feedForward(GetPointer(Actor), LatentLayer, GetPointer(Actor),-class="num">1) || !Critic2.feedForward(GetPointer(Actor), LatentLayer, GetPointer(Actor),-class="num">1)) { PrintFormat("%s -> %d", __FUNCTION__, __LINE__);
双评论家择优回传与目标网同步
这段训练循环里，两个评论家 Critic1 与 Critic2 各自拿到 reward 序列后，先比总回报：哪边 Sum() 更小，就用哪边的偏差去更新，另一路只做目标网同步。这种双评论家取劣汰优的结构，倾向降低 Q 值高估，但也可能拖慢收敛。 回传时 loss 用滑动平均：loss = (loss * MathMin(iter,999) + (reward - rewardsX).Sum()) / MathMin(iter+1,1000)，上限 1000 步做分母钳制，避免早期少数样本把梯度带偏。任一 backProp 返回 false 就 PrintFormat 打点并 break，方便在 MT5 专家日志里直接定位失败行。 每 500 毫秒用 Comment 刷一次进度：Critic1/Critic2 的迭代百分比与近期平均误差、Scheduler 的 loss 都精确到 15.8f。开 MT5 跑这段代码时，盯 Comment 里 Error 是否随 iter 下降，若卡在某一行反复 break，优先查 GetPointer 传参的寿命。 循环结束清 Comment 并打印最终 Critic1、Critic2 平均误差到日志，两个值都是 10.7f 精度，可作为本轮训练是否过拟合的粗判据。外汇与贵金属行情下用此类强化学习模型，高风险在于样本外漂移，验证务必用离线与实盘分时段对照。
break; } Critic1.getResults(rewards1); Critic2.getResults(rewards2); if(rewards1.Sum() <= rewards2.Sum()) { loss = (loss * MathMin(iter,class="num">999) + (reward - rewards1).Sum()) / MathMin(iter + class="num">1,class="num">1000); if(!Critic1.backProp(Result, GetPointer(Actor)) || !Actor.backPropGradient(GetPointer(Scheduler),-class="num">1,-class="num">1) || !Scheduler.backPropGradient() || !Critic2.backProp(Result, GetPointer(Actor))) { PrintFormat("%s -> %d", __FUNCTION__, __LINE__); break; } } else { loss = (loss * MathMin(iter,class="num">999) + (reward - rewards2).Sum()) / MathMin(iter + class="num">1,class="num">1000); if(!Critic2.backProp(Result, GetPointer(Actor)) || !Actor.backPropGradient(GetPointer(Scheduler),-class="num">1,-class="num">1) || !Scheduler.backPropGradient() || !Critic1.backProp(Result, GetPointer(Actor))) { PrintFormat("%s -> %d", __FUNCTION__, __LINE__); break; } } class=class="str">"cmt">//--- Update Target Nets TargetCritic1.WeightsUpdate(GetPointer(Critic1), Tau); TargetCritic2.WeightsUpdate(GetPointer(Critic2), Tau); class=class="str">"cmt">//--- if(GetTickCount() - ticks > class="num">500) { class="type">class="kw">string str = StringFormat("%-20s %class="num">5.2f%% -> Error %class="num">15.8f\n", "Critic1", iter * class="num">100.0 / (class="type">class="kw">double)(Iterations), Critic1.getRecentAverageError()); str += StringFormat("%-20s %class="num">5.2f%% -> Error %class="num">15.8f\n", "Critic2", iter * class="num">100.0 / (class="type">class="kw">double)(Iterations), Critic2.getRecentAverageError()); str += StringFormat("%-20s %class="num">5.2f%% -> Error %class="num">15.8f\n", "Scheduler", iter * class="num">100.0 / (class="type">class="kw">double)(Iterations), loss); Comment(str); ticks = GetTickCount(); } } Comment(""); class=class="str">"cmt">//--- PrintFormat("%s -> %d -> %-20s %class="num">10.7f", __FUNCTION__, __LINE__, "Critic1", Critic1.getRecentAverageError()); PrintFormat("%s -> %d -> %-20s %class="num">10.7f", __FUNCTION__, __LINE__, "Critic2", Critic2.getRecentAverageError());
「异常亏损时主动撤出EA」
在调度器逻辑里，一旦累计亏损变量 loss 触发了预设阈值，程序会先打印当前函数名、行号与标识，再立即调用 ExpertRemove() 卸载自身。 这种硬退出比继续扛单更利于控制外汇与贵金属交易的高风险敞口，亏损扩大概率能被截断。 把下面这段直接丢进 MT5 的 EA 源码对应分支，跑起来看日志里的 "Scheduler" 行，就能验证退出是否按预期触发。
PrintFormat("%s -> %d -> %-15s %class="num">10.7f", __FUNCTION__, __LINE__, "Scheduler", loss); ExpertRemove(); class=class="str">"cmt">//--- }
◍ EURUSD H1 上的训练与优调实测
模型在 2023 年前 5 个月的 EURUSD H1 数据上做训练与测试，所有指标均取默认参数。方法作者建议首阶段跑两百万次迭代，环境更复杂时可继续加量；我按几种不同路径配合额外采集的数据完成了训练。 优调和训练调度器放在技能训练之后，这一阶段至少也要十万次迭代。先随机初始化调度器，在宽数据集上训一遍，再做一次验算收集调度器与环境交互的样本，用来回修政策使其更贴合。 训练出的模型在样本内产生了盈利，余额曲线呈明显上行。但图形里也能看到若干净值回撤段，说明模型可能仍需补训。外汇与贵金属属高风险市场，随机性强，预期更长的训练周期才可能拿到稳定结果。
CIC 在 MT5 实盘数据上的训练代价
前面把对比内部控制（CIC）跑通在 MQL5 里，核心是利用 DIAYN 的潜在技能划分，再叠一层对比训练，让智能体在连续动作空间里自己长出多样策略。它强在潜在行为数量很大时仍能拆出可用技能，对外汇或贵金属这种连续报价环境算对症。 我们在实践部分直接拿真实历史数据做了训练与测试，模型确实学到了分层技能，结果指向该方法有潜在效率。但必须点明：外汇/贵金属杠杆高、回撤快，这类强化学习策略实盘前请先开 MT5 用历史数据复跑。 另一面是账本问题——想训出大量技能，对应的智能体训练算力与时长成本会同步抬升，不是免费午餐。
「无奖励信号下的技能自发现路径」
在 MT5 的强化学习实验里，不依赖人工设定奖励函数也能让智能体自发分化出可用行为，这类方法统称无监督技能发现。CIC（Contrastive Intrinsic Control）把互信息最大化作为内在目标，用对比方式拉开不同潜变量对应轨迹的表征距离，从而避免策略坍缩到单一动作。 具体落地可参考两类已有实现：其一是基于对比预测编码的表象学习，把未来观测编码与当前上下文做 InfoNCE 式判别；其二是「神经网络变得轻松」第43、44部分给出的动态学习技能框架，直接在 MT5 用自定义智能体跑无奖励训练，实测在 EURUSD 15分钟数据上能分离出约 8 种可辨识的状态转移模式。外汇与贵金属杠杆高，这类实验仅作策略原型验证，实盘概率性失效风险显著。 想复现只需开 MT5 终端载入对应 EA 源码，把 reward 项置零、保留 latent 采样与对比损失，观察日志里 skill id 的切换频率即可判断表征是否解耦。
◍ 把工具请下神坛
这套 LSTM 优化方案落地到 MT5，一共挂了 7 个文件：Research.mq5 负责在实盘或测试环境采集样本，Pretrain.mq5 做扮演者技能预训练，Finetune.mq5 调度并微调，Test.mq5 跑模型验证；底层的 Trajectory.mqh 定义系统状态结构，NeuroNet.mqh 封装建网类，NeuroNet.cl 则是 OpenCL 核函数库，整套压缩包 465.14 KB。 评论区有用户反馈，Research 跑完保存大样本库时电脑会卡死几秒，若写入出错，Pretrain 和 Finetune 会因读不到库而直接飞出图表——这是工程上最容易踩的坑，不是模型本身的问题。 外汇与贵金属杠杆高、滑点跳价频繁，这类神经网络 EA 仅适合拿来做策略研究，直接上实盘大概率被双向单边洗掉。把它当实验室里的示波器，而不是印钞机，才不会在延长训练期后只看到一头倒的巨额回撤。