🧠
神经网络变得简单（第 66 部分）：离线学习中的探索问题·进阶篇
（2/3）·在线探索算法在离线场景频频失效，训练集子空间窄化正悄悄拖垮你的策略泛化
很多交易者把在线 RL 里好用的探索机制直接搬进离线训练，结果发现策略一上手就飘。根子在离线数据只在任务的小子空间里采集，代理稍越界就失去预测锚点。本篇接着系列第 1 篇的基础，深挖 ExORL 怎么靠换数据而不是换算法来补这个洞。
神经网络模型加载失败的兜底重建
在 MT5 的 EA 初始化阶段，Actor、Critic 与 Convolution 三个网络各自尝试从本地 .nnw 文件载入权重；一旦 Load() 返回 false，说明模型文件缺失或损坏，代码会打印 Init new Critic and Encoder models 并进入新建分支。
新建时先 new 出三组 CArrayObj 指针（actor / critic / convolution），调用 CreateDescriptions() 填充层结构描述；任意一步失败立即 delete 全部指针并返回 INIT_FAILED，避免悬空内存。
注意三个分支的清理逻辑不对称：Actor 分支只 delete actor 和 critic，而 Critic、Convolution 分支多清理了 convolution 指针——若你在自己的强化学习 EA 里抄这段代码，得核对指针集合是否真的只用到了两个对象。
模型全部就绪后，Critic.SetOpenCL(Actor.GetOpenCL()) 把计算设备上下文对齐，再切到 TrainMode(false) 做推理；若 Result.Total() != NActions 则后续还会报错，说明动作空间维度对不上，开 MT5 跑之前先确认 NActions 宏定义。
MQL5 / C++
if(!Critic.Load(FileName + "Crt1.nnw", temp, temp, temp, dtStudied, true)) { Print("Init new Critic and Encoder models"); CArrayObj *actor = new CArrayObj(); CArrayObj *critic = new CArrayObj(); CArrayObj *convolution = new CArrayObj(); if(!CreateDescriptions(actor, critic, convolution)) { class="kw">delete actor; class="kw">delete critic; class="kw">delete convolution; class="kw">return INIT_FAILED; } if(!Critic.Create(critic)) { class="kw">delete actor; class="kw">delete critic; class="kw">delete convolution; class="kw">return INIT_FAILED; } class="kw">delete actor; class="kw">delete critic; class="kw">delete convolution; class=class="str">"cmt">//--- } if(!Convolution.Load(FileName + "CNN.nnw", temp, temp, temp, dtStudied, true)) { Print("Init new Critic and Encoder models"); CArrayObj *actor = new CArrayObj(); CArrayObj *critic = new CArrayObj(); CArrayObj *convolution = new CArrayObj(); if(!CreateDescriptions(actor, critic, convolution)) { class="kw">delete actor; class="kw">delete critic; class="kw">delete convolution; class="kw">return INIT_FAILED; } if(!Convolution.Create(convolution)) { class="kw">delete actor; class="kw">delete critic; class="kw">delete convolution; class="kw">return INIT_FAILED; } class="kw">delete actor; class="kw">delete critic; class="kw">delete convolution; class=class="str">"cmt">//--- }
「初始化校验与状态向量拼装」
EA 在 OnInit 里先核对网络输出维度：Actor 的动作数必须和 NActions 一致，否则直接 INIT_FAILED；输入层尺寸则要等于 HistoryBars * BarDescr，比如历史 50 根 K 线、每根描述 10 维，那 Result.Total() 就得是 500，对不上就崩初始化。 账户快照在初始化阶段就存好 PrevBalance 与 PrevEquity，bGradient 缓冲区按 AccountDescr 和 NActions 的较大值开，初始填 0，这一步决定了后面梯度回传的内存上限。 OnDeinit 里把 Agent 网络按 ActEx%d.nnw 命名落盘，Save 失败会打印 GetLastError 码；Result 对象必须手动 delete，漏掉会在 MT5 里吃内存泄漏。 CreateEmbeddings 是状态表征的核心：先跑卷积拿 temp，再 LoadTotalBase 装载基础数据，失败就返回空矩阵。它把多笔交易 Buffer 里的状态全量摊平——total_states 是所有 tr 的 Buffer[tr].Total 累加，再 Resize 成矩阵。 每个状态除了原始 state，还追加了 7 个浮点特征：余额变化率、权益占比、权益变化率，以及 account[2]~[5] 里的仓位与浮盈类字段，其中涉及余额的都除以 prevBalance 做归一。外汇与贵金属波动大，这类归一若 prevBalance 接近 0 会爆 NaN，实盘前务必在策略测试器里用极端账户值跑一遍。
MQL5 / C++
PrintFormat("The scope of the actor does not match the actions count(%d <> %d)", NActions, Result.Total()); class="kw">return INIT_FAILED; } class=class="str">"cmt">//--- Actor.GetLayerOutput(class="num">0, Result); if(Result.Total() != (HistoryBars * BarDescr)) { PrintFormat("Input size of Actor doesn&class="macro">#x27;t match state description(%d <> %d)", Result.Total(), (HistoryBars * BarDescr)); class="kw">return INIT_FAILED; } PrevBalance = AccountInfoDouble(ACCOUNT_BALANCE); PrevEquity = AccountInfoDouble(ACCOUNT_EQUITY); BaseLoaded = class="kw">false; bGradient.BufferInit(MathMax(AccountDescr, NActions), class="num">0); class=class="str">"cmt">//--- class="kw">return(INIT_SUCCEEDED); } class="type">void OnDeinit(const class="type">int reason) { class=class="str">"cmt">//--- ResetLastError(); if(!Actor.Save(StringFormat("%sActEx%d.nnw", FileName, Agent), class="num">0, class="num">0, class="num">0, TimeCurrent(), true)) PrintFormat("Error of saving Agent %d: %d", Agent, GetLastError()); class="kw">delete Result; } matrix<class="type">class="kw">float> CreateEmbeddings(class="type">void) { vector<class="type">class="kw">float> temp; CBufferFloat State; Convolution.getResults(temp); matrix<class="type">class="kw">float> result = matrix<class="type">class="kw">float>::Zeros(class="num">0, temp.Size()); BaseLoaded = LoadTotalBase(); if(!BaseLoaded) { PrintFormat("%s - %d => Error of load base", __FUNCTION__, __LINE__); class="kw">return result; } class="type">int total_tr = ArraySize(Buffer); class=class="str">"cmt">//--- class="type">int total_states = Buffer[class="num">0].Total; for(class="type">int i = class="num">1; i < total_tr; i++) total_states += Buffer[i].Total; result.Resize(total_states, temp.Size()); class="type">int state = class="num">0; for(class="type">int tr = class="num">0; tr < total_tr; tr++) { for(class="type">int st = class="num">0; st < Buffer[tr].Total; st++) { State.AssignArray(Buffer[tr].States[st].state); class="type">class="kw">float prevBalance = Buffer[tr].States[MathMax(st - class="num">1, class="num">0)].account[class="num">0]; class="type">class="kw">float prevEquity = Buffer[tr].States[MathMax(st - class="num">1, class="num">0)].account[class="num">1]; State.Add((Buffer[tr].States[st].account[class="num">0] - prevBalance) / prevBalance); State.Add(Buffer[tr].States[st].account[class="num">1] / prevBalance); State.Add((Buffer[tr].States[st].account[class="num">1] - prevEquity) / prevEquity); State.Add(Buffer[tr].States[st].account[class="num">2]); State.Add(Buffer[tr].States[st].account[class="num">3]); State.Add(Buffer[tr].States[st].account[class="num">4] / prevBalance); State.Add(Buffer[tr].States[st].account[class="num">5] / prevBalance);
◍ 把账户状态塞进卷积前的特征拼装
这段逻辑干的事，是把每一笔历史状态的账户字段和动作向量，拼成一条定长的特征行，再喂给卷积网络做前向推理。特征里既有余额比值，也有按年、月、周、日周期归一化后的正弦/余弦相位，相当于把时间尺度压缩进 embedding。 先看账户侧：account[6] 除以 prevBalance 得到余额相对变化率，直接 Add 进 State；account[7] 则被分别除以 365 天秒数、PERIOD_MN1、PERIOD_W1、PERIOD_D1 的秒长，再乘 2π 取 sin/cos。注意除零保护——x 为 0 时相位直接给 0，避免 NaN 把整行污染。 动作向量 action 用 AddArray 整体追加，随后 Convolution.feedForward 以 GetPointer(State) 推入网络。若返回 false，打印函数名与行号并 break，说明这一条状态被丢弃，不进结果矩阵。 喂完之后 getResults(temp)，再按 Row 写回 result；只有成功写入的才让 state++ 计数。循环结束若 state 与 total_states 不等，用 Reshape 把 result 截到实际有效行数，最后释放 Buffer 返回。 后面 ResearchReward 是另一层：它先校验 embedding 维度与 state_embedding 列数一致，否则直接返回零向量并报警。按 quant 比例取 k 个近邻状态，建临时矩阵和 k+1 行 embedding 矩阵，为后续 reward 估算留接口——外汇与贵金属场景下，这类近端采样对过拟合敏感，回测参数需谨慎。
MQL5 / C++
State.Add(Buffer[tr].States[st].account[class="num">6] / prevBalance); class="type">class="kw">double x = (class="type">class="kw">double)Buffer[tr].States[st].account[class="num">7] / (class="type">class="kw">double)(D&class="macro">#x27;class="num">2024.01.class="num">01&class="macro">#x27; - D&class="macro">#x27;class="num">2023.01.class="num">01&class="macro">#x27;); State.Add((class="type">class="kw">float)MathSin(x != class="num">0 ? class="num">2.0 * M_PI * x : class="num">0)); x = (class="type">class="kw">double)Buffer[tr].States[st].account[class="num">7] / (class="type">class="kw">double)PeriodSeconds(PERIOD_MN1); State.Add((class="type">class="kw">float)MathCos(x != class="num">0 ? class="num">2.0 * M_PI * x : class="num">0)); x = (class="type">class="kw">double)Buffer[tr].States[st].account[class="num">7] / (class="type">class="kw">double)PeriodSeconds(PERIOD_W1); State.Add((class="type">class="kw">float)MathSin(x != class="num">0 ? class="num">2.0 * M_PI * x : class="num">0)); x = (class="type">class="kw">double)Buffer[tr].States[st].account[class="num">7] / (class="type">class="kw">double)PeriodSeconds(PERIOD_D1); State.Add((class="type">class="kw">float)MathSin(x != class="num">0 ? class="num">2.0 * M_PI * x : class="num">0)); State.AddArray(Buffer[tr].States[st].action); if(!Convolution.feedForward(GetPointer(State), class="num">1, class="kw">false, NULL)) { PrintFormat("%s -> %d", __FUNCTION__, __LINE__); break; } Convolution.getResults(temp); if(!result.Row(temp, state)) class="kw">continue; state++; } } if(state != total_states) result.Reshape(state, result.Cols()); ArrayFree(Buffer); class=class="str">"cmt">//--- class="kw">return result; } vector<class="type">class="kw">float> ResearchReward(class="type">class="kw">double quant, vector<class="type">class="kw">float> &embedding, matrix<class="type">class="kw">float> &state_embedding) { vector<class="type">class="kw">float> result = vector<class="type">class="kw">float>::Zeros(NRewards); if(embedding.Size() != state_embedding.Cols()) { PrintFormat("%s -> %d Inconsistent embedding size", __FUNCTION__, __LINE__); class="kw">return result; } class="type">ulong size = embedding.Size(); class="type">ulong states = state_embedding.Rows(); class="type">ulong k = class="type">ulong(states * quant); matrix<class="type">class="kw">float> temp = matrix<class="type">class="kw">float>::Zeros(states, size); vector<class="type">class="kw">float> min_dist = vector<class="type">class="kw">float>::Zeros(k); matrix<class="type">class="kw">float> k_embedding = matrix<class="type">class="kw">float>::Zeros(k + class="num">1, size); matrix<class="type">class="kw">float> U, V;
状态嵌入与每根新K线的特征拼装
这段逻辑干两件事：一是把历史状态嵌入做软距离筛选再降维，给奖励函数喂两个标量；二是 OnTick 里只在新柱生成后，把多指标快照塞进固定宽度的状态向量。外汇与贵金属杠杆高，实盘前务必在策略测试器用历史数据核对维度，避免数组越界静默丢信号。 距离计算先用 temp 存各维度绝对差，取最大差 alpha 当尺度；若全零则置 1 防除零。dist 用 log-sum-exp 近似软最大值，再按分位数 quant 砍掉过远状态，只留 k 个近邻做 SVD，S.Sum() 除以范数乘积得到第一个奖励，第二个直接取潜状态熵。 OnTick 先 IsNewBar 拦截，CopyRates 拉 HistoryBars 根柱，ArraySetAsSeries 倒序后逐个刷 RSI/CCI/ATR/MACD。每根 bar 占 BarDescr=7 个 float：收盘减开盘、最高减开盘、最低减开盘、tick_volume/1000，再加 rsi、cci、atr——任何指标为 EMPTY_VALUE 就跳过该根。 注意 tick_volume 除以 1000.0f 只是缩放，不代表真实成交手数；MT5 模拟盘该值常偏小，黄金跳空时 atr 会瞬间拉大 state[shift+6]，可能扭曲近邻筛选。开 MT5 把 HistoryBars 调到 200 以上，看 sState.state 总长度是否等于 HistoryBars*7 即可验证。
MQL5 / C++
vector<class="type">class="kw">float> S; for(class="type">ulong i = class="num">0; i < size; i++) temp.Col(MathAbs(state_embedding.Col(i) - embedding[i]), i); class="type">class="kw">float alpha = temp.Max(); if(alpha == class="num">0) alpha = class="num">1; vector<class="type">class="kw">float> dist = MathLog(MathExp(temp / (-alpha)).Sum(class="num">1)) * (-alpha); class="type">class="kw">float max = dist.Quantile(quant); for(class="type">ulong i = class="num">0, cur = class="num">0; (i < states && cur < k); i++) { if(max < dist[i]) class="kw">continue; min_dist[cur] = dist[i]; k_embedding.Row(state_embedding.Row(i), cur); cur++; } k_embedding.Row(embedding, k); k_embedding.SVD(U, V, S); result[NRewards - class="num">2] = S.Sum() / (MathSqrt(MathPow(k_embedding, class="num">2.0f).Sum() * MathMax(k + class="num">1, size))); result[NRewards - class="num">1] = EntropyLatentState(Actor); class=class="str">"cmt">//--- class="kw">return result; } class=class="str">"cmt">//+------------------------------------------------------------------+ class=class="str">"cmt">//| Expert tick function | class=class="str">"cmt">//+------------------------------------------------------------------+ class="type">void OnTick() { class=class="str">"cmt">//--- if(!IsNewBar()) class="kw">return; class=class="str">"cmt">//--- class="type">int bars = CopyRates(Symb.Name(), TimeFrame, iTime(Symb.Name(), TimeFrame, class="num">1), HistoryBars, Rates); if(!ArraySetAsSeries(Rates, true)) class="kw">return; class=class="str">"cmt">//--- RSI.Refresh(); CCI.Refresh(); ATR.Refresh(); MACD.Refresh(); Symb.Refresh(); Symb.RefreshRates(); class="type">class="kw">float atr = class="num">0; for(class="type">int b = class="num">0; b < (class="type">int)HistoryBars; b++) { class="type">class="kw">float open = (class="type">class="kw">float)Rates[b].open; class="type">class="kw">float rsi = (class="type">class="kw">float)RSI.Main(b); class="type">class="kw">float cci = (class="type">class="kw">float)CCI.Main(b); atr = (class="type">class="kw">float)ATR.Main(b); class="type">class="kw">float macd = (class="type">class="kw">float)MACD.Main(b); class="type">class="kw">float sign = (class="type">class="kw">float)MACD.Signal(b); if(rsi == EMPTY_VALUE || cci == EMPTY_VALUE || atr == EMPTY_VALUE || macd == EMPTY_VALUE || sign == EMPTY_VALUE) class="kw">continue; class=class="str">"cmt">//--- class="type">int shift = b * BarDescr; sState.state[shift] = (class="type">class="kw">float)(Rates[b].close - open); sState.state[shift + class="num">1] = (class="type">class="kw">float)(Rates[b].high - open); sState.state[shift + class="num">2] = (class="type">class="kw">float)(Rates[b].low - open); sState.state[shift + class="num">3] = (class="type">class="kw">float)(Rates[b].tick_volume / class="num">1000.0f); sState.state[shift + class="num">4] = rsi; sState.state[shift + class="num">5] = cci; sState.state[shift + class="num">6] = atr;
「把持仓与账户状态塞进特征数组」
这段逻辑干的事很直接：先给状态数组补上 MACD 与信号线两维（偏移 +7、+8），再把账户余额、净值写进 sState.account[0] 和 [1]，随后用 PositionsTotal 遍历当前品种的所有持仓。 遍历里按 POSITION_TYPE 分流多空：buy_value / sell_value 累计手数，buy_profit / sell_profit 累计浮动盈亏；同时算出一个 position_discount——用持仓利润减去「当前时间距开仓时间的秒数 × 1/36000 × 利润绝对值」，相当于给长周期持仓的利润打了个时间折扣。 账户数组 [2]~[6] 依次落地多空手数、多空利润、折扣值，[7] 存最新 K 线时间。下面的 bAccount 向量则是相对前一刻的变化率：余额增长率、净值占比、权益波动率、多空利润占余额比、折扣占余额比，共 8 个归一化特征。 最后两行把 K 线时间映射成周期函数：x 先除以 2023→2024 的秒数差再取 sin，再除以月线秒数取 cos，给模型喂了年周期与月周期两个相位量。外汇与贵金属杠杆高，这类特征直接进实盘前务必在 MT5 策略测试器里跑一遍确认维度顺序。
MQL5 / C++
sState.state[shift + class="num">7] = macd; sState.state[shift + class="num">8] = sign; } bState.AssignArray(sState.state); sState.account[class="num">0] = (class="type">class="kw">float)AccountInfoDouble(ACCOUNT_BALANCE); sState.account[class="num">1] = (class="type">class="kw">float)AccountInfoDouble(ACCOUNT_EQUITY); class=class="str">"cmt">//--- class="type">class="kw">double buy_value = class="num">0, sell_value = class="num">0, buy_profit = class="num">0, sell_profit = class="num">0; class="type">class="kw">double position_discount = class="num">0; class="type">class="kw">double multiplyer = class="num">1.0 / (class="num">60.0 * class="num">60.0 * class="num">10.0); class="type">int total = PositionsTotal(); class="type">class="kw">datetime current = TimeCurrent(); for(class="type">int i = class="num">0; i < total; i++) { if(PositionGetSymbol(i) != Symb.Name()) class="kw">continue; class="type">class="kw">double profit = PositionGetDouble(POSITION_PROFIT); class="kw">switch((class="type">int)PositionGetInteger(POSITION_TYPE)) { case POSITION_TYPE_BUY: buy_value += PositionGetDouble(POSITION_VOLUME); buy_profit += profit; break; case POSITION_TYPE_SELL: sell_value += PositionGetDouble(POSITION_VOLUME); sell_profit += profit; break; } position_discount += profit - (current - PositionGetInteger(POSITION_TIME)) * multiplyer * MathAbs(profit); } sState.account[class="num">2] = (class="type">class="kw">float)buy_value; sState.account[class="num">3] = (class="type">class="kw">float)sell_value; sState.account[class="num">4] = (class="type">class="kw">float)buy_profit; sState.account[class="num">5] = (class="type">class="kw">float)sell_profit; sState.account[class="num">6] = (class="type">class="kw">float)position_discount; sState.account[class="num">7] = (class="type">class="kw">float)Rates[class="num">0].time; bAccount.Clear(); bAccount.Add((class="type">class="kw">float)((sState.account[class="num">0] - PrevBalance) / PrevBalance)); bAccount.Add((class="type">class="kw">float)(sState.account[class="num">1] / PrevBalance)); bAccount.Add((class="type">class="kw">float)((sState.account[class="num">1] - PrevEquity) / PrevEquity)); bAccount.Add(sState.account[class="num">2]); bAccount.Add(sState.account[class="num">3]); bAccount.Add((class="type">class="kw">float)(sState.account[class="num">4] / PrevBalance)); bAccount.Add((class="type">class="kw">float)(sState.account[class="num">5] / PrevBalance)); bAccount.Add((class="type">class="kw">float)(sState.account[class="num">6] / PrevBalance)); class="type">class="kw">double x = (class="type">class="kw">double)Rates[class="num">0].time / (class="type">class="kw">double)(D&class="macro">#x27;class="num">2024.01.class="num">01&class="macro">#x27; - D&class="macro">#x27;class="num">2023.01.class="num">01&class="macro">#x27;); bAccount.Add((class="type">class="kw">float)MathSin(class="num">2.0 * M_PI * x)); x = (class="type">class="kw">double)Rates[class="num">0].time / (class="type">class="kw">double)PeriodSeconds(PERIOD_MN1); bAccount.Add((class="type">class="kw">float)MathCos(class="num">2.0 * M_PI * x));
◍ 把网络输出换算成真实下单手数
强化学习 Actor 推完前向之后，拿到的 temp 向量并不是直接能用的手数，而是买/卖各自的仓位权重与 TP、SL 档位。代码里先把买权与卖权做互斥抵消：temp[0] 和 temp[3] 谁大就减谁，保证同一时刻不会两头都开仓，这是避免自成交的最直接一道闸。 真正下单前还要过经纪商门槛。min_lot 取 Symb.LotsMin()，step_lot 取 LotsStep()，stops 用 StopsLevel 和点值算最小止损距离；只要 temp[0] 小于最小手数，或 TP/SL 换算后点数不够 stops，就触发 CloseByDirection 平掉已有买仓，不勉强交易。外汇与贵金属杠杆高，这类硬约束踩漏就可能直接报单失败。 通过校验后，buy_lot 用 MathRound((temp[0]-min_lot)/step_lot)*step_lot 对齐步长，TP 挂在 Ask+temp[1]*MaxTP*Point，SL 挂在 Ask-temp[2]*MaxSL*Point。若已持买仓，先 TrailPosition 移动止损止盈；仓位不相等时，多出部分 ClosePartial 减仓，不足部分 Trade.Buy 补单。卖侧逻辑镜像处理 temp[3]~temp[5]，用 Bid 而非 Ask 计算挂单价。 开 MT5 把这段接在你自己的 Actor 后处理里，先打印 temp 和 buy_lot 看步长对齐是否如预期，再决定要不要放开实盘。
MQL5 / C++
x = (class="type">class="kw">double)Rates[class="num">0].time / (class="type">class="kw">double)PeriodSeconds(PERIOD_W1); bAccount.Add((class="type">class="kw">float)MathSin(class="num">2.0 * M_PI * x)); x = (class="type">class="kw">double)Rates[class="num">0].time / (class="type">class="kw">double)PeriodSeconds(PERIOD_D1); bAccount.Add((class="type">class="kw">float)MathSin(class="num">2.0 * M_PI * x)); if(bAccount.GetIndex() >= class="num">0) if(!bAccount.BufferWrite()) class="kw">return; class=class="str">"cmt">//--- if(!Actor.feedForward(GetPointer(bState), class="num">1, class="kw">false, GetPointer(bAccount))) class="kw">return; PrevBalance = sState.account[class="num">0]; PrevEquity = sState.account[class="num">1]; class=class="str">"cmt">//--- vector<class="type">class="kw">float> temp; Actor.getResults(temp); class=class="str">"cmt">//--- class="type">class="kw">double min_lot = Symb.LotsMin(); class="type">class="kw">double step_lot = Symb.LotsStep(); class="type">class="kw">double stops = MathMax(Symb.StopsLevel(), class="num">1) * Symb.Point(); if(temp[class="num">0] >= temp[class="num">3]) { temp[class="num">0] -= temp[class="num">3]; temp[class="num">3] = class="num">0; } else { temp[class="num">3] -= temp[class="num">0]; temp[class="num">0] = class="num">0; } class=class="str">"cmt">//--- buy control if(temp[class="num">0] < min_lot || (temp[class="num">1] * MaxTP * Symb.Point()) <= stops || (temp[class="num">2] * MaxSL * Symb.Point()) <= stops) { if(buy_value > class="num">0) CloseByDirection(POSITION_TYPE_BUY); } else { class="type">class="kw">double buy_lot = min_lot + MathRound((class="type">class="kw">double)(temp[class="num">0] - min_lot) / step_lot) * step_lot; class="type">class="kw">double buy_tp = NormalizeDouble(Symb.Ask() + temp[class="num">1] * MaxTP * Symb.Point(), Symb.Digits()); class="type">class="kw">double buy_sl = NormalizeDouble(Symb.Ask() - temp[class="num">2] * MaxSL * Symb.Point(), Symb.Digits()); if(buy_value > class="num">0) TrailPosition(POSITION_TYPE_BUY, buy_sl, buy_tp); if(buy_value != buy_lot) { if(buy_value > buy_lot) ClosePartial(POSITION_TYPE_BUY, buy_value - buy_lot); else Trade.Buy(buy_lot - buy_value, Symb.Name(), Symb.Ask(), buy_sl, buy_tp); } } class=class="str">"cmt">//--- sell control if(temp[class="num">3] < min_lot || (temp[class="num">4] * MaxTP * Symb.Point()) <= stops || (temp[class="num">5] * MaxSL * Symb.Point()) <= stops) { if(sell_value > class="num">0) CloseByDirection(POSITION_TYPE_SELL); } else {
把轨迹回采交给小布
初步模型训完后再往训练集灌轨迹这套重复劳动，小布盯盘的 AIGC 流程已内置，打开对应品种页就能自动跑，你只管看分布偏移是否收敛。
常见问题
离线环境信息被训练集规模锁死，数据多在窄子空间收集，代理决策越界后结果难预测，原探索激励失效。
主要三阶段：无监督收集未标记探索数据、用历史策略存每回合状态行动序列、填满受限训练集后用于后续训练。
涵盖随机基线、最大化预测误差（ICM/分歧/RND）、覆盖估计（APT/Proto-RL）及基于能力学技能（DIAYN/SMM 等）共九种。
可以，小布把分布外轨迹检测和训练集覆盖可视化做成现成模块，省去自己写脚本比对子空间缝隙。
受可用算力资源和与环境交互成本约束，往往没法无节制扩样，这也是 ExORL 重数据质量而非数量的原因。