📘
您应当知道的 MQL5 向导技术（第 51 部分）：配以 SAC 的强化学习·进阶篇
「用双评论员网络给交易动作打分」
这段脚本搭的是一个离在线混合的强化学习骨架：评论员（critic）接收「状态+动作」拼接后的张量，经两层 ReLU 全连接（hidden_units 宽）输出单个 Q 值，用来估计某状态下采取某动作的期望回报。代码里直接建了 critic1 与 critic2 两个结构相同的网络，输入维度 input_dim=1、输出维度 output_dim=3，对应买、卖、持三种离散动作。 数据切分上，CSV 读入后只留 <STATE> 与 <REWARD> 两列，训练集占总量 2/3（states_size = len(states)*2/3），测试集取剩余 1/3。三个 Adam 优化器学习率都设 0.001，actor 与两个 critic 各自独立更新，这种双评论员设定倾向缓解单网络过估 Q 值的问题。 训练循环按 epoch_size 跑，每个样本用 tf.GradientTape(persistent=True) 同时记录 actor 与 critic 梯度。actor 输出先拆成 mu 与 log_std 各 3 维，用 tf.exp 转 std 后从高斯分布采样动作；状态与采样动作转 float32 张量后再拼给 critic 算 Q。外汇与贵金属杠杆高、滑点跳空频繁，这类模型在 MT5 历史数据上回测可能盈利，但实盘概率分布漂移大，须以小资金验证。
MQL5 / C++
state_action_inputs = layers.Input(shape=(None, input_size, class="num">1)) # Concatenate state and action # Hidden layers(Dense layers with ReLU activation) x = layers.Dense(hidden_units, activation=&class="macro">#x27;relu&class="macro">#x27;)(state_action_inputs) x = layers.Dense(hidden_units, activation=&class="macro">#x27;relu&class="macro">#x27;)(x) # Output layer: Q-value for the given state-action pair q_value_output = layers.Dense(class="num">1)(x) # Single output for Q-value # Create the model critic_model = tf.keras.Model(inputs=state_action_inputs, outputs=q_value_output) class="kw">return critic_model # Filter the DataFrame to keep only the &class="macro">#x27;<state>&class="macro">#x27; column df = pd.read_csv(name_csv) states = df.filter([&class="macro">#x27;<STATE>&class="macro">#x27;]).astype(class="type">int).values # Extract the &class="macro">#x27;<state>&class="macro">#x27; column as an integer array rewards = df.filter([&class="macro">#x27;<REWARD>&class="macro">#x27;]).values states_size = class="type">int(len(states)*(class="num">2.0/class="num">3.0)) actor_x_train = states[class="num">0:states_size,:] actor_x_test = states[states_size:,:class="num">1] rewards_size = class="type">int(len(rewards)*(class="num">2.0/class="num">3.0)) critic_y_train = rewards[class="num">0:rewards_size,:] critic_y_test = rewards[rewards_size:,:class="num">1] # Initialize networks and optimizers input_dim = class="num">1 # class="num">2 states, of class="num">3 gradations are flattened into a single index output_dim = class="num">3 # possible actions buy, sell, hold actor = ActorNetwork(input_dim, output_dim) critic1 = CriticNetwork(input_dim, output_dim) # Input paired with action critic2 = CriticNetwork(input_dim, output_dim) # Input paired with action critic_optimizer_1 = tf.keras.optimizers.Adam(learning_rate=class="num">0.001) critic_optimizer_2 = tf.keras.optimizers.Adam(learning_rate=class="num">0.001) actor_optimizer = tf.keras.optimizers.Adam(learning_rate=class="num">0.001) # Training loop for e in range(epoch_size): train_critic_loss1 = class="num">0 train_critic_loss2 = class="num">0 train_actor_loss = class="num">0 for i in range(actor_x_train.shape[class="num">0]): input_state = tf.expand_dims(actor_x_train[i], axis=class="num">0) # Select a single sample and maintain batch dim target_q = tf.expand_dims(critic_y_train[i], axis=class="num">0) # Actor forward pass and sampling with tf.GradientTape(persistent=True) as tape: actor_output = actor(input_state) # Split the vector into mean and log_std mu = actor_output[:, :output_dim] # First class="num">3 values log_std = actor_output[:, output_dim:] # Last class="num">3 values std = tf.exp(log_std) sampled_action = tf.random.normal(shape=mu.shape, mean=mu, stddev=std) # Sample action from Gaussian # Concatenate the state and action tensors in_state = tf.convert_to_tensor(tf.cast(input_state, dtype=tf.float32), dtype=tf.float32) # Ensure it&class="macro">#x27;s a tensor in_action = tf.convert_to_tensor(tf.cast(sampled_action, dtype=tf.float32), dtype=tf.float32) # Concatenate along the last axis
双评论员与策略网络的梯度回传细节
这段训练循环把状态与动作在最后一维拼起来，得到形状为 [batch, 4] 的原始 critic 输入，再 reshape 成 [-1,1,4,1] 的四维张量喂给两个 critic 网络。用 -1 让 batch 维动态推断，避免写死样本数，在 MT5 导出的不同批量回测里都能直接复用。 两个 critic 各自算 MSE 损失：critic_loss = mean((Q预测 - target_q)^2)，而 actor 的损失取 min(q_value1, q_value2) 的均值，用负号把「最大化 Q」翻转为优化器能最小化的目标。这是 TD3 抑制过估计的核心一步，外汇与贵金属行情噪声大，过估计会让策略在实盘倾向于重仓追单，风险偏高。 梯度通过 persistent tape 取出后分别 apply，actor 的梯度手动取负再更新。训练完按 actor_x_train.shape[0] 归一打印 loss，数值保留 4 位小数，方便你横向比对不同 epoch 的收敛斜率。 最后用 onnxruntime 载回三个已导出的 onnx，逐个打印 input 的 name/shape/type。若 shape 显示 [1,1,4,1] 且 type 为 float32，说明 MT5 的 ONNX 模型导入接口能对齐；若 name 带多余后缀，需在导出时改掉，否则 EA 调用会报 input mismatch。
MQL5 / C++
critic_raw_input = tf.concat([in_state, in_action], axis=-class="num">1) # Ensure correct axis critic_input = tf.reshape(critic_raw_input, [-class="num">1, class="num">1, class="num">4, class="num">1]) # -class="num">1 infers the batch size dynamically q_value1 = critic1(critic_input) q_value2 = critic2(critic_input) # Critic loss(mean squared error) critic_loss1 = tf.reduce_mean((tf.cast(q_value1, tf.float32) - tf.cast(target_q, tf.float32)) ** class="num">2) critic_loss2 = tf.reduce_mean((tf.cast(q_value2, tf.float32) - tf.cast(target_q, tf.float32)) ** class="num">2) # Actor loss(maximize expected Q-value based on minimum critic output) min_q_value = tf.minimum(q_value1, q_value2) # Take the minimum Q-value actor_loss = tf.reduce_mean(min_q_value) # Maximize expected Q-value(negative for minimization) # Backpropagation critic_gradients1 = tape.gradient(critic_loss1, critic1.trainable_variables) critic_gradients2 = tape.gradient(critic_loss2, critic2.trainable_variables) actor_gradients = [-grad for grad in tape.gradient(actor_loss, actor.trainable_variables)] del tape # Free up resources from persistent GradientTape critic_optimizer_1.apply_gradients(zip(critic_gradients1, critic1.trainable_variables)) critic_optimizer_2.apply_gradients(zip(critic_gradients2, critic2.trainable_variables)) actor_optimizer.apply_gradients(zip(actor_gradients, actor.trainable_variables)) # Accumulate losses for epoch summary train_critic_loss1 += critic_loss1.numpy() train_critic_loss2 += critic_loss2.numpy() train_actor_loss += actor_loss.numpy() print(f" Epoch {e + class="num">1}/{epoch_size}:") print(f" Train Critic Loss class="num">1: {train_critic_loss1 / actor_x_train.shape[class="num">0]:.4f}") print(f" Train Critic Loss class="num">2: {train_critic_loss2 / actor_x_train.shape[class="num">0]:.4f}") print(f" Train Actor Loss: {train_actor_loss / actor_x_train.shape[class="num">0]:.4f}") print("-" * class="num">40) critic2.summary() critic1.summary() actor.summary() # Check input and output layer shapes for importing ONNX class="kw">import onnxruntime as ort session_critic2 = ort.InferenceSession(path_critic2_onnx) session_critic1 = ort.InferenceSession(path_critic1_onnx) session_actor = ort.InferenceSession(path_actor_onnx) for i in session_critic2.get_inputs(): print(f"in critic2 Name: {i.name}, Shape: {i.shape}, Type: {i.type}") for i in session_critic1.get_inputs(): print(f"in critic1 Name: {i.name}, Shape: {i.shape}, Type: {i.type}") for i in session_actor.get_inputs(): print(f"in actor Name: {i.name}, Shape: {i.shape}, Type: {i.type}")
◍ 把三个推理会话的输出张量打印出来
在 MT5 的 Python 环境里，强化学习模型常被拆成 actor 与两个 critic 网络分别加载。要确认它们前向推理真的跑通了，最直接的方式是遍历各自会话的 outputs 并打印元信息。 下面这段代码依次取出 session_critic2、session_critic1、session_actor 的输出节点，把每个节点的 name、shape、type 打到控制台。shape 能告诉你输出是动作概率分布还是价值标量，type 则暴露底层张量精度，比如 float32 还是 float16。 实盘前在脚本里跑一遍，若 actor 输出 shape 为 (1, N) 且 type 为 float32，基本说明策略头已就位；若某个 critic 的 shape 对不上预期，多半是 onnx 导出时少了残差连接，得回训练侧查图。外汇与贵金属杠杆高，模型推理异常可能直接放大下单偏差，验证环节不能跳过。
MQL5 / C++
for o in session_critic2.get_outputs(): print(f"out critic2 Name: {o.name}, Shape: {o.shape}, Type: {o.type}") for o in session_critic1.get_outputs(): print(f"out critic1 Name: {o.name}, Shape: {o.shape}, Type: {o.type}") for o in session_actor.get_outputs(): print(f"out actor Name: {o.name}, Shape: {o.shape}, Type: {o.type}")
「把 ONNX 模型塞进 MT5 信号类」
做 EURUSD 的 H1 强化学习信号，先得把 Python 训好的模型当资源导进 MQL5。原文用的数据是 2023.12.12 到 2024.12.12 共一年 H1 行情，其中前三分之二（2023.12.12–2024.08.12，约八个月）拿去训练，剩下 2024.08.12 之后的四个多月做前向测试。H1 级别下四个月样本不算长，但足够在 MT5 里跑出可观察的行为偏差。 反向传播已在 Python 完成，所以信号类里没有留给优化器调的特殊参数，模型权重是冻结的。导入时要注意 ONNX 层形状得迁就导出逻辑：Actor 网络本来该输出均值向量加对数标准差两个向量，但拆开定义容易出错，所以在 Python 端先拼成单向量；Critic 网络原应接环境状态和动作分布两个张量，也压成一个长度为 4 的向量。 初始化时务必校验每个 ONNX 的输入输出层尺寸，否则 MT5 加载会直接报模型不兼容。下面这段信号类标头展示了三个 .onnx 资源挂载和动作/环境数宏定义，动作数 3、环境数 3 与之前保持一致。 用向导编译后，对数据窗口剩余 4 个月做测试运行，能直接看到信号在真实 H1 柱上的触发分布。外汇和贵金属杠杆高，这类 RL 信号仅作辅助参考，实盘前请在策略测试器用历史数据复核。
MQL5 / C++
class=class="str">"cmt">//+------------------------------------------------------------------+ class=class="str">"cmt">//| SignalSAC.mqh | class=class="str">"cmt">//| Copyright class="num">2009-class="num">2017, MetaQuotes Software Corp. | class=class="str">"cmt">//| [MQL5官方文档] | class=class="str">"cmt">//+------------------------------------------------------------------+ class="macro">#include <Expert\ExpertSignal.mqh> class="macro">#include <My\Cql.mqh> class="macro">#resource "Python/EURUSD_H1_D1_critic2.onnx" as class="type">uchar __CRITIC_2[] class="macro">#resource "Python/EURUSD_H1_D1_critic1.onnx" as class="type">uchar __CRITIC_1[] class="macro">#resource "Python/EURUSD_H1_D1_actor.onnx" as class="type">uchar __ACTOR[] class="macro">#define __ACTIONS class="num">3 class="macro">#define __ENVIONMENTS class="num">3 class=class="str">"cmt">//+------------------------------------------------------------------+ class=class="str">"cmt">//| SACs CSignalSAC. | class=class="str">"cmt">//| Purpose: Soft Actor Critic for Reinforcement-Learning. | class=class="str">"cmt">//| Derives from class CExpertSignal. | class=class="str">"cmt">//+------------------------------------------------------------------+ class=class="str">"cmt">//+------------------------------------------------------------------+ class=class="str">"cmt">//| | class=class="str">"cmt">//+------------------------------------------------------------------+ class CSignalSAC : class="kw">public CExpertSignal { class="kw">protected: class="type">long m_critic_2_handle; class="type">long m_critic_1_handle; class="type">long m_actor_handle; class="kw">public: class="type">void CSignalSAC(class="type">void); class="type">void ~CSignalSAC(class="type">void); class=class="str">"cmt">//--- methods of setting adjustable parameters class=class="str">"cmt">//--- method of verification of arch class="kw">virtual class="type">bool ValidationSettings(class="type">void); class=class="str">"cmt">//--- method of creating the indicator and timeseries
把强化学习模型塞进 MT5 前的闸门
CSignalSAC 在真正跑模型之前，用 ValidationSettings 把环境和 ONNX 句柄全部过一遍。基类 CExpertSignal::ValidationSettings 先返回 false 的话，后面全免谈，这是第一道继承链上的关卡。 时间框架被硬性卡在 PERIOD_H1 以内，m_period 一旦大于 H1 就直接 Print(" time frame too large ") 并退出。做外汇或贵金属的朋友注意，这种 SAC 信号在高于一小时的周期上未经校验，实盘漂移概率倾向升高，属高风险用法。 三个 ONNX 模型句柄——actor、critic_1、critic_2——任何一个等于 INVALID_HANDLE 都会打印具体错误码并 return false。这意味着你丢进缓冲区的模型文件只要有一个没加载成功，信号类直接瘫痪，不会静默运行。 输入输出维度也写死在代码里：critic 吃 {1,4,1} 出 {1,4,1,1}，actor 吃 {1} 出 {1,6}。开 MT5 把这段抄进自己的信号类，改 _actor_out_shape 的 6 之前先想清楚动作空间是不是真有六种，否则 OnnxSetOutputShape 会给你报错而不是瞎跑。
MQL5 / C++
class="kw">virtual class="type">bool InitIndicators(CIndicators *indicators); class=class="str">"cmt">//--- methods of checking if the market models are formed class="kw">virtual class="type">int LongCondition(class="type">void); class="kw">virtual class="type">int ShortCondition(class="type">void); class="kw">protected: vectorf GetOutput(); vectorf LogProbabilities(vectorf &Mean, vectorf &Log_STD); }; class=class="str">"cmt">//+------------------------------------------------------------------+ class=class="str">"cmt">//| Validation arch class="kw">protected data. | class=class="str">"cmt">//+------------------------------------------------------------------+ class="type">bool CSignalSAC::ValidationSettings(class="type">void) { if(!CExpertSignal::ValidationSettings()) class="kw">return(false); class=class="str">"cmt">//--- initial data checks if(m_period > PERIOD_H1) { Print(" time frame too large "); class="kw">return(false); } ResetLastError(); if(m_critic_2_handle == INVALID_HANDLE) { Print("Crit class="num">2 OnnxCreateFromBuffer error ", GetLastError()); class="kw">return(false); } if(m_critic_1_handle == INVALID_HANDLE) { Print("Crit class="num">1 OnnxCreateFromBuffer error ", GetLastError()); class="kw">return(false); } if(m_actor_handle == INVALID_HANDLE) { Print("Actor OnnxCreateFromBuffer error ", GetLastError()); class="kw">return(false); } class=class="str">"cmt">// Set input shapes const class="type">long _critic_in_shape[] = {class="num">1, class="num">4, class="num">1}; const class="type">long _actor_in_shape[] = {class="num">1}; class=class="str">"cmt">// Set output shapes const class="type">long _critic_out_shape[] = {class="num">1, class="num">4, class="num">1, class="num">1}; const class="type">long _actor_out_shape[] = {class="num">1, class="num">6}; if(!OnnxSetInputShape(m_critic_2_handle, ONNX_DEFAULT, _critic_in_shape)) { Print("Crit class="num">2 OnnxSetInputShape error ", GetLastError()); class="kw">return(false); } if(!OnnxSetOutputShape(m_critic_2_handle, class="num">0, _critic_out_shape)) { Print("Crit class="num">2 OnnxSetOutputShape error ", GetLastError()); class="kw">return(false); } if(!OnnxSetInputShape(m_critic_1_handle, ONNX_DEFAULT, _critic_in_shape)) { Print("Crit class="num">1 OnnxSetInputShape error ", GetLastError()); class="kw">return(false); } if(!OnnxSetOutputShape(m_critic_1_handle, class="num">0, _critic_out_shape)) { Print("Crit class="num">1 OnnxSetOutputShape error ", GetLastError());
常见问题
两个评论员各算一个 Q 值，取较小值作为动作评分，避免单一网络过度乐观导致策略选到差动作。
要分开。策略网络用评论员最小值反传梯度更新，两个评论员各自用 TD 误差独立更新，别混成一个优化器。
可以。小布能按你设的阈值把模型输出的动作概率和评论员评分标红，只把临界信号推给你复核。
在信号类里分别缓存策略、评论员A、评论员B的输出数组，用日志按 bar 序号逐行落盘，对比维度是否对齐。
闸门做输入归一化范围和输出合法值校验，境外品种波动大，漏掉这步容易在极端行情发出无效下单。