掌握ONNX:MQL5交易者的游戏规则改变者·进阶篇
(2/3)·从格式原理到数据规范,绕开在MQL5里重造轮子的坑
抓 EURUSD H1 的 ONNX 训练与实时张量
在 MT5 里接 ONNX 模型做 EURUSD H1 推理时,先得把历史窗口喂成矩阵。日志里同一秒(23:10:54.837)打了 7 组 Train data,分别是 HF 到 NL 的 4 维向量,比如 HF 为 [1.3976599,1.3894916,1.4053394,1.11189],PM 为 [1.3848507,1.3761013,1.3718294,1.11022],第四维都落在 1.109~1.112 的窄区,前三维是不同价位的归一化值。 实时侧用 matrixf live_data = GetLiveData(0,1) 取最新一根,Print 出来是 [[-0.21183228,-0.23540309,-0.20334835]],三个负向分量说明当下价位相对训练分布偏低,EURUSD 短线可能偏弱,但外汇高杠杆下这只是概率倾向。 坑在参数尺寸:2023.09.18 18:03:53.212 报 ONNX: invalid parameter size, expected 1044480 bytes instead of 32640。模型期望的输入字节数是实际传的 32 倍,通常是窗口长度或 batch 维没对齐,开 MT5 查模型输入输出节点 shape 就能复现。
matrixf live_data = GetLiveData(class="num">0,class="num">1); Print("Live data\n",live_data);
「训练与实盘必须用同一套归一参数」
给机器学习模型喂数据前,规范化是绕不开的一步。最容易被忽略的坑是:训练集用的归一方式(比如 MinMaxScaler)必须原样套到测试集和实时行情上,min、max 或 mean、std 这些统计量不能重新算,否则模型接到的分布会漂移,预测倾向失真。 只规范自变量,目标变量(label)保持原值,不管做的是分类还是回归都别动它。我们在 MT5 侧用 Preprocessing 类统一处理,并把每种 scaler 的参数落盘成 csv,供 Python 端复用同一组数值。 下面这段 switch 按 NORM 宏分支把均值归一、min-max、标准化三类的参数分别写文件:均值归一存 mean/min/max,min-max 存 min/max,标准化存 mean/std。注意 GetTrainData 里取完 OHLC 后直接拼成矩阵,归一逻辑集成在 GetData 类内部,保证任何出口拿到的都是规范后的价格矩阵。 外汇与贵金属波动受杠杆和消息面影响,高频重算归一参数可能放大过拟合风险,实盘前建议在策略测试器跑一轮不同品种的 csv 一致性校验。
class=class="str">"cmt">//--- Saving the normalization prameters class="kw">switch(NORM) { case NORM_MEAN_NORM: class=class="str">"cmt">//--- saving the mean norm_params.Assign(norm_x.mean_norm_scaler.mean); WriteCsv(normparams_folder+csv_name_+".mean_norm_scaler.mean.csv",norm_params,x_vars); class=class="str">"cmt">//--- saving the min norm_params.Assign(norm_x.mean_norm_scaler.min); WriteCsv(normparams_folder+csv_name_+".mean_norm_scaler.min.csv",norm_params,x_vars); class=class="str">"cmt">//--- saving the max norm_params.Assign(norm_x.mean_norm_scaler.max); WriteCsv(normparams_folder+csv_name_+".mean_norm_scaler.max.csv",norm_params,x_vars); break; case NORM_MIN_MAX_SCALER: class=class="str">"cmt">//--- saving the min norm_params.Assign(norm_x.min_max_scaler.min); WriteCsv(normparams_folder+csv_name_+".min_max_scaler.min.csv",norm_params,x_vars); class=class="str">"cmt">//--- saving the max norm_params.Assign(norm_x.min_max_scaler.max); WriteCsv(normparams_folder+csv_name_+".min_max_scaler.max.csv",norm_params,x_vars); break; case NORM_STANDARDIZATION: class=class="str">"cmt">//--- saving the mean norm_params.Assign(norm_x.standardization_scaler.mean); WriteCsv(normparams_folder+csv_name_+".standardization_scaler.mean.csv",norm_params,x_vars); class=class="str">"cmt">//--- saving the std norm_params.Assign(norm_x.standardization_scaler.std); WriteCsv(normparams_folder+csv_name_+".standardization_scaler.std.csv",norm_params,x_vars); break; } matrixf GetTrainData(class="type">uint start, class="type">uint total) { matrixf return_matrix(total, class="num">3); class="type">ulong last_col; OPEN.CopyRates(Symbol(), PERIOD_CURRENT, COPY_RATES_OPEN, start, total); HIGH.CopyRates(Symbol(), PERIOD_CURRENT, COPY_RATES_HIGH, start, total); LOW.CopyRates(Symbol(), PERIOD_CURRENT, COPY_RATES_LOW, start, total); CLOSE.CopyRates(Symbol(), PERIOD_CURRENT, COPY_RATES_CLOSE, start, total); return_matrix.Col(OPEN, class="num">0); return_matrix.Col(HIGH, class="num">1); return_matrix.Col(LOW, class="num">2); matrixf norm_params = {}; csv_name_ = Symbol()+"."+EnumToString(Period())+"."+class="type">class="kw">string(total_bars);
◍ 把归一化参数落盘再拼回收盘价列
做特征工程时最容易被忽略的一步:归一化对象销毁前必须把 scaler 参数存下来,否则换周期重跑模型会尺度错乱。下面这段代码在删除旧的 norm_x 指针后,新建 CPreprocessing 实例,并按 NORM 分支把 mean/min/max/std 分别写成 CSV,列名绑定 OPEN,HIGH,LOW 三个变量。 x_vars = "OPEN,HIGH,LOW"; 声明后续 CSV 表头只覆盖这三列行情字段。 while (CheckPointer(norm_x) != POINTER_INVALID) delete (norm_x); 先清掉上一次残留的归一化对象,防止内存泄漏与旧参数串味。 norm_x = new CPreprocessing<vectorf, matrixf>(return_matrix, NORM); 用当前 return_matrix 和指定 NORM 模式重建预处理器。 switch 里三种归一化各自落盘:NORM_MEAN_NORM 存 mean/min/max 三个文件;NORM_MIN_MAX_SCALER 只存 min/max;NORM_STANDARDIZATION 存 mean/std。每个文件都通过 WriteCsv 写到 normparams_folder,文件名带 csv_name_ 前缀,方便小布后续按品种+周期追溯。 return_matrix.Resize(total, 4); 把矩阵扩成 4 列,多出来的最后一列留给标签。last_col = return_matrix.Cols()-1; 取到末列索引,return_matrix.Col(CLOSE, last_col); 把收盘价塞进末列,csv_name_ 追加 .targ=CLOSE,表头变成 OPEN,HIGH,LOW,CLOSE。最后写进 ONNX Datafolder,失败就 Print 报警——外汇与贵金属数据高频跳空,这种落盘失败在实盘前必须排查清楚,否则训练集可能缺了目标列。
x_vars = "OPEN,HIGH,LOW"; while (CheckPointer(norm_x) != POINTER_INVALID) class="kw">delete (norm_x); norm_x = new CPreprocessing<vectorf, matrixf>(return_matrix, NORM); class=class="str">"cmt">//--- Saving the normalization prameters class="kw">switch(NORM) { case NORM_MEAN_NORM: class=class="str">"cmt">//--- saving the mean norm_params.Assign(norm_x.mean_norm_scaler.mean); WriteCsv(normparams_folder+csv_name_+".mean_norm_scaler.mean.csv",norm_params,x_vars); class=class="str">"cmt">//--- saving the min norm_params.Assign(norm_x.mean_norm_scaler.min); WriteCsv(normparams_folder+csv_name_+".mean_norm_scaler.min.csv",norm_params,x_vars); class=class="str">"cmt">//--- saving the max norm_params.Assign(norm_x.mean_norm_scaler.max); WriteCsv(normparams_folder+csv_name_+".mean_norm_scaler.max.csv",norm_params,x_vars); break; case NORM_MIN_MAX_SCALER: class=class="str">"cmt">//--- saving the min norm_params.Assign(norm_x.min_max_scaler.min); WriteCsv(normparams_folder+csv_name_+".min_max_scaler.min.csv",norm_params,x_vars); class=class="str">"cmt">//--- saving the max norm_params.Assign(norm_x.min_max_scaler.max); WriteCsv(normparams_folder+csv_name_+".min_max_scaler.max.csv",norm_params,x_vars); break; case NORM_STANDARDIZATION: class=class="str">"cmt">//--- saving the mean norm_params.Assign(norm_x.standardization_scaler.mean); WriteCsv(normparams_folder+csv_name_+".standardization_scaler.mean.csv",norm_params,x_vars); class=class="str">"cmt">//--- saving the std norm_params.Assign(norm_x.standardization_scaler.std); WriteCsv(normparams_folder+csv_name_+".standardization_scaler.std.csv",norm_params,x_vars); break; } return_matrix.Resize(total, class="num">4); class=class="str">"cmt">//if we are collecting the train data collect the target variable also last_col = return_matrix.Cols()-class="num">1; class=class="str">"cmt">//Column located at the last index is the last column return_matrix.Col(CLOSE, last_col); class=class="str">"cmt">//put the close price information in the last column of a matrix csv_name_ +=".targ=CLOSE"; csv_header = x_vars + ",CLOSE"; if (!WriteCsv("ONNX Datafolder\\"+csv_name_+".csv", return_matrix, csv_header)) Print("Failed to Write to a csv file"); else
把回测结果落盘到 CSV 再读回
上面这段收尾代码只做一件事:把矩阵算完之后,往终端日志打一行 "Data saved to a csv file successfully",然后 return return_matrix 把结果交回去给调用方。 真正落地时,你得在 Print 之前先把 return_matrix 用 FileOpen / FileWrite 写进 MT5 的 MQL5\Files 目录下的 csv,否则只靠 Print 没法拿去 Excel 做分布检验。外汇与贵金属品种点差跳空多,回测矩阵建议至少跑 500 根以上日线再导出,样本太少正态假设会塌。 打开 MT5 按 F4 进 MetaEditor,把这段嵌进你自己的指标或 EA 末尾,编译后切到「专家」标签页就能看到那行成功提示,证明数据已就绪可进一步处理。
Print("Data saved to a csv file successfully"); class="kw">return return_matrix; }
「用 Python 把 MLP 训成 ONNX 给 MT5 调」
先隔离环境再写模型,能少踩很多坑。Windows 下从 CMD 跑 pip3 install virtualenv、virtualenv venv、venv\Scripts\activate 三连,就能建出独立虚拟环境,避免不同项目的包版本互相打架。依赖统一放 requirements.txt,pip install -r 一把装齐,Mac 和 Linux 的激活命令略有差异,但思路一致。 MT5 这侧要先 initialize() 拉起终端,收尾记得 shutdown() 释放。数据不从网络抓,而是直接读 terminal_info.data_path 拼出 \\MQL5\\Files\\ONNX Datafolder——若 os.path.exists 为假,说明 MT5 端还没落盘数据,脚本该直接 quit() 而不是硬跑。 模型本体包成一个类,__init__ 里把 CSV 读进 pandas,按 target_column 拆 X/Y,用 train_test_split 以 0.3 测试比、random_state=42 切分。输入维度取 train_x.shape[-1],输出恒为 1 个神经元,对应回归式预测。 BuildNeuralNetwork 用 keras.Sequential 堆单层全连接,relu 激活、默认 10 神经元即可跑通。实际训练中准确率约 93%,样本外测试约 95%,存在过拟合倾向,但作为原型够用。满意后装 onnx 与 tf2onnx,在 train_network 里导出 ONNX 存进 Files 父目录——这样后续 EA 或指标把模型当资源挂进去最省事。外汇与贵金属波动剧烈,此类模型仅作概率参考,实盘前务必在 MT5 策略测试器复核。
$ pip3 install virtualenv $ virtualenv venv $ venv\Scripts\activate class="kw">import MetaTrader5 as mt5 if not mt5.initialize(): class="macro">#This will open MT5 app in your pc print("initialize() failed, error code =",mt5.last_error()) quit() # program logic and ML code will be here mt5.shutdown() class="macro">#This closes the program # Getting the data we stored in the Files path on Metaeditor data_path = terminal_info.data_path dataset_path = data_path + "\MQL5\Files\ONNX Datafolder" class="kw">import os if not os.path.exists(dataset_path): print("Dataset folder doesn&class="macro">#x27;t exist | Be sure you are referring to the correct path and the data is collected from MT5 side of things") quit() class NeuralNetworkClass(): def __init__(self, csv_name, target_column, batch_size=class="num">32): # Loading the dataset and storing to a variable Array self.data = pd.read_csv(dataset_path+"\"+csv_name) if self.data.empty: print(f"No such dataset or Empty dataset csv = {csv_name}") quit() # quit the program print(self.data.head()) # Print class="num">5 first rows of a given data self.target_column = target_column # spliting the data into training and testing samples X = self.data.drop(columns=self.target_column).to_numpy() # droping the targeted column, the rest is x variables Y = self.data[self.target_column].to_numpy() # We convert data arrays to numpy arrays compartible with sklearn and tensorflow self.train_x, self.test_x, self.train_y, self.test_y = train_test_split(X, Y, test_size=class="num">0.3, random_state=class="num">42) # splitting the data into training and testing samples print(f"train x shape {self.train_x.shape} test x shape {self.test_x.shape}") self.input_size = self.train_x.shape[-class="num">1] # obtaining the number of columns in x variable as our inputs self.output_size = class="num">1 # We are solving for a regression problem we need to have a single output neuron self.batch_size = batch_size self.model = None # Object to store the model self.plots_directory = "Plots" self.models_directory = "Models" def BuildNeuralNetwork(self, activation_function=&class="macro">#x27;relu&class="macro">#x27;, neurons = class="num">10): # Create a Feedforward Neural Network model self.model = keras.Sequential([ keras.layers.Input(shape=(self.input_size,)), # Input layer
◍ 训练回路与早停机制怎么落地
模型搭完之后真正跑起来的是 train_network 这个方法。默认跑 100 个 epoch,学习率 0.001,损失函数用 mean_squared_error,优化器选 Adam;这些参数在调用时都能改,外汇和贵金属序列噪声大,学习率设太高容易过拟合,实盘前建议在历史数据上先扫一遍。 早停靠 EarlyStopping 盯着 val_loss,patience=10 意味着连续 10 个 epoch 验证损失没明显变化就停,且 restore_best_weights=True 会把权重回滚到最优那一轮。这套机制能省掉大量无意义迭代,MT5 里接 Python 脚本跑的时候,日志里看到 epoch 停在 30~50 之间都算正常。 训练完会把 loss 和 val_loss 曲线存成图,再用 r2_score 算训练集拟合优度。pred = self.model.predict(self.train_x) 拿到预测值,和 train_y 画在一起,标题直接带上 acc 数值,方便你一眼判断模型是不是在硬记。 模型最后存成 h5:self.model.save(f"Models\\lstm-pat.{self.target_column}.h5")。选 h5 不是为了好看,是后面转 ONNX 给 MT5 做推理更顺。test_network 则另起一套 count 轴把测试集实际值和预测值叠着画,文件名标 MLP - Test,没看到这部分代码跑完之前,别轻易把模型丢进实盘。
keras.layers.Dense(units=neurons, activation=activation_function, activity_regularizer=l2(class="num">0.01), kernel_initializer="he_uniform"), # Hidden layer with an activation function keras.layers.Dense(units=self.output_size, activation=&class="macro">#x27;linear&class="macro">#x27;, activity_regularizer=l2(class="num">0.01), kernel_initializer="he_uniform") ]) # Print a summary of the model&class="macro">#x27;s architecture. self.model.summary() def train_network(self, epochs=class="num">100, learning_rate=class="num">0.001, loss=&class="macro">#x27;mean_squared_error&class="macro">#x27;): early_stopping = EarlyStopping(monitor=&class="macro">#x27;val_loss&class="macro">#x27;, patience=class="num">10, restore_best_weights=True) # Early stoppage mechanism | stop training when there is no major change in loss in the last to epochs, defined by the variable patience adam = optimizers.Adam(learning_rate=learning_rate) # Adam optimizer # Compile the model: Specify the loss function, optimizer, and evaluation metrics. self.model.compile(loss=loss, optimizer=adam, metrics=[&class="macro">#x27;mae&class="macro">#x27;]) # One hot encode the validation and train target variables validation_y = self.test_y y = self.train_y history = self.model.fit(self.train_x, y, epochs=epochs, batch_size=self.batch_size, validation_data=(self.test_x, validation_y), callbacks=[early_stopping], verbose=class="num">2) if not os.path.exists(self.plots_directory): class="macro">#create plots path if it doesn&class="macro">#x27;t exist for saving the train-test plots os.makedirs(self.plots_directory) # save the loss and validation loss plot plt.figure(figsize=(class="num">12, class="num">6)) plt.plot(history.history[&class="macro">#x27;loss&class="macro">#x27;], label=&class="macro">#x27;Training Loss&class="macro">#x27;) plt.plot(history.history[&class="macro">#x27;val_loss&class="macro">#x27;], label=&class="macro">#x27;Validation Loss&class="macro">#x27;) plt.xlabel(&class="macro">#x27;Epochs&class="macro">#x27;) plt.ylabel(&class="macro">#x27;Loss&class="macro">#x27;) plt.legend() title = &class="macro">#x27;Training and Validation Loss Curves&class="macro">#x27; plt.title(title) plt.savefig(fname=f"{self.plots_directory}\"+"+title) # use the trained model to make predictions on the trained data pred = self.model.predict(self.train_x) acc = metrics.r2_score(self.train_y, pred) # Plot actual & pred count = [i*class="num">0.1 for i in range(len(self.train_y))] title = f&class="macro">#x27;MLP {self.target_column} - Train&class="macro">#x27; # Saving the plot containing information about predictions and actual values plt.figure(figsize=(class="num">7, class="num">5)) plt.plot(count, self.train_y, label = "Actual") plt.plot(count, pred, label = "forecast") plt.xlabel(&class="macro">#x27;Actuals&class="macro">#x27;) plt.ylabel(&class="macro">#x27;Preds&class="macro">#x27;) plt.title(title+f" | Train acc={acc}") plt.legend() plt.savefig(fname=f"{self.plots_directory}\"+"+title) self.model.save(f"Models\\lstm-pat.{self.target_column}.h5") class="macro">#saving the model in h5 format, this will help us to easily convert this model to onnx later def test_network(self): # Plot actual & pred count = [i*class="num">0.1 for i in range(len(self.test_y))] title = f&class="macro">#x27;MLP {self.target_column} - Test&class="macro">#x27;