数据科学和机器学习（第 33 部分）：MQL5 中的 Pandas 数据帧，为机器学习收集数据更加容易·进阶篇
（2/3）· 从 CSV 读写到数据帧筛选，把 Python 系数据准备习惯搬进 MQL5 实盘环境
接上篇铺垫的数据帧基础，我们继续深挖 MQL5 里 Pandas 类怎么真正落地。多数人在 MQL5 手工拼二维数组，等到要接 ONNX 模型才发现列名错乱、索引对不上，回测和实盘用的不是同一套数据形状。
把 CSV 直接灌进数据帧
在 MQL5 里做价格行为复盘，常要把导出的 EURUSD.PERIOD_D1.csv 这类日线文件读进来矩阵化。下面这套 CDataFrame::ReadCSV 就是干这事的核心：打开 CSV、按行写进 matrix、首行当列名跳过。 调用时只需 df.ReadCSV("EURUSD.PERIOD_D1.csv")，随后 df.Head() 就能在日志看到表头。日线 CSV 通常上千行，verbosity 开 true 会逐行 printf 进度，调试文件占用问题很实用。
| FileOpen 用了 FILE_READ | FILE_CSV | FILE_ANSI，delimiter 默认逗号；若句柄为 INVALID_HANDLE，多半是文件被别的程序占用或路径不对。外汇和贵金属数据读取本身无风险，但据此交易杠杆品种属高风险，仓位需自担。 |
|---|
代码逐行要点：FileReadString 逐字段读，rows==0 时存 m_columns 列头；rows>0 才写 mat_ 数值矩阵，避开头行。FileIsLineEnding 触发才 rows++ 并 Resize，最后 m_values = mat_ 完成赋值。
df = pd.read_csv("EURUSD.PERIOD_D1.csv") class="type">bool CDataFrame::ReadCSV(class="type">class="kw">string file_name,class="type">class="kw">string delimiter=",",class="type">bool is_common=class="kw">false, class="type">bool verbosity=class="kw">false) { matrix mat_ = {}; class="type">int rows_total=class="num">0; class="type">int handle = FileOpen(file_name,FILE_READ|FILE_CSV|FILE_ANSI|(is_common?FILE_IS_COMMON:FILE_ANSI),delimiter); class=class="str">"cmt">//Open a csv file ResetLastError(); if(handle == INVALID_HANDLE) class=class="str">"cmt">//Check if the file handle is ok if not class="kw">return class="kw">false { printf("Invalid %s handle Error %d ",file_name,GetLastError()); Print(GetLastError()==class="num">0?" TIP | File Might be in use Somewhere else or in another Directory":""); class="kw">return class="kw">false; } else { class="type">int column = class="num">0, rows=class="num">0; class="kw">while(!FileIsEnding(handle)) { class="type">class="kw">string data = FileReadString(handle); class=class="str">"cmt">//--- if(rows ==class="num">0) { ArrayResize(m_columns,column+class="num">1); m_columns[column] = data; } if(rows>class="num">0) class=class="str">"cmt">//Avoid the first column which contains the column&class="macro">#x27;s header mat_[rows-class="num">1,column] = (class="type">class="kw">double(data)); class=class="str">"cmt">//add a value to the matrix column++; class=class="str">"cmt">//--- if(FileIsLineEnding(handle)) class=class="str">"cmt">//At the end of the each line { rows++; mat_.Resize(rows,column); class=class="str">"cmt">//Resize the matrix to accomodate new values column = class="num">0; } } if (verbosity) class=class="str">"cmt">//if verbosity is set to true, we print the information to let the user know the progress, Useful for debugging purposes printf("Reading a CSV file... record [%d]",rows); rows_total = rows; FileClose(handle); class=class="str">"cmt">//Close the file after reading it } mat_.Resize(rows_total-class="num">1,mat_.Cols()); m_values = mat_; class="kw">return true; } class="type">void OnStart() { class=class="str">"cmt">//--- CDataFrame df; df.ReadCSV("EURUSD.PERIOD_D1.csv"); df.Head(); }
「在 MT5 里复刻 Pandas 的 head 预览」
处理日线 EURUSD 这类大样本时，数据帧往往几百上千行，直接全量打印既刷屏又难定位。Pandas 的 head() 默认吐前 5 行，用来快速确认列名和数值格式是否对路，是探索阶段最高频的动作之一。 下面这段 MQL5 把同样逻辑搬进自定义 CDataFrame：不依赖 Python，直接在策略测试器日志里画出带边框的简表。默认 count=5，传参可改行数；列宽按字符串长度和 8 位小数动态算，留 4 字符 padding 防挤。 日志实测输出里，首行表头为 Open/High/Low/Close，紧接 1.09381000 / 1.09548000 / 1.09003000 / 1.09373000 这一行，说明 CSV 读取和浮点格式串 "%.8f" 都对得上。外汇与贵金属杠杆品种波动剧烈，用 head 先核验数据再跑回测，能少踩一半脏数据坑。
df = pd.read_csv("EURUSD.PERIOD_D1.csv") df.head() class="type">void CDataFrame::Head(class="kw">const class="type">uint count=class="num">5) { class=class="str">"cmt">// Calculate maximum width needed for each column class="type">uint num_cols = m_columns.Size(); class="type">uint col_widths[]; ArrayResize(col_widths, num_cols); for (class="type">uint col = class="num">0; col < num_cols; col++) class=class="str">"cmt">//Determining column width for visualizing a simple table { class="type">uint max_width = StringLen(m_columns[col]); for (class="type">uint row = class="num">0; row < count && row < m_values.Rows(); row++) { class="type">class="kw">string num_str = StringFormat("%.8f", m_values[row][col]); max_width = MathMax(max_width, StringLen(num_str)); } col_widths[col] = max_width + class="num">4; class=class="str">"cmt">// Extra padding for readability } class=class="str">"cmt">// Print column headers with calculated padding class="type">class="kw">string header = ""; for (class="type">uint col = class="num">0; col < num_cols; col++) { header += StringFormat("| %-*s ", col_widths[col], m_columns[col]); } header += "|"; Print(header); class=class="str">"cmt">// Print rows with padding for each column for (class="type">uint row = class="num">0; row < count && row < m_values.Rows(); row++) { class="type">class="kw">string row_str = ""; for (class="type">uint col = class="num">0; col < num_cols; col++) { row_str += StringFormat("| %-*.*f ", col_widths[col], class="num">8, m_values[row][col]); } row_str += "|"; Print(row_str); } class=class="str">"cmt">// Print dimensions printf("(%dx%d)", m_values.Rows(), m_values.Cols()); } class="type">void OnStart() { class=class="str">"cmt">//--- CDataFrame df; df.ReadCSV("EURUSD.PERIOD_D1.csv"); df.Head(); }
◍ Volatility 75 指数 H1 上的四组同步快照
在 12:37:02.984 这一时间戳，针对 Volatility 75 Index 的 H1 周期同时输出了四组报价快照，标记分别为 GI、DI、EI、CI，外加一行 FE 的 1000x4 批量标记。这类同毫秒多组打印常用于校验不同采样逻辑是否在同一根 H1 棒内取到一致上下文。 具体看数值：GI 组开 1.09678、高 1.09810、低 1.09361、收 1.09399；DI 组开 1.09701、高 1.09973、低 1.09606、收 1.09805；EI 组开 1.09639、高 1.09869、低 1.09542、收 1.09742；CI 组开 1.10302、高 1.10396、低 1.09513、收 1.09757。四组收盘位集中在 1.09399 到 1.09805 之间，最大偏差约 41 点。 CI 组的高点 1.10396 明显脱离其余三组约 1.0987 的上沿，说明该组可能采用了包含影线拉伸或不同报价源的合成逻辑。外汇与指数差价合约波动剧烈，Volatility 75 类合成指数高风险，上述同毫秒偏差只代表当时采样，不构成方向判断。 直接开 MT5 接 pandas 回读这类日志，把四组 OHLC 拉成 DataFrame 比对标准差，能快速定位是哪一路取值在异常。
GI class="num">0 class="num">12:class="num">37:class="num">02.984 pandas test(Volatility class="num">75 Index,H1) | class="num">1.09678000 | class="num">1.09810000 | class="num">1.09361000 | class="num">1.09399000 | DI class="num">0 class="num">12:class="num">37:class="num">02.984 pandas test(Volatility class="num">75 Index,H1) | class="num">1.09701000 | class="num">1.09973000 | class="num">1.09606000 | class="num">1.09805000 | EI class="num">0 class="num">12:class="num">37:class="num">02.984 pandas test(Volatility class="num">75 Index,H1) | class="num">1.09639000 | class="num">1.09869000 | class="num">1.09542000 | class="num">1.09742000 | CI class="num">0 class="num">12:class="num">37:class="num">02.984 pandas test(Volatility class="num">75 Index,H1) | class="num">1.10302000 | class="num">1.10396000 | class="num">1.09513000 | class="num">1.09757000 | FE class="num">0 class="num">12:class="num">37:class="num">02.984 pandas test(Volatility class="num">75 Index,H1) (1000x4)
把数据帧甩出 MT5 存成 CSV
在 MT5 里把多类行情与指标凑进一个数据帧后，下一步往往是丢到外部做机器学习，CSV 是最顺手的桥接格式，Python 端用 Pandas 的 read_csv 能直接接上。 MQL5 侧封装了一个 CDataFrame::ToCSV 方法，默认生成不带索引列的逗号分隔文件，例如 EURUSDcopy.csv；默认小数位 digits=5，符合外汇报价格式。 调用时若把 verbosity 设为 true，终端会逐行打印「Writing a CSV file... record [i/N]」，方便核对大样本是否写漏。注意 common 参数控制文件落在终端私有目录还是公共目录，跨平台调度时要留心路径。 下面这段是方法本体与触发调用的骨架，逐行拆完你就能改参数自己跑。 [CODE] df.to_csv("EURUSDcopy.csv", index=False) bool CDataFrame::ToCSV(string csv_name, bool common=false, int digits=5, bool verbosity=false) { FileDelete(csv_name);
| int handle = FileOpen(csv_name,FILE_WRITE | FILE_SHARE_WRITE | FILE_CSV | FILE_ANSI | (common?FILE_COMMON:FILE_ANSI),",",CP_UTF8); //open a csv file |
|---|
if(handle == INVALID_HANDLE) //Check if the handle is OK { printf("Invalid %s handle Error %d ",csv_name,GetLastError()); return (false); } //--- string concstring; vector row = {}; vector colsinrows = m_values.Row(0); if (ArraySize(m_columns) != (int)colsinrows.Size()) { printf("headers=%d and columns=%d from the matrix vary is size ",ArraySize(m_columns),colsinrows.Size()); DebugBreak(); return false; } //--- string header_str = ""; for (int i=0; i<ArraySize(m_columns); i++) //We concatenate the header only separating it with a comma delimeter header_str += m_columns[i] + (i+1 == colsinrows.Size() ? "" : ","); FileWrite(handle,header_str); FileSeek(handle,0, SEEK_SET); for(ulong i=0; i<m_values.Rows() && !IsStopped(); i++) { ZeroMemory(concstring); row = m_values.Row(i); for(ulong j=0, cols =1; j<row.Size() && !IsStopped(); j++, cols++) { concstring += (string)NormalizeDouble(row[j],digits) + (cols == m_values.Cols() ? "" : ","); } if (verbosity) //if verbosity is set to true, we print the information to let the user know the progress, Useful for debugging purposes printf("Writing a CSV file... record [%d/%d]",i+1,m_values.Rows()); FileSeek(handle,0,SEEK_END); FileWrite(handle,concstring); } FileClose(handle); return (true); } void OnStart() { CDataFrame df; df.ReadCSV("EURUSD.PERIOD_D1.csv"); //Assign a csv file into the dataframe
df.to_csv("EURUSDcopy.csv", index=False) class="type">bool CDataFrame::ToCSV(class="type">class="kw">string csv_name, class="type">bool common=class="kw">false, class="type">int digits=class="num">5, class="type">bool verbosity=class="kw">false) { FileDelete(csv_name); class="type">int handle = FileOpen(csv_name,FILE_WRITE|FILE_SHARE_WRITE|FILE_CSV|FILE_ANSI|(common?FILE_COMMON:FILE_ANSI),",",CP_UTF8); class=class="str">"cmt">//open a csv file if(handle == INVALID_HANDLE) class=class="str">"cmt">//Check if the handle is OK { printf("Invalid %s handle Error %d ",csv_name,GetLastError()); class="kw">return (class="kw">false); } class=class="str">"cmt">//--- class="type">class="kw">string concstring; vector row = {}; vector colsinrows = m_values.Row(class="num">0); if (ArraySize(m_columns) != (class="type">int)colsinrows.Size()) { printf("headers=%d and columns=%d from the matrix vary is size ",ArraySize(m_columns),colsinrows.Size()); DebugBreak(); class="kw">return class="kw">false; } class=class="str">"cmt">//--- class="type">class="kw">string header_str = ""; for (class="type">int i=class="num">0; i<ArraySize(m_columns); i++) class=class="str">"cmt">//We concatenate the header only separating it with a comma delimeter header_str += m_columns[i] + (i+class="num">1 == colsinrows.Size() ? "" : ","); FileWrite(handle,header_str); FileSeek(handle,class="num">0, SEEK_SET); for(class="type">class="kw">ulong i=class="num">0; i<m_values.Rows() && !IsStopped(); i++) { ZeroMemory(concstring); row = m_values.Row(i); for(class="type">class="kw">ulong j=class="num">0, cols =class="num">1; j<row.Size() && !IsStopped(); j++, cols++) { concstring += (class="type">class="kw">string)NormalizeDouble(row[j],digits) + (cols == m_values.Cols() ? "" : ","); } if (verbosity) class=class="str">"cmt">//if verbosity is set to true, we print the information to let the user know the progress, Useful for debugging purposes printf("Writing a CSV file... record [%d/%d]",i+class="num">1,m_values.Rows()); FileSeek(handle,class="num">0,SEEK_END); FileWrite(handle,concstring); } FileClose(handle); class="kw">return (true); } class="type">void OnStart() { CDataFrame df; df.ReadCSV("EURUSD.PERIOD_D1.csv"); class=class="str">"cmt">//Assign a csv file into the dataframe
「把清洗后的数据落盘成副本」
在 DataFrame 完成列裁剪或缺失值处理后，用 ToCSV 方法可以把结果写回文件系统，避免覆盖原始行情。上面这行把内存里的表存成 EURUSDcopy.csv，相当于留了一份可回溯的副本。 实盘前建议先对 2023 年 EURUSD 的 M1 数据跑一遍，确认副本行数与源文件一致，外汇与贵金属杠杆品种波动剧烈，回测数据错一行都可能让信号偏移。
df.ToCSV("EURUSDcopy.csv"); class=class="str">"cmt">//Save the dataframe back into a CSV file as a copy }
◍ 给自定义数据帧加上切片与索引能力
在 MT5 里跑模型，常常只要最新一根 K 线的数值，或者训练时只取开头若干行。把 Pandas 那套 loc / iloc / at / iat 搬到 MQL5 类里，就能用熟悉的方式切数据。 列访问最直接，重载 [] 运算符传列名即可返回数值向量。下面这段代码把字符串索引映射到 GetColumn： vector operator[] (const string index) {return GetColumn(index); } //按列名取列 Print("Close column: ",df["Close"]); 实盘日志里 Close 列打出了 64 个 EURUSD H1 收盘价，从 1.09373 一路走到 1.08986，能直接拿去算均线或特征。 loc 按标签或布尔取数，我加了 axis 参数区分行(0)和列(1)，负值倒着数，-1 就是末尾。调用 df.loc[0] 返回的是第 0 行 Open/High/Low/Close 四条向量，和 Python 端输出结构一致。 iloc 按整数位置切，返回新数据帧；at 取单值、iat 按位置取单值，都照 Pandas 语义实现。drop 则用来扔掉训练不需要的列，比如冗余的成交量字段。 负索引转换的逻辑藏在 Loc 函数里：index<0 时加上总行数再取，越界就 printf 报错并返回空向量。复制这段到你的 CDataFrame 类，编译后挂 EURUSD H1 就能验证切片是否和 Python 端对得上。外汇与贵金属杠杆高，回测通过不代表实盘稳，参数先跑模拟。
vector class="kw">operator[] (class="kw">const class="type">class="kw">string index) {class="kw">return GetColumn(index); } class=class="str">"cmt">//Access a column by its name Print("Close column: ",df["Close"]); class="num">2025.01.class="num">27 class="num">16:class="num">16:class="num">19.726 pandas test(EURUSD,H1) Close column: [class="num">1.09373,class="num">1.09399,class="num">1.09805,class="num">1.09742,class="num">1.09757,class="num">1.10297,class="num">1.10453,class="num">1.10678,class="num">1.1135,class="num">1.11594,class="num">1.11765,class="num">1.11327,class="num">1.11797,class="num">1.11107,class="num">1.1163,class="num">1.11616,class="num">1.11177,class="num">1.11141,class="num">1.11326,class="num">1.10745,class="num">1.10747,class="num">1.10111,class="num">1.10192,class="num">1.10351,class="num">1.10861,class="num">1.11106,class="num">1.1083,class="num">1.10435,class="num">1.10723,class="num">1.10483,class="num">1.1078,class="num">1.11199,class="num">1.11843,class="num">1.1161,class="num">1.11932,class="num">1.11113,class="num">1.11499,class="num">1.113,class="num">1.10852,class="num">1.10267,class="num">1.09712,class="num">1.10124,class="num">1.09928,class="num">1.09321,class="num">1.09156,class="num">1.09188,class="num">1.09236,class="num">1.09315,class="num">1.09511,class="num">1.09107,class="num">1.07913,class="num">1.08258,class="num">1.08142,class="num">1.08211,class="num">1.08551,class="num">1.0845,class="num">1.08392,class="num">1.08529,class="num">1.08905,class="num">1.08818,class="num">1.08959,class="num">1.09396,class="num">1.08986,] df.loc[class="num">0] Open class="num">1.09381 High class="num">1.09548 Low class="num">1.09003 Close class="num">1.09373 Name: class="num">0, dtype: float64 vector CDataFrame::Loc(class="type">int index, class="type">uint axis=class="num">0) { if(axis == class="num">0) { vector row = {}; class=class="str">"cmt">//--- Convert negative index to positive if(index < class="num">0) index = (class="type">int)m_values.Rows() + index; if(index < class="num">0 || index >= (class="type">int)m_values.Rows()) { printf("%s Error: Row index out of bounds. Given index: %d", __FUNCTION__, index); class="kw">return row; } class="kw">return m_values.Row(index); } else if(axis == class="num">1) { vector column = {}; class=class="str">"cmt">//--- Convert negative index to positive if(index < class="num">0) index = (class="type">int)m_values.Cols() + index;