在任何市场中获得优势（第四部分）：CBOE欧元和黄金波动率指数·进阶篇
（2/3）·当波动率指数与MT5行情相关性趋近0，是该弃用还是换个切法？本篇用建模拆给你看
「用波动率指数拆开 XAUEUR 的隐藏结构」
相关性热力图先泼了盆冷水：替代数据和 XAUEUR 之间只是间歇性相关，但两个替代数据集彼此却绑得挺紧。别急着下结论，强相关缺失不等于真没关系，样本本身就有局限。 把黄金波动率指数（GVZCLS）当 x 轴、XAUEUR 收盘当 y 轴画散点，能看到一个硬现象：GVZCLS 突破 30–35 时，价格走势一致偏多；换成欧元波动率（EVZCLS）做 x 轴，阈值在 14–16 之上也出现同样的价格抬升。外汇和贵金属的高波动环境里，这种分离度值得拿 MT5 历史数据复核对一下。 三维散点把 GVZCLS、EVZCLS、Close 全铺开，看涨 K 线还是清晰成团，说明替代数据在某些点位确实能把行情分开。再用 t-SNE 把 6 维市场数据压成 2 维，随机初始化导致图难复现，但聚类数稳定在 4 个，图的弯曲形态暗示聚类内可能带时间依赖。 ACF 图三个数据集尾巴都长且慢衰减，证实之前热力图说的持久性成分；PACF 则指出最多回溯 4 个滞后项就能解释大部分自相关。下面这段是上面所有图的生成核心，逐行拆开方便你直接抄去跑。 #Exploratory Data Analysis #Analyzing correlation levels sns.heatmap(merged_data[predictors].corr(),annot=True) # 画相关性热力图，单元格标出具体相关系数 #Let's create scatter plots sns.scatterplot(data=merged_data,x="GVZCLS",y="Close",hue="Binary Target") # 以黄金波动率为横轴、收盘价为纵轴，按涨跌标签着色画散点 #Let's create scatter plots sns.scatterplot(data=merged_data,x="EVZCLS",y="Close",hue="Binary Target") # 换成欧元波动率做横轴，同样按标签着色 #Let's create scatter plots sns.scatterplot(data=merged_data,x="GVZCLS",y="EVZCLS",hue="Binary Target") # 两个替代数据互画散点，看它们之间耦合 #Define the 3D Plot fig = plt.figure(figsize=(7,7)) # 建 7x7 英寸画布 ax = plt.axes(projection="3d") # 开三维坐标轴 ax.scatter(merged_data["GVZCLS"],merged_data["EVZCLS"],merged_data["Close"],c=merged_data["Binary Target"],cmap="plasma") # 三维散点：两波动率+收盘，标签着色用 plasma 色表 ax.set_xlabel("GVZCLS") ax.set_ylabel("EVZCLS") ax.set_zlabel("Close") # 三个轴分别命名 #Let's create a TSNE Plot from sklearn.manifold import TS # 导入降维模块（原文 TS 为简写） #Create a TSNE object which will reduce the data to 2 dimensions tsne = TSNE(n_components=2,perplexity=30) # 实例化 t-SNE：压到 2 维，困惑度 30 #Apply TSNE to the data tsne_data = tsne.fit_transform(merged_data[predictors]) # 对预测变量矩阵做拟合变换 #Create a scatter plot plt.scatter(tsne_data[:,0],tsne_data[:,1]) # 把降维结果画成二维散点 #Let's look at an autocorrelation plot of the data close_acf = plot_acf(merged_data["Close"]) # 对收盘价画自相关图 #Let's look at an partial autocorrelation plot of the close data close_pacf = plot_pacf(merged_data["Close"]) # 对收盘价画偏自相关图，看滞后截断点
class="macro">#Exploratory Data Analysis class="macro">#Analyzing correlation levels sns.heatmap(merged_data[predictors].corr(),annot=True) class="macro">#Let&class="macro">#x27;s create scatter plots sns.scatterplot(data=merged_data,x="GVZCLS",y="Close",hue="Binary Target") class="macro">#Let&class="macro">#x27;s create scatter plots sns.scatterplot(data=merged_data,x="EVZCLS",y="Close",hue="Binary Target") class="macro">#Let&class="macro">#x27;s create scatter plots sns.scatterplot(data=merged_data,x="GVZCLS",y="EVZCLS",hue="Binary Target") class="macro">#Define the 3D Plot fig = plt.figure(figsize=(class="num">7,class="num">7)) ax = plt.axes(projection="3d") ax.scatter(merged_data["GVZCLS"],merged_data["EVZCLS"],merged_data["Close"],c=merged_data["Binary Target"],cmap="plasma") ax.set_xlabel("GVZCLS") ax.set_ylabel("EVZCLS") ax.set_zlabel("Close") class="macro">#Let&class="macro">#x27;s create a TSNE Plot from sklearn.manifold class="kw">import TS class="macro">#Create a TSNE object which will reduce the data to class="num">2 dimensions tsne = TSNE(n_components=class="num">2,perplexity=class="num">30) class="macro">#Apply TSNE to the data tsne_data = tsne.fit_transform(merged_data[predictors]) class="macro">#Create a scatter plot plt.scatter(tsne_data[:,class="num">0],tsne_data[:,class="num">1]) class="macro">#Let&class="macro">#x27;s look at an autocorrelation plot of the data close_acf = plot_acf(merged_data["Close"]) class="macro">#Let&class="macro">#x27;s look at an partial autocorrelation plot of the close data close_pacf = plot_pacf(merged_data["Close"])
建模前的数据与切分处理
在拿深度神经网络去拟合行情数据之前，先得把输入变量做标准化和缩放。用 RobustScaler 处理比较合适，它按中位数和四分位距缩放，对贵金属或外汇里偶尔出现的极端跳空点不像 MinMaxScaler 那样敏感。 切分训练集和测试集时，三组预测因子都要设 shuffle=False。时间序列一旦随机打乱，未来信息会漏进训练集，回测结果好看但实盘必然失真，这是分析完整性的硬伤。 代码里三组切分的 train_size 都设为 0.5，也就是一半样本训练、一半验证；随后用 TimeSeriesSplit(n_splits=5, gap=look_ahead) 做滚动交叉验证，gap 参数把预测步长空出来，避免前一段的标签泄漏到后一段训练里。验证误差存进一个 5 行 3 列的 DataFrame，分别对应 OHLC、FRED 和全因子三组表现。 开 MT5 把历史数据导出来跑这段，先确认 scaler 后的字段没有 NaN，再去看 validation_error 里哪组预测因子 MSE 更低，据此决定后面网络喂哪些输入。外汇和贵金属杠杆高、波动剧烈，模型仅提供概率倾向，实盘仍须严控仓位。
class="macro">#Preparing to model the data from sklearn.preprocessing class="kw">import RobustScaler from sklearn.model_selection class="kw">import TimeSeriesSplit,train_test_split from sklearn.metrics class="kw">import mean_squared_error from sklearn.neural_network class="kw">import MLPRegressor class="macro">#Reset the index of our data merged_data.reset_index(inplace=True) X = merged_data.loc[:,predictors] y = merged_data.loc[:,target] class="macro">#Scale our data scaler = RobustScaler() X = pd.DataFrame(scaler.fit_transform(merged_data[predictors]),columns=predictors) class="macro">#Perform train test splits ohlc_train_X,ohlc_test_X,train_y,test_y = train_test_split(X.loc[:,ohlc_predictors],y,shuffle=False,train_size=class="num">0.5) fred_train_X,fred_test_X,_,_ = train_test_split(X.loc[:,fred_predictors],y,shuffle=False,train_size=class="num">0.5) train_X,test_X,_,_ = train_test_split(X.loc[:,predictors],y,shuffle=False,train_size=class="num">0.5) class="macro">#Let&class="macro">#x27;s now cross-validate each of the predictors class="macro">#Create the time-series split object tscv = TimeSeriesSplit(n_splits=class="num">5,gap=look_ahead) validation_error = pd.DataFrame(columns=["OHLC Predictors","FRED Predictors","All Predictors"],index=np.arange(class="num">0,class="num">5))
◍ 三种数据源下交叉验证误差的对比
用时间序列交叉验证跑完模型后，先把各折的验证误差摊开看。表格里三列分别是 MT5 的 OHLC 数据、FRED/CBOE 替代数据、以及全部数据混合三组输入下的 MSE：第一行 875423.6 / 881892.5 / 857846.1，最小的是第三列；但单看一行说明不了问题，得拉平均值。 把第一列均值当基准减掉之后，第三列整体贴近 0 且多为负偏移，说明全数据模型的误差倾向最低。箱线图进一步印证：仅用 OHLC 时方差明显被拉开，而混合数据那组的中位线和须长都更收敛，平均误差率可能更稳。 外汇与贵金属行情受流动性跳变影响大，这类回归误差只代表样本内交叉验证表现，实盘置换成本与滑点会让真实 MSE 偏高，拿来调参前先在 MT5 历史中心复算一遍。
class="macro">#Performing cross validation model = MLPRegressor(hidden_layer_sizes=(class="num">20,class="num">5)) for i,(train,test) in enumerate(tscv.split(train_X)): model.fit(train_X.loc[train[class="num">0]:train[-class="num">1],:],train_y.loc[train[class="num">0]:train[-class="num">1]]) validation_error.iloc[i,class="num">2] = mean_squared_error(train_y.loc[test[class="num">0]:test[-class="num">1]],model.predict(train_X.loc[test[class="num">0]:test[-class="num">1],:])) class="macro">#Our validation error validation_error class="macro">#Our mean error levels val_err = validation_error.mean() val_err = val_err.iloc[:] - val_err.iloc[class="num">0] val_err.plot(kind="bar") class="macro">#Let&class="macro">#x27;s perform boxplots of our validation error sns.boxplot(validation_error)
「用 ALE 和 SHAP 拆开模型黑箱」
低误差指标不等于能直接上实盘。XAUEUR 这类高相关行情数据训出来的模型，必须用全局解释工具看它到底学了什么关系。 累积局部效应（ALE）适合高度相关特征，能隔离单个输入对输出的边际影响。对深度神经网络跑 ALE 后，开盘价和最高价的 ALE 图显示：价格初期上行时模型预测先降，继续拉高后敏感度明显走弱；最低价、收盘价的形状与之几乎一致。 换到 FRED CBOE 欧元波动率指数上，ALE 图补上了前面变量没覆盖的一段方差，且斜率向上——波动率读数越高，模型给出的预测值倾向同步抬升。 SHAP 基于博弈论，遍历特征组合算平均贡献。 beeswarm 图印证：XAUEUR 自身市场数据权重最大，且现价越高，目标变量倾向向下。外汇与贵金属杠杆高，解释结论只代表样本内结构，实盘前请在 MT5 用历史数据复算。
class="macro">#Feature importance from alibi.explainers class="kw">import ALE , plot_ale class="macro">#Explaining our deep neural network model = MLPRegressor(hidden_layer_sizes=(class="num">20,class="num">5)) model.fit(train_X,train_y) dnn_ale = ALE(model.predict,feature_names=predictors,target_names=["XAUEUR Close"]) class="macro">#Obtaining the explanation ale_X = X.to_numpy() dnn_explanations = dnn_ale.explain(ale_X) class="macro">#Plotting feature importance plot_ale(dnn_explanations,n_cols=class="num">3,fig_kw={&class="macro">#x27;figwidth&class="macro">#x27;:class="num">8,&class="macro">#x27;figheight&class="macro">#x27;:class="num">8},sharey=None) class="macro">#SHAP Values class="kw">import shap class="macro">#Calculating SHAP values explainer = shap.Explainer(model.predict,train_X) shap_values = explainer(test_X)class="macro">#Calculating SHAP values class="macro">#Plot the beeswarm plot shap.plots.beeswarm(shap_values)
两层搜索榨出模型冗余参数
单轮随机搜索只能在大网格里捞一遍，MLPRegressor 在 hidden_layer_sizes=(20,5) 时，RandomizedSearchCV 跑 500 次迭代、5 折交叉验证，给出的较优组合是 activation='identity'、solver='lbfgs'、alpha=0.1、tol=1e-07、learning_rate='adaptive'、shuffle=True。这组值可作为二阶连续优化的落点，而不是从头乱猜。 Scipy 的 minimize 走的是连续空间寻优，把 alpha 和 tol 当自变量，目标函数用训练集上时间序列交叉验证（tscv）的平均均方误差。边界给到 1e-19 到 1e10，范围极宽，L-BFGS-B 方法会在该框内收敛到局部最小误差。 外汇与贵金属行情具有高风险，模型回测误差低不代表实盘概率占优，MT5 接数据后应先以小样本验证再放大。下面代码可直接复制到 Notebook 跑通两段调优。
class="macro">#Parameter tuning from sklearn.model_selection class="kw">import RandomizedSearchCV class="macro">#Reinitialize the model model = MLPRegressor(hidden_layer_sizes=(class="num">20,class="num">5)) class="macro">#Define the tuner tuner = RandomizedSearchCV( model, { "activation" : ["relu","logistic","tanh","identity"], "solver":["adam","sgd","lbfgs"], "alpha":[class="num">0.1,class="num">0.01,class="num">0.001,class="num">0.0001,class="num">0.00001,class="num">0.00001,class="num">0.0000001], "tol":[class="num">0.1,class="num">0.01,class="num">0.001,class="num">0.0001,class="num">0.00001,class="num">0.000001,class="num">0.0000001], "learning_rate":[&class="macro">#x27;constant&class="macro">#x27;,&class="macro">#x27;adaptive&class="macro">#x27;,&class="macro">#x27;invscaling&class="macro">#x27;], "shuffle": [True,False] }, n_iter=class="num">500, cv=class="num">5, n_jobs=-class="num">1, scoring="neg_mean_squared_error" ) class="macro">#Fit the tuner tuner_results = tuner.fit(train_X,train_y) class="macro">#The best parameters we found tuner_results.best_params_ class="macro">#Deeper optimization from scipy.optimize class="kw">import minimize class="macro">#Create a dataframe to store our accuracy current_error_rate = pd.DataFrame(index = np.arange(class="num">0,class="num">5),columns=["Current Error"]) class="macro">#Define the objective function def objective(x): class="macro">#The parameter x represents a new value for our neural network&class="macro">#x27;s settings class="macro">#In order to find optimal settings, we will perform class="num">10 fold cross validation class="kw">using the new setting class="macro">#And class="kw">return the average RMSE from all class="num">10 tests class="macro">#We will first turn the model&class="macro">#x27;s Alpha parameter, which controls the amount of L2 regularization model = MLPRegressor(hidden_layer_sizes=(class="num">20,class="num">5),activation=&class="macro">#x27;identity&class="macro">#x27;,learning_rate=&class="macro">#x27;adaptive&class="macro">#x27;,solver=&class="macro">#x27;lbfgs&class="macro">#x27;,shuffle=True,alpha=x[class="num">0],tol=x[class="num">1]) class="macro">#Now we will cross validate the model for i,(train,test) in enumerate(tscv.split(train_X)): class="macro">#Train the model model.fit(train_X.loc[train[class="num">0]:train[-class="num">1],:],train_y.loc[train[class="num">0]:train[-class="num">1]]) class="macro">#Measure the RMSE current_error_rate.iloc[i,class="num">0] = mean_squared_error(train_y.loc[test[class="num">0]:test[-class="num">1]],model.predict(train_X.loc[test[class="num">0]:test[-class="num">1],:])) class="macro">#Return the Mean CV RMSE class="kw">return(current_error_rate.iloc[:,class="num">0].mean()) class="macro">#Define the starting point pt = [class="num">0.1,class="num">0.00000001] bnds = ((class="num">0.0000000000000000001,class="num">10000000000),(class="num">0.0000000000000000001,class="num">10000000000)) class="macro">#Searchin deeper for parameters result = minimize(objective,pt,method="L-BFGS-B",bounds=bnds)
◍ 用交叉验证拆穿过拟合
模型记住噪声而非规律，是机器学习择时里最隐蔽的坑。把默认 MLP、随机搜索调参模型、L-BFGS-B 定制模型放在同一训练集上拟合，再丢到一直没碰过的测试集做 5 折交叉验证，误差一眼见分晓。 实测 MSE 数据摆出来：默认模型五折误差为 22360、17385、13782、3082484、4076009，随机搜索模型为 5917、36726、5128、6950、27729，L-BFGS-B 模型为 3734、35886、20886、5789、22931。默认模型后两折直接崩到百万级，说明它把训练噪声当成了结构。 随机搜索模型在多数折里误差最低，综合表现最好；箱线图也显示定制组相对默认模型中位数拉开了数量级。外汇与贵金属行情序列非平稳，这类回测误差仍可能随样本外 regime 切换而失效，高风险。 下面这段 Python 可直接在 MT5 导出的特征 CSV 上重跑，验证你自己的模型是否也过拟合了。
class="macro">#Testing for overfitting default_model = MLPRegressor(hidden_layer_sizes=(class="num">20,class="num">5)) customized_model = MLPRegressor(hidden_layer_sizes=(class="num">20,class="num">5),activation=&class="macro">#x27;identity&class="macro">#x27;,learning_rate=&class="macro">#x27;adaptive&class="macro">#x27;,solver=&class="macro">#x27;lbfgs&class="macro">#x27;,shuffle=True,alpha=class="num">0.1,tol=class="num">0.0000001) customized_lbfgs_model = MLPRegressor(hidden_layer_sizes=(class="num">20,class="num">5),activation=&class="macro">#x27;identity&class="macro">#x27;,learning_rate=&class="macro">#x27;adaptive&class="macro">#x27;,solver=&class="macro">#x27;lbfgs&class="macro">#x27;,shuffle=True,alpha=result.x[class="num">0],tol=result.x[class="num">1]) <span class="preprocessor">class="macro">#Preparing </span>to cross validate the models models = [ default_model, customized_model, customized_lbfgs_model ] <span class="preprocessor">class="macro">#We </span>will store our validation error here validation_error = pd.DataFrame(columns=["Default Model","Customized Model","L-BFGS Model"],index=np.arange(class="num">0,class="num">5)) <span class="preprocessor">class="macro">#We </span>will now reset the indexes test_y = test_y.reset_index() test_X = test_X.reset_index() <span class="preprocessor">class="macro">#Fit </span>each of the models <span class="keyword">for</span> m in models: m.fit(train_X,train_y) class="macro">#Cross validating each model <span class="keyword">for</span> j <span class="keyword">in</span> np.arange(class="num">0,len(models)): model = models[j] <span class="keyword">for</span> i,(train,test) <span class="keyword">in</span> enumerate(tscv.split(test_X)): model.fit(test_X.loc[train[class="num">0]:train[-class="num">1],:],test_y.loc[train[class="num">0]:train[-class="num">1],"Target"]) validation_error.iloc[i,j] = mean_squared_error(test_y.loc[test[class="num">0]:test[-class="num">1],"Target"],model.predict(test_X.loc[test[class="num">0]:test[-class="num">1],:])) <span class="preprocessor">class="macro">#Our </span>validation error validation_error <span class="preprocessor">class="macro">#Plotting </span>the difference in our performance levels mean = validation_error.mean() mean = mean.iloc[:] - mean.iloc[class="num">0] mean.plot(kind="bar") <span class="preprocessor">class="macro">#Visualizing </span>the results validation_error.plot() <span class="preprocessor">class="macro">#Visualizing </span>our results sns.boxplot(validation_error)