数据科学和机器学习(第 13 部分):配合主成分分析(PCA)改善您的金融市场分析·进阶篇
(2/3)· 从标准化到成分提取,把高维行情特征压成可解释的低维信号
「一行矩阵乘法拿到 PCA 得分」
主成分得分的求解在代码层极简:把归一化后的原始数据矩阵与成分矩阵做乘法即可,不需要自己写循环。下例在 NAS100 的 H1 周期上跑出 10 行样本、4 个主成分的原始得分矩阵。 从日志看,首行得分为 [-0.6500472384886967, 1.199407986803537, 0.1425145462368588, 0.1006701620494091],末行则是 [1.664404875867474, 1.051245703485609, 0.1413817973120564, 0.2119289033750197],说明不同 K 线在四个主成分轴上的投影差异明显。 拿到得分后还不能直接喂给模型,得再做一次标准化。示例里用 CPreprocessing 以 NORM_STANDARDIZATION 模式处理,标准化后首行变成 [-0.4187491401035159, 0.9970295470975233, 0.68746486754918, 0.3182591681100855],量纲被压到可比区间。外汇与贵金属行情受事件驱动剧烈,PCA 降维仅描述历史协方差结构,对后市方向只具概率性参考。
pca_scores = Matrix.MatMul(component_matrix); CS class="num">0 class="num">10:class="num">17:class="num">31.957 PCA Test(NAS100,H1) PCA SCORES CS class="num">0 class="num">10:class="num">17:class="num">31.957 PCA Test(NAS100,H1) [[-class="num">0.6500472384886967,class="num">1.199407986803537,class="num">0.1425145462368588,class="num">0.1006701620494091] CS class="num">0 class="num">10:class="num">17:class="num">31.957 PCA Test(NAS100,H1) [class="num">1.819562596624738,class="num">1.393614599196321,-class="num">0.1510888243020112,class="num">0.1670753033981925] CS class="num">0 class="num">10:class="num">17:class="num">31.957 PCA Test(NAS100,H1) [class="num">0.2688014256048517,class="num">1.420914385142756,class="num">0.001937917070391801,-class="num">0.6847663538666366] CS class="num">0 class="num">10:class="num">17:class="num">31.957 PCA Test(NAS100,H1) [-class="num">1.110534258768705,-class="num">0.06593596223641518,-class="num">0.4827665581567511,class="num">0.09571954869438426] CS class="num">0 class="num">10:class="num">17:class="num">31.957 PCA Test(NAS100,H1) [class="num">2.475561333978323,-class="num">1.768915328424386,-class="num">0.0006861487484489809,class="num">0.2983796568520111] CS class="num">0 class="num">10:class="num">17:class="num">31.957 PCA Test(NAS100,H1) [-class="num">0.6245145789301378,-class="num">1.503882637300733,-class="num">0.1738415909335406,-class="num">0.2393186981373224] CS class="num">0 class="num">10:class="num">17:class="num">31.957 PCA Test(NAS100,H1) [-class="num">2.608156175249579,class="num">0.0662886285379769,class="num">0.1774740257067155,class="num">0.4223436077935874] CS class="num">0 class="num">10:class="num">17:class="num">31.957 PCA Test(NAS100,H1) [class="num">0.4325302694103054,-class="num">1.589321053467977,class="num">0.2509606394263523,-class="num">0.337079680008286] CS class="num">0 class="num">10:class="num">17:class="num">31.957 PCA Test(NAS100,H1) [-class="num">1.667608250048573,-class="num">0.2034163217366656,class="num">0.09411419638842802,-class="num">0.03495245015036286] CS class="num">0 class="num">10:class="num">17:class="num">31.957 PCA Test(NAS100,H1) [class="num">1.664404875867474,class="num">1.051245703485609,class="num">0.1413817973120564,class="num">0.2119289033750197]] pre_processing = new CPreprocessing(pca_scores_standardized, NORM_STANDARDIZATION); CS class="num">0 class="num">10:class="num">17:class="num">31.957 PCA Test(NAS100,H1) PCA SCORES | STANDARDIZED CS class="num">0 class="num">10:class="num">17:class="num">31.957 PCA Test(NAS100,H1) [[-class="num">0.4187491401035159,class="num">0.9970295470975233,class="num">0.68746486754918,class="num">0.3182591681100855]
◍ NAS100小时图的主成分投影样本
在 NAS100 的 H1 周期上跑主成分分析,同一毫秒(10:17:31.957)输出了 9 组四维投影向量,说明降维计算是批量完成的,不是逐根 K 线触发。 看前两组:第一组是 [1.172, 1.158, -0.729, 0.528],第二组骤变为 [0.173, 1.181, 0.009, -2.165]。第三主成分从 -0.729 跳到 0.009,第四主成分从 0.528 砸到 -2.165,波动幅度超过 2 个标准差,说明样本窗口内尾部结构变化很陡。 外汇和贵金属做类似降维时,这种陡变往往对应流动性切换或数据窗口重组,属高风险场景,不能直接当方向信号。 开 MT5 把这段日志贴进专家日志窗口,对照自己 PCA 函数的 Print 格式,确认第四主成分绝对值大于 2 的频次,再决定是否拿来做特征过滤。
CS class="num">0 class="num">10:class="num">17:class="num">31.957 PCA Test(NAS100,H1) [class="num">1.172130620033975,class="num">1.15846730049564,-class="num">0.7288256625700642,class="num">0.528192723531639] CS class="num">0 class="num">10:class="num">17:class="num">31.957 PCA Test(NAS100,H1) [class="num">0.1731572094549987,class="num">1.181160740523977,class="num">0.009348167869829477,-class="num">2.164823873278453] CS class="num">0 class="num">10:class="num">17:class="num">31.957 PCA Test(NAS100,H1) [-class="num">0.715386880184365,-class="num">0.05481045923432144,-class="num">2.328780161211247,class="num">0.3026082735855334] CS class="num">0 class="num">10:class="num">17:class="num">31.957 PCA Test(NAS100,H1) [class="num">1.594713612332284,-class="num">1.470442808583469,-class="num">0.003309859736641006,class="num">0.9432989819176616] CS class="num">0 class="num">10:class="num">17:class="num">31.957 PCA Test(NAS100,H1) [-class="num">0.4023014443028848,-class="num">1.250129598312728,-class="num">0.8385809690405054,-class="num">0.7565833632510734] CS class="num">0 class="num">10:class="num">17:class="num">31.957 PCA Test(NAS100,H1) [-class="num">1.68012890598631,class="num">0.05510361946569121,class="num">0.8561031894464458,class="num">1.335199254045385] CS class="num">0 class="num">10:class="num">17:class="num">31.957 PCA Test(NAS100,H1) [class="num">0.2786284867625921,-class="num">1.321151824538665,class="num">1.210589566461227,-class="num">1.06564543418136] CS class="num">0 class="num">10:class="num">17:class="num">31.957 PCA Test(NAS100,H1) [-class="num">1.074244269325531,-class="num">0.1690934905926844,class="num">0.4539901733759543,-class="num">0.1104988556867913] CS class="num">0 class="num">10:class="num">17:class="num">31.957 PCA Test(NAS100,H1) [class="num">1.072180711318756,class="num">0.8738669736790375,class="num">0.6820006878558206,class="num">0.6699931252073736]
从 PCA 分数里抠出主成分
拿到 PCA 分数后,真正的目标是提取主成分。非标准化分数的系数其实就是各 PCA 分数列自身的方差,算出来就能知道每个成分扛了多少波动。 选成分不是拍脑袋,常见有五套准则:特征值最大、累计方差比例、碎石图拐点、凯撒(特征值>1)、交叉验证。本库只实装了方差比例、凯撒、碎石图三种,理由是计算更轻、在 MT5 上跑起来不卡。 方差比例准则默认截在 90% 以上——比如 NAS100 的 H1 测试里,四个原始变量的 PCA 分数系数为 [2.4098, 1.4472, 0.0430, 0.1001],前两个就覆盖了约 94% 的总方差,4 维直接压到 2 维。凯撒准则则只留系数大于均值(此处均值约 0.999)的那些列。 提取函数的核心是先算百分比向量,再按你传的枚举走分支。下面这段代码是系数计算和 ExtractComponents 的骨架,注意 variance 那行就是系数的来源。
pca_scores_coefficients.Resize(cols); vector v_row; for (class="type">class="kw">ulong i=class="num">0; i<cols; i++) { v_row = pca_scores.Col(i); pca_scores_coefficients[i] = v_row.Var(); class=class="str">"cmt">//variance of the pca scores } class="num">2023.02.class="num">25 class="num">10:class="num">17:class="num">31.957 PCA Test(NAS100,H1) SCORES COEFF [class="num">2.409805431408367,class="num">1.447164215914809,class="num">0.04297514703684173,class="num">0.1000552056399828] enum criterion { CRITERION_VARIANCE, CRITERION_KAISER, CRITERION_SCREE_PLOT }; matrix Cpca::ExtractComponents(criterion CRITERION_) { vector vars = pca_scores_coefficients; vector vars_percents = (vars/(class="type">class="kw">double)vars.Sum())*class="num">100.0; class=class="str">"cmt">//--- for Kaiser class="type">class="kw">double vars_mean = pca_scores_coefficients.Mean(); class=class="str">"cmt">//--- for scree class="type">class="kw">double x[], y[]; class=class="str">"cmt">//--- matrix PCAS = {}; class="type">class="kw">double sum=class="num">0; class="type">class="kw">ulong max; vector v_cols = {}; class="kw">switch(CRITERION_) { case CRITERION_VARIANCE: class="macro">#ifdef DEBUG_MODE Print("vars percentages ",vars_percents); class="macro">#endif for (class="type">int i=class="num">0, count=class="num">0; i<(class="type">int)cols; i++) { count++; max = vars_percents.ArgMax(); sum += vars_percents[max]; vars_percents[max] = class="num">0; v_cols.Resize(count); v_cols[count-class="num">1] = (class="type">int)max; if (sum >= class="num">90.0) class="kw">break; } PCAS.Resize(rows, v_cols.Size()); for (class="type">class="kw">ulong i=class="num">0; i<v_cols.Size(); i++) PCAS.Col(pca_scores.Col((class="type">class="kw">ulong)v_cols[i]), i); class="kw">break; case CRITERION_KAISER: class="macro">#ifdef DEBUG_MODE
「用碎石图挑主成分与均值阈值截尾」
PCA 降维后有两种常见保留主成分的做法:一是按系数均值做阈值过滤,二是画碎石图(Scree Plot)肉眼看拐点。代码里 CRITERION_SCREE_PLOT 分支先把 pca_scores_coefficients 升序排再反转成降序,用 ScatterCurvePlots 画出「方差—PCA序号」散点,便于判断第几主成分后斜率变平。 均值截尾那段逻辑很直接:遍历 cols 个系数,凡是大于 vars_mean 的就 count++,并把对应列从 pca_scores 拷进 PCAS 矩阵,最后 Resize 到 rows×count。注意循环里 Resize 每次都按当前 count 重设列数,属于边扫边扩,MT5 里对中小样本量可以这么写,大矩阵建议先算好 count 再一次性分配。 实跑日志里 NAS100 H1 的 PCA Test 输出前五行为:[[-0.65,1.199],[1.819,1.393],[0.268,1.420],[-1.110,-0.065],[2.475,-1.768]],说明前两主成分已承载主要方差结构。外汇与贵金属用同样流程时波动更剧烈,主成分切换可能更频繁,属高风险场景,截尾阈值建议用样本内回测调,而非固定拍脑袋。 把 vars_percents 那行加上去就能直接看累计贡献率:vars/(double)vars.Sum()*100.0 把系数转成百分比,配合碎石图能更快定保留几维。开 MT5 把这段接在你自己的 PCA 函数尾端,跑一遍 EURUSD M15 对比 NAS100 的 count 差异,会比只读文档直观。
Print("var ",vars," scores mean ",vars_mean); class="macro">#endif vars = pca_scores_coefficients; for (class="type">class="kw">ulong i=class="num">0, count=class="num">0; i<cols; i++) if (vars[i] > vars_mean) { count++; PCAS.Resize(rows, count); PCAS.Col(pca_scores.Col(i), count-class="num">1); } class="kw">break; case CRITERION_SCREE_PLOT: v_cols.Resize(cols); for (class="type">class="kw">ulong i=class="num">0; i<v_cols.Size(); i++) v_cols[i] = (class="type">int)i+class="num">1; vars = pca_scores_coefficients; SortAscending(vars); class=class="str">"cmt">//Make sure they are in ascending first order ReverseOrder(vars); class=class="str">"cmt">//Set them to descending order VectorToArray(v_cols, x); VectorToArray(vars, y); plt.ScatterCurvePlots("Scree plot",x,y,"variance","PCA","Variance"); class=class="str">"cmt">//--- vars = pca_scores_coefficients; for (class="type">class="kw">ulong i=class="num">0, count=class="num">0; i<cols; i++) if (vars[i] > vars_mean) { count++; PCAS.Resize(rows, count); PCAS.Col(pca_scores.Col(i), count-class="num">1); } class="kw">break; } class="kw">return (PCAS); } vector vars = pca_scores_coefficients; vector vars_percents = (vars/(class="type">class="kw">double)vars.Sum())*class="num">100.0;
◍ NAS100 的 PCA 投影日志长这样
在 MT5 用 EA 跑主成分分析(PCA)时,专家日志里会直接吐出每个 K 线在降维空间的坐标。上面这段是 NAS100 的 H1 周期、同一毫秒(12:03:49.579)连续打印的投影样本,第一主成分 PC1 跨度从 -2.608 到 +2.475,第二主成分 PC2 从 -1.768 到 +1.420。 注意日志里出现了两次完全相同的五行坐标块,说明脚本对窗口做了重复计算或回环打印,并不是新数据。做成分监控时,这种重复行要先在代码层去重,否则会把样本量虚增一倍。 PC1 绝对值最大的点 [-2.608, 0.066] 和 [2.475, -1.768] 分别代表该窗口内偏离均值最远的两个状态,后续做聚类或异常检测时,这两个向量值得单独拉出来看。外汇与贵金属品种的高杠杆属性意味着这类统计特征只描述历史分布,实盘信号仍可能失效。
CS class="num">0 class="num">12:class="num">03:class="num">49.579 PCA Test(NAS100,H1) [-class="num">0.6245145789301378,-class="num">1.503882637300733] CS class="num">0 class="num">12:class="num">03:class="num">49.579 PCA Test(NAS100,H1) [-class="num">2.608156175249579,class="num">0.0662886285379769] CS class="num">0 class="num">12:class="num">03:class="num">49.579 PCA Test(NAS100,H1) [class="num">0.4325302694103054,-class="num">1.589321053467977] CS class="num">0 class="num">12:class="num">03:class="num">49.579 PCA Test(NAS100,H1) [-class="num">1.667608250048573,-class="num">0.2034163217366656] CS class="num">0 class="num">12:class="num">03:class="num">49.579 PCA Test(NAS100,H1) [class="num">1.664404875867474,class="num">1.051245703485609]] CS class="num">0 class="num">12:class="num">03:class="num">49.579 PCA Test(NAS100,H1) PCA&class="macro">#x27;S CS class="num">0 class="num">12:class="num">03:class="num">49.579 PCA Test(NAS100,H1) [[-class="num">0.6500472384886967,class="num">1.199407986803537] CS class="num">0 class="num">12:class="num">03:class="num">49.579 PCA Test(NAS100,H1) [class="num">1.819562596624738,class="num">1.393614599196321] CS class="num">0 class="num">12:class="num">03:class="num">49.579 PCA Test(NAS100,H1) [class="num">0.2688014256048517,class="num">1.420914385142756] CS class="num">0 class="num">12:class="num">03:class="num">49.579 PCA Test(NAS100,H1) [-class="num">1.110534258768705,-class="num">0.06593596223641518] CS class="num">0 class="num">12:class="num">03:class="num">49.579 PCA Test(NAS100,H1) [class="num">2.475561333978323,-class="num">1.768915328424386] CS class="num">0 class="num">12:class="num">03:class="num">49.579 PCA Test(NAS100,H1) [-class="num">0.6245145789301378,-class="num">1.503882637300733] CS class="num">0 class="num">12:class="num">03:class="num">49.579 PCA Test(NAS100,H1) [-class="num">2.608156175249579,class="num">0.0662886285379769] CS class="num">0 class="num">12:class="num">03:class="num">49.579 PCA Test(NAS100,H1) [class="num">0.4325302694103054,-class="num">1.589321053467977] CS class="num">0 class="num">12:class="num">03:class="num">49.579 PCA Test(NAS100,H1) [-class="num">1.667608250048573,-class="num">0.2034163217366656] CS class="num">0 class="num">12:class="num">03:class="num">49.579 PCA Test(NAS100,H1) [class="num">1.664404875867474,class="num">1.051245703485609]]
用 PCA 给十个振荡器瘦身
在 MT5 里堆了十种振荡器(ATR、熊市力量、MACD、柴金、CCI、DeMarker、推动力、动量、RSI、WPR),肉眼看线形高度重叠,但相关矩阵显示多数两两相关系数偏低——例如 ATR 与 MACD 仅 -0.0165,ATR 与 WPR 也才 0.3071。这说明它们虽同属振荡类,信息冗余没想象中高。 对 NAS100 的 H1 图表跑一遍相关矩阵后套用主成分分析,碎石图准则只挑出 3 个主成分,就把原本 10 个变量压到 3 个。变量数量砍掉七成,EA 的计算开销会明显降下来,这是实打实能省的资源。 下面这段是抓取十个指标缓冲并算相关矩阵的核心代码,直接在 MT5 脚本里改 period 和 bars 就能复现: 别把指标多当信息全。图表上叠一堆同类型振荡器,大概率只是视觉噪音;PCA 降维后留 3 个主成分,就够覆盖大部分波动结构,外汇和贵金属波动受消息面突袭,高风险下少算点反而更稳。
handles[class="num">0] = iATR(Symbol(),PERIOD_CURRENT, period); handles[class="num">1] = iBearsPower(Symbol(), PERIOD_CURRENT, period); handles[class="num">2] = iMACD(Symbol(),PERIOD_CURRENT,class="num">12, class="num">26,class="num">9,PRICE_CLOSE); handles[class="num">3] = iChaikin(Symbol(), PERIOD_CURRENT,class="num">12,class="num">26,MODE_SMMA,VOLUME_TICK); handles[class="num">4] = iCCI(Symbol(),PERIOD_CURRENT,period, PRICE_CLOSE); handles[class="num">5] = iDeMarker(Symbol(),PERIOD_CURRENT,period); handles[class="num">6] = iForce(Symbol(),PERIOD_CURRENT,period,MODE_EMA,VOLUME_TICK); handles[class="num">7] = iMomentum(Symbol(),PERIOD_CURRENT,period, PRICE_CLOSE); handles[class="num">8] = iRSI(Symbol(),PERIOD_CURRENT,period,PRICE_CLOSE); handles[class="num">9] = iWPR(Symbol(),PERIOD_CURRENT,period); for (class="type">int i=class="num">0; i<class="num">10; i++) { matrix_utiils.CopyBufferVector(handles[i],class="num">0,class="num">0,bars,buff_v); ind_Matrix.Col(buff_v, i); class=class="str">"cmt">//store each indicator in ind_matrix columns } Print("Oscillators Correlation Matrix\n",ind_Matrix.CorrCoef(class="kw">false));