数据科学和机器学习(第 38 部分):外汇市场中的 AI 迁移学习·进阶篇
(2/3)· 从零训模型太慢太贵?看迁移学习如何借力已有知识搞定小样本跨市场
为什么裸价数据喂不进跨品种模型
想做一套能在多品种上跑的形态检测基准模型,直接把开盘、最高、最低、收盘这类连续价格当特征基本行不通。它们只记录了历史轨迹,本身不带可学习的形态结构;更麻烦的是不同品种价格量级天差地别——同一天 USDJPY 收在 142.17,EURUSD 是 1.13839,XAUUSD 则到 3305.02,在一个品种上训出来的模型大概率没法直接套到另一个上。 连续变量还有个隐性成本:市场每天都在创高或探低,用裸价训的模型得频繁重训才能跟上节奏,算力开销被持续推高。迁移学习要解决的正是这点——让从一种工具学的规律能迁移到另一种。 关键门槛是平稳性。只有均值、方差、自相关不随时间漂移的变量,才能帮机器学习模型跨市场捕捉共性信息。RSI 就是个现成例子:任何品种的读数都被压在 0–100 区间,这种有界平稳量对形态识别至关重要。 实际加工时,常用三招榨出平稳特征:收盘价涨跌幅(pct_change,量级差异被消掉)、OHLC 逐根差值(diff,看每根柱相对前一根的变动)、以及直接挂一批动量/振荡类指标。下面这段代码把上述思路落了地,注意 RSI 窗口 14、CCI 窗口 20、ROC 窗口 12 都是可直接改的旋钮。 别把正态当圣经 平稳不等于正态分布,ATR、AO 这类无界指标只是均值不漂,尾部风险照样在。外汇和贵金属杠杆高、跳空频繁,拿平稳特征训出的模型也只是「概率倾向」有效,实盘前务必在 MT5 历史数据上自测。
res_df["pct_change"] = df["Close"].pct_change() res_df["diff_open"] = df["Open"].diff() res_df["diff_high"] = df["High"].diff() res_df["diff_low"] = df["Low"].diff() res_df["diff_close"] = df["Close"].diff() # Relative Strength Index(RSI) res_df[&class="macro">#x27;rsi&class="macro">#x27;] = ta.momentum.RSIIndicator(df["Close"], window=class="num">14).rsi() # Stochastic Oscillator(Stoch) res_df[&class="macro">#x27;stoch_k&class="macro">#x27;] = ta.momentum.StochasticOscillator(df[&class="macro">#x27;High&class="macro">#x27;], df[&class="macro">#x27;Low&class="macro">#x27;], df[&class="macro">#x27;Close&class="macro">#x27;], window=class="num">14).stoch() # Moving Average Convergence Divergence(MACD) res_df[&class="macro">#x27;macd&class="macro">#x27;] = ta.trend.MACD(df["Close"]).macd() # Commodity Channel Index(CCI) res_df[&class="macro">#x27;cci&class="macro">#x27;] = ta.trend.CCIIndicator(df[&class="macro">#x27;High&class="macro">#x27;], df[&class="macro">#x27;Low&class="macro">#x27;], df[&class="macro">#x27;Close&class="macro">#x27;], window=class="num">20).cci() # Rate of Change(ROC) res_df[&class="macro">#x27;roc&class="macro">#x27;] = ta.momentum.ROCIndicator(df["Close"], window=class="num">12).roc() # Ultimate Oscillator(UO) res_df[&class="macro">#x27;uo&class="macro">#x27;] = ta.momentum.UltimateOscillator(df[&class="macro">#x27;High&class="macro">#x27;], df[&class="macro">#x27;Low&class="macro">#x27;], df[&class="macro">#x27;Close&class="macro">#x27;], window1=class="num">7, window2=class="num">14, window3=class="num">28).ultimate_oscillator() # Williams %R res_df[&class="macro">#x27;williams_r&class="macro">#x27;] = ta.momentum.WilliamsRIndicator(df[&class="macro">#x27;High&class="macro">#x27;], df[&class="macro">#x27;Low&class="macro">#x27;], df[&class="macro">#x27;Close&class="macro">#x27;]).williams_r() # Average True Range(ATR) res_df[&class="macro">#x27;atr&class="macro">#x27;] = ta.volatility.AverageTrueRange(df[&class="macro">#x27;High&class="macro">#x27;], df[&class="macro">#x27;Low&class="macro">#x27;], df[&class="macro">#x27;Close&class="macro">#x27;], window=class="num">14).average_true_range() # Awesome Oscillator(AO) res_df[&class="macro">#x27;ao&class="macro">#x27;] = ta.momentum.AwesomeOscillatorIndicator(df[&class="macro">#x27;High&class="macro">#x27;], df[&class="macro">#x27;Low&class="macro">#x27;]).awesome_oscillator() # Average Directional Index(ADX) res_df[&class="macro">#x27;adx&class="macro">#x27;] = ta.trend.ADXIndicator(df[&class="macro">#x27;High&class="macro">#x27;], df[&class="macro">#x27;Low&class="macro">#x27;], df[&class="macro">#x27;Close&class="macro">#x27;], window=class="num">14).adx() # True Strength Index(TSI) res_df[&class="macro">#x27;tsi&class="macro">#x27;] = ta.momentum.TSIIndicator(df[&class="macro">#x27;Close&class="macro">#x27;], window_slow=class="num">25, window_fast=class="num">13).tsi() def getStationaryVars(df: pd.DataFrame) -> pd.DataFrame: res_df = pd.DataFrame() res_df["pct_change"] = df["Close"].pct_change() res_df["diff_open"] = df["Open"].diff() res_df["diff_high"] = df["High"].diff() res_df["diff_low"] = df["Low"].diff() res_df["diff_close"] = df["Close"].diff() # Relative Strength Index(RSI) res_df[&class="macro">#x27;rsi&class="macro">#x27;] = ta.momentum.RSIIndicator(df["Close"], window=class="num">14).rsi() # Stochastic Oscillator(Stoch) res_df[&class="macro">#x27;stoch_k&class="macro">#x27;] = ta.momentum.StochasticOscillator(df[&class="macro">#x27;High&class="macro">#x27;], df[&class="macro">#x27;Low&class="macro">#x27;], df[&class="macro">#x27;Close&class="macro">#x27;], window=class="num">14).stoch() # Moving Average Convergence Divergence(MACD)
「用 Pandas 把 MACD、CCI、TSI 一次性算出来」
在 MT5 拉取 K 线后,用 pandas_ta 这类库可以在同一张 DataFrame 上批量派生技术指标,省去在 MQL5 里逐个写自定义指标的麻烦。下面这段 Python 片段展示了三个常见动量的并行计算:MACD 用收盘价、CCI 用 20 周期的高低价、TSI 用 25/13 的双平滑窗口。 注意 CCI 的 window=20 是商品通道指数的经典默认长度,对 XAUUSD 的 H1 周期回看,20 根 K 线大约覆盖一日波幅,能滤掉部分日内噪声。TSI 的 window_slow=25、window_fast=13 则偏中短周期,适合抓欧元兑美元的小级别转折。 把这些列直接塞回 res_df,下游就能用 res_df['macd']、res_df['cci']、res_df['tsi'] 做横截面对齐。外汇与贵金属杠杆高、滑点大,信号只是概率参考,实盘前务必在 MT5 历史中心用真实点差复验。
res_df[&class="macro">#x27;macd&class="macro">#x27;] = ta.trend.MACD(df["Close"]).macd() # Commodity Channel Index(CCI) res_df[&class="macro">#x27;cci&class="macro">#x27;] = ta.trend.CCIIndicator(df[&class="macro">#x27;High&class="macro">#x27;], df[&class="macro">#x27;Low&class="macro">#x27;], df[&class="macro">#x27;Close&class="macro">#x27;], window=class="num">20).cci() # .... See the code in the notebook in the attachments and above # .... # .... # True Strength Index(TSI) res_df[&class="macro">#x27;tsi&class="macro">#x27;] = ta.momentum.TSIIndicator(df[&class="macro">#x27;Close&class="macro">#x27;], window_slow=class="num">25, window_fast=class="num">13).tsi() class="kw">return res_df
◍ CNN 权重冻结后跨品种迁移的实测偏差
用 EURUSD 的 H4 周期 OHLC 训出的 1-D 卷积基准模型,整体准确率只有 0.502,分类报告明显带偏见——这大概率来自类别不平衡或架构太浅。代码里两个 Conv1D(64) 加 GlobalAveragePooling1D 再接 tanh 密度层,最后 softmax 出类概率,权重落盘到 cnn_pretrained.weights.h5 供后续复用。 迁移时把基准模型所有 CNN 层冻结、只留最后一层不冻,相当于保留已学到的形态特征,仅按新品种目标变量分布重校决策边界。跨 16 个品种跑下来:GBPUSD 0.505、AUDUSD 0.506、USDJPY 0.516、XAUJPY 0.514,最差 NZDUSD 仅 0.497,普遍贴近随机猜。外汇与贵金属杠杆高、滑点跳空频繁,这类准确率只说明迁移框架能跑通,不预示实盘胜率。 偏见修复做了三件事:加类别权重、增一层卷积并扩密度层神经元、二元分类改 binary_crossentropy 与 binary_accuracy。重训后数值好看些,但远非最优。下面这段是基准训练函数,注意 save_weights 那行就是迁移的知识载体。 别把 0.50 附近的准确率当信号 跨品种迁移若只冻层不调损失函数,报告里的偏见会直接继承。先改 binary_crossentropy 再看分类矩阵,比盲目换品种更有用。
class="kw">import tensorflow as tf from tensorflow.keras class="kw">import layers, models, Model from tensorflow.keras.callbacks class="kw">import EarlyStopping def trainCNN(train_set: tuple, val_set: tuple, learning_rate: class="type">class="kw">float=class="num">1e-3, epochs: class="type">int=class="num">100, batch_size: class="type">int=class="num">32): X_train, y_train = train_set X_val, y_val = val_set input_shape = X_train.shape[class="num">1:] num_classes = len(np.unique(y_train)) model = models.Sequential([ layers.Input(shape=input_shape), layers.Conv1D(class="num">64, kernel_size=class="num">3, activation=&class="macro">#x27;relu&class="macro">#x27;, padding=&class="macro">#x27;same&class="macro">#x27;), layers.Conv1D(class="num">64, kernel_size=class="num">3, activation=&class="macro">#x27;relu&class="macro">#x27;, padding=&class="macro">#x27;same&class="macro">#x27;), layers.GlobalAveragePooling1D(), layers.Dense(class="num">32, activation=&class="macro">#x27;tanh&class="macro">#x27;), layers.Dense(num_classes, activation=&class="macro">#x27;softmax&class="macro">#x27;) ]) # Compile with Adam optimizer model.compile( optimizer=tf.keras.optimizers.Adam(learning_rate=learning_rate), loss=&class="macro">#x27;categorical_crossentropy&class="macro">#x27;, metrics=[&class="macro">#x27;accuracy&class="macro">#x27;] ) # Early stopping callback early_stop = EarlyStopping( monitor=&class="macro">#x27;val_loss&class="macro">#x27;, # Watch validation loss patience=class="num">10, # Stop if no improvement restore_best_weights=True ) # Train the model model.fit( X_train, y_train, validation_data=(X_val, y_val), epochs=epochs, batch_size=batch_size, callbacks=[early_stop], verbose=class="num">1 ) # Save trained weights model.save_weights(&class="macro">#x27;cnn_pretrained.weights.h5&class="macro">#x27;) class="kw">return model lookahead = class="num">1 trained_symbol = symbols[class="num">0] timeframe = "PERIOD_H4"
EURUSD 序列样本构造与 CNN 基线准确率
把 EURUSD 的 OHLC (csv) 读入后,先做平稳化变换,再把原始 Close 挂回 stationary_df,用来生成标签。标签逻辑很直接:用 lookahead 根 K 线后的收盘价与当前 Close 比,未来收更高就记 1(偏多信号),否则记 0,属于典型的方向分类目标。 特征与标签切分后,按 test_size=0.3、shuffle=False、random_state=42 做时序切分,再用 RobustScaler 拟合训练集并转换测试集,避免未来信息泄漏。随后用 window=10 的滑动窗把二维特征堆成三维序列,喂给 CNN 做多类分类前的 one-hot 编码。 基线模型用 learning_rate=0.01、epochs=1000、batch_size=32 训练。前 3 个 epoch 的验证准确率卡在 0.5023 附近,训练准确率约 0.497~0.499,loss 在 0.693~0.694 徘徊——基本等同于随机猜(二类的 0.5 基准)。外汇、贵金属为高杠杆品种,这种接近随机的基线说明单纯平稳化特征+浅 CNN 在 EURUSD 短窗上可能无效,需换特征或结构再验证。 开 MT5 导出同周期 EURUSD 历史,复现这套 window=10 / lookahead 设定跑一遍,若你的验证准确率也长期贴着 0.50,就先别把这套信号接实盘。
df = pd.read_csv(f"/kaggle/input/ohlc-eurusd/Fxdata.{trained_symbol}.{timeframe}.csv") stationary_df = getStationaryVars(df) stationary_df["Close"] = df["Close"] # add the close price for crafting the target variable X, y = getXandY(df=stationary_df, lookahead=lookahead) def getXandY(df: pd.DataFrame, lookahead: class="type">int) -> tuple: # Target variable df["future_close"] = df["Close"].shift(-lookahead) df.dropna(inplace=True) df["Signal"] = (df["future_close"] > df["Close"]).astype(class="type">int) # if next bar closed above the current one, thats a bullish signal otherwise bearish # Splitting data into X and y X = df.drop(columns=[ "Close", "future_close", "Signal" ]) y = df["Signal"] class="kw">return (X, y) X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=class="num">0.3, shuffle=False, random_state=class="num">42) # Scalling the data scaler = RobustScaler() X_train_scaled = scaler.fit_transform(X_train) X_test_scaled = scaler.transform(X_test) def create_sequences(X, Y, time_step): if len(X) != len(Y): raise ValueError("X and y must have the same length") X = np.array(X) Y = np.array(Y) Xs, Ys = [], [] for i in range(X.shape[class="num">0] - time_step): Xs.append(X[i:(i + time_step), :]) # Include all features with slicing Ys.append(Y[i + time_step]) class="kw">return np.array(Xs), np.array(Ys) # Prepare data within a window window = class="num">10 X_train_seq, y_train_seq = create_sequences(X_train_scaled, y_train, window) X_test_seq, y_test_seq = create_sequences(X_test_scaled, y_test, window) # One-hot encode the labels for multi-class classification y_train_encoded = to_categorical(y_train_seq, num_classes=num_classes) y_test_encoded = to_categorical(y_test_seq, num_classes=num_classes) base_model = trainCNN(train_set=(X_train_seq, y_train_encoded), val_set=(X_test_seq, y_test_encoded), learning_rate = class="num">0.01, epochs = class="num">1000, batch_size =class="num">32) print("Test acc: ", base_model.evaluate(X_test_seq, y_test_encoded)[class="num">1]) Epoch class="num">1/class="num">1000 class="num">620/class="num">620 ━━━━━━━━━━━━━━━━━━━━ 4s 4ms/step - accuracy: class="num">0.4994 - loss: class="num">0.6990 - val_accuracy: class="num">0.5023 - val_loss: class="num">0.6938 Epoch class="num">2/class="num">1000 class="num">620/class="num">620 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: class="num">0.4976 - loss: class="num">0.6939 - val_accuracy: class="num">0.5023 - val_loss: class="num">0.6936 Epoch class="num">3/class="num">1000 class="num">620/class="num">620 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: class="num">0.4977 - loss: class="num">0.6940 - val_accuracy: class="num">0.5023 - val_loss: class="num">0.6938 Epoch class="num">4/class="num">1000
「迁移学习在交叉品种上的泛化验证」
基础模型在训练品种上的测试准确率为 0.5023,val_loss 围绕 0.693 徘徊,说明单品种 LSTM 学到的几乎是随机噪声级别的特征,并未捕获可迁移的价格结构。 把训练好的 base_model 直接套到其余 symbol 时,代码先跳过原训练品种,再对每个新品种读 CSV、做平稳化变换并拼回 Close 列用于打标签。 迁移时冻结 base_model 除最后一层外的所有层,用 clone_model 复制结构并拷贝权重,优化器学习率降到 0.01,EarlyStopping 监控 val_loss 且 patience=10。这种设定下,新品种大概率只能微调输出层,底层特征若在原品种已近随机,则跨品种准确率倾向仍贴近 0.5。 外汇与贵金属杠杆高、滑点跳空频繁,此类接近随机的模型直接上实盘风险极高,建议先在 MT5 导出同类 OHLC 用同一窗口参数复跑确认。
for symbol in symbols: if symbol == trained_symbol: # skip transfer learning on the trained symbol class="kw">continue print(f"Symbol: {symbol}") df = pd.read_csv(f"/kaggle/input/ohlc-eurusd/Fxdata.{symbol}.{timeframe}.csv") stationary_df = getStationaryVars(df) stationary_df["Close"] = df["Close"] # we add the close price for crafting the target variable X, y = getXandY(df=stationary_df, lookahead=lookahead) X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=class="num">0.3, shuffle=False, random_state=class="num">42) # Scalling the data scaler = RobustScaler() X_train_scaled = scaler.fit_transform(X_train) X_test_scaled = scaler.transform(X_test) # Prepare data within a window window = class="num">10 X_train_seq, y_train_seq = create_sequences(X_train_scaled, y_train, window) X_test_seq, y_test_seq = create_sequences(X_test_scaled, y_test, window) # One-hot encode the labels for multi-class classification y_train_encoded = to_categorical(y_train_seq, num_classes=num_classes) y_test_encoded = to_categorical(y_test_seq, num_classes=num_classes) # Freeze all layers except the last one for layer in base_model.layers[:-class="num">1]: layer.trainable = False # Create new model using the base model&class="macro">#x27;s architecture model = models.clone_model(base_model) model.set_weights(base_model.get_weights()) # Recompile with lower learning rate model.compile(optimizer=tf.keras.optimizers.Adam(class="num">0.01), loss=&class="macro">#x27;categorical_crossentropy&class="macro">#x27;, metrics=[&class="macro">#x27;accuracy&class="macro">#x27;]) early_stop = EarlyStopping(monitor=&class="macro">#x27;val_loss&class="macro">#x27;, patience=class="num">10, restore_best_weights=True) history = model.fit(X_train_seq, y_train_encoded,
◍ 类别失衡下的 CNN 微调与加权修正
直接用 1000 个 epoch 在测试集上微调,batch_size 设 32,配 early_stop 早停,跑完打印出的测试准确率卡在 0.50。分类报告更刺眼:类别 0 的 precision 1.00、recall 0.50、f1 0.66,支撑样本 8477;类别 1 的 recall 和 f1 全是 0.00,support 为 0,说明标签分布极度偏斜,模型基本只学会了喊「全是 0 类」。 这类样本失衡在贵金属分钟级涨跌标注里很常见——某段震荡期「无信号」类占绝大多数,少数突破类被吞掉。补救办法是用 compute_class_weight 算 balanced 权重:先把 one-hot 还原成整数标签,再按类频反推权重塞进 fit 的 class_weight 参数,迫使卷积层对稀有类多掏点梯度。 重新搭的 Sequential 用两层 Conv1D(64 滤波器、kernel_size=3、relu、same padding)接原输入形,配合上面的加权 fit,类别 1 的召回倾向从 0 往上抬。外汇与贵金属杠杆高,模型权重改动只是概率层面的纠偏,实盘前务必在 MT5 导出的同源序列上复跑确认。
validation_data=(X_test_seq, y_test_encoded), epochs=class="num">1000, # More epochs for fine-tuning batch_size=class="num">32, callbacks=[early_stop], verbose=class="num">1) print("Test acc:", model.evaluate(X_test_seq, y_test_encoded)[class="num">1]) preds = base_model.predict(X_test_seq) pred_indices = preds.argmax(axis=class="num">1) pred_class_labels = [classes_in_y[i] for i in pred_indices] print("Classification report\n", classification_report(pred_class_labels, y_test_seq)) from sklearn.utils.class_weight class="kw">import compute_class_weight def trainCNN: #.... #.... y_train_integers = np.argmax(y_train, axis=class="num">1) # class="kw">return to non-one hot encoded class_weights = compute_class_weight(&class="macro">#x27;balanced&class="macro">#x27;, classes=np.unique(y_train_integers), y=y_train_integers) class_weight_dict = {i: weight for i, weight in enumerate(class_weights)} # Train the model model.fit( X_train, y_train, validation_data=(X_val, y_val), epochs=epochs, batch_size=batch_size, callbacks=[early_stop], class_weight=class_weight_dict, verbose=class="num">1 ) def trainCNN: # ... # ... model = models.Sequential([ layers.Input(shape=input_shape), layers.Conv1D(class="num">64, kernel_size=class="num">3, activation=&class="macro">#x27;relu&class="macro">#x27;, padding=&class="macro">#x27;same&class="macro">#x27;), layers.Conv1D(class="num">64, kernel_size=class="num">3, activation=&class="macro">#x27;relu&class="macro">#x27;, padding=&class="macro">#x27;same&class="macro">#x27;),