神经网络变得轻松（第十一部分）：自 GPT 获取·进阶篇
（2/3）·解码器-only的GPT和前篇变换器差在哪，又该怎么写进EA的神经类里
多头注意力里的梯度回流路径
在反向传播循环的开头，先由 ConvolutionInputGradients 方法计算前馈模块全连接层传来的误差梯度，方法跑完立即释放本地缓冲区。因为整个算法走的是数据流贯穿，误差梯度也要走同一套路：把前馈模块拿到的梯度和上一层传来的梯度相加，再做一次常规化来压住崩溃梯度（vanishing/exploding），这部分封装在 SumAndNormilize 里。 随后把误差梯度按关注目击者（attention heads）拆分，通过 ConvolutionInputGradients 作用于 W0 矩阵实现；AttentionInsideGradients 方法则规划沿 head 维度的梯度传播。循环末尾，把上一次迭代收到的梯度经级联张量 QKV_Weights 送回上一层，并和自注意力前馈模块的梯度求和、常规化。注意前一层的数据缓冲区要留在 GPU 内存，别误删。 调用频次最高的是 ConvolutionInputGradients，它借用了卷积层内核并针对当前任务裁剪过。入参包含权重张量、下一层梯度、上一层输出、结果存放张量，以及输入/输出窗口尺寸和激活函数标识；方法先校验指针有效性，在 GPU 开缓冲区，再发起 OpenCL 内核调用。 MHAttentionInsideGradients 内核以二维线程启动（序列元素 × 关注目击者），先取序列与 head 序号和尺寸、重算 Scores 系数，再循环算 Scores 梯度，用屏障同步确保所有线程完成后再切到 QKV 梯度循环。权重更新走 updateInputWeights，顺序调 ConvolutuionUpdateWeights 即可，思路和前几篇的 feedForward 一致。 基类改动很机械：加新类标识符、在定义模块补 OpenCL 内核常量、构造函数里声明并实例化新神经元，再把新类接进 CNeuronBaseOCL 的调度方法。附件里有完整类和方法代码，开 MT5 把头文件挂上就能编译验证。
class CNeuronMLMHAttentionOCL : class="kw">public CNeuronBaseOCL { class="kw">protected: class="type">uint iLayers; class=class="str">"cmt">/// Number of inner layers class="type">uint iHeads; class=class="str">"cmt">/// Number of heads class="type">uint iWindow; class=class="str">"cmt">/// Input window size class="type">uint iUnits; class=class="str">"cmt">/// Number of units class="type">uint iWindowKey; class=class="str">"cmt">/// Size of Key/Query window class=class="str">"cmt">//--- CCollection *QKV_Tensors; class=class="str">"cmt">/// The collection of tensors of Queries, Keys and Values CCollection *QKV_Weights; class=class="str">"cmt">/// The collection of Matrix of weights to previous layer CCollection *S_Tensors; class=class="str">"cmt">/// The collection of Scores tensors CCollection *AO_Tensors; class=class="str">"cmt">/// The collection of Attention Out tensors CCollection *FF_Tensors; class=class="str">"cmt">/// The collection of tensors of Feed Forward output
◍ 多头部注意力层的 OpenCL 接口拆解
在 MT5 的神经网络扩展里，CNeuronMLMHAttentionOCL 把多头注意力机制的计算全部下沉到 OpenCL 内核，类内只保留调度入口。理解这些虚函数，等于看清了 GPU 上一次前向和反向传播究竟调了哪些 kernel。 前向侧由 feedForward 接上层指针拉起基础全连接，ConvolutionForward 单独处理带 window 与 window_out 参数的卷积前向，AttentionScore 与 AttentionOut 则分别算注意力分数与输出，mask 默认开启以避免未来信息泄漏。SumAndNormilize 内部连发 SumMatrix 与 Normalize 两个内核，完成残差相加后的层归一化。 反向侧按优化器分支：updateInputWeights 依据 ENUM_OPTIMIZATION 在 Momentum 与 Adam 内核间二选一；卷积梯度由 ConvolutionInputGradients 与 ConvolutuionUpdateWeights 配合传递并更新，后者同样按优化类型切内核。AttentionInsideGradients 负责把梯度穿过注意力层回传。 Init 的形参直接决定拓扑：window 控制输入输出窗步长，heads 设头数，units_count 与 layers 定神经元和层数，optimization_type 在初始化时就锁死了反向更新路径。开 MT5 把这类头文件丢进 MetaEditor 搜 CNeuronMLMHAttentionOCL，对照调用栈能验证上述内核名是否真实存在。
CCollection *FF_Weights; class=class="str">"cmt">///< The collection of Matrix of Feed Forward weights class=class="str">"cmt">///\ingroup neuron_base_ff class="kw">virtual class="type">bool feedForward(CNeuronBaseOCL *NeuronOCL); class=class="str">"cmt">///< \brief Feed Forward method of calling kernel ::FeedForward().@param NeuronOCL Pointer to previos layer. class="kw">virtual class="type">bool ConvolutionForward(CBufferDouble *weights, CBufferDouble *inputs,CBufferDouble *outputs, class="type">uint window, class="type">uint window_out, ENUM_ACTIVATION activ); class=class="str">"cmt">///< \brief Convolution Feed Forward method of calling kernel ::FeedForwardConv(). class="kw">virtual class="type">bool AttentionScore(CBufferDouble *qkv, CBufferDouble *scores, class="type">bool mask=true); class=class="str">"cmt">///< \brief Multi-heads attention scores method of calling kernel ::MHAttentionScore(). class="kw">virtual class="type">bool AttentionOut(CBufferDouble *qkv, CBufferDouble *scores, CBufferDouble *out); class=class="str">"cmt">///< \brief Multi-heads attention out method of calling kernel ::MHAttentionOut(). class="kw">virtual class="type">bool SumAndNormilize(CBufferDouble *tensor1, CBufferDouble *tensor2, CBufferDouble *out); class=class="str">"cmt">///< \brief Method sum and normalize class="num">2 tensors by calling class="num">2 kernels ::SumMatrix() and ::Normalize(). class=class="str">"cmt">///\ingroup neuron_base_opt class="kw">virtual class="type">bool updateInputWeights(CNeuronBaseOCL *NeuronOCL); class=class="str">"cmt">///< Method for updating weights.\details Calling one of kernels ::UpdateWeightsMomentum() or ::UpdateWeightsAdam() in depends on optimization type(class="macro">#ENUM_OPTIMIZATION).@param NeuronOCL Pointer to previos layer. class="kw">virtual class="type">bool ConvolutuionUpdateWeights(CBufferDouble *weights, CBufferDouble *gradient, CBufferDouble *inputs, CBufferDouble *momentum1, CBufferDouble *momentum2, class="type">uint window, class="type">uint window_out); class=class="str">"cmt">///< Method for updating weights in convolution layer.\details Calling one of kernels ::UpdateWeightsConvMomentum() or ::UpdateWeightsConvAdam() in depends on optimization type(class="macro">#ENUM_OPTIMIZATION). class="kw">virtual class="type">bool ConvolutionInputGradients(CBufferDouble *weights, CBufferDouble *gradient, CBufferDouble *inputs, CBufferDouble *inp_gradient, class="type">uint window, class="type">uint window_out, class="type">uint activ); class=class="str">"cmt">///< Method of passing gradients through a convolutional layer. class="kw">virtual class="type">bool AttentionInsideGradients(CBufferDouble *qkv,CBufferDouble *qkv_g,CBufferDouble *scores,CBufferDouble *scores_g,CBufferDouble *gradient); class=class="str">"cmt">///< Method of passing gradients through attention layer. class="kw">public: class=class="str">"cmt">/** Constructor */CNeuronMLMHAttentionOCL(class="type">void); class=class="str">"cmt">/** Destructor */~CNeuronMLMHAttentionOCL(class="type">void); class="kw">virtual class="type">bool Init(class="type">uint numOutputs,class="type">uint myIndex,COpenCLMy *open_cl, class="type">uint window, class="type">uint window_key, class="type">uint heads, class="type">uint units_count, class="type">uint layers, ENUM_OPTIMIZATION optimization_type); class=class="str">"cmt">///< Method of initialization class.@param[in] numOutputs Number of connections to next layer.@param[in] myIndex Index of neuron in layer.@param[in] open_cl Pointer to class="macro">#COpenCLMy object.@param[in] window Size of in/out window and step.@param[in] units_countNumber of neurons.@param[in] optimization_type Optimization type(class="macro">#ENUM_OPTIMIZATION)@class="kw">return Boolen result of operations. class="kw">virtual class="type">bool calcInputGradients(CNeuronBaseOCL *prevLayer); class=class="str">"cmt">///< Method to transfer gradients to previous layer @param[in] prevLayer Pointer to previous layer. class=class="str">"cmt">//---
「多头注意力神经元的构造与张量尺寸」
CNeuronMLMHAttentionOCL 这个类封装了基于 OpenCL 的多头自注意力机制，用在 MT5 的行情序列建模上。构造函数里把 iLayers、iHeads、iWindow、iWindowKey、iUnits 全部初始化为 0，随后新建 6 个 CCollection 对象分别管 QKV 张量、QKV 权重、Score 张量、Attention 输出、前馈张量和前馈权重。 析构时逐个用 CheckPointer 判断有效性再 delete，避免悬空指针。这一套在 EA 退出或重新训练时很关键，否则 MT5 终端可能悄悄漏内存。 Init 方法里先用 fmax 把 window、window_key、units_count、heads、layers 都兜底成至少 1，再算各张量尺寸：QKV 张量 = 3*iWindowKey*iHeads*iUnits，QKV 权重矩阵 = 3*(iWindow+1)*iWindowKey*iHeads，Score 张量 = iUnits*iUnits*iHeads，多头输出 = iWindowKey*iHeads*iUnits，本层输出 = iWindow*iUnits。 以外汇或贵金属品种跑这套网络属于高风险实验，过拟合和历史漂移都可能让样本外表现明显走弱。改 iHeads 或 iUnits 会直接撬动显存占用，建议先在策略测试器里用小窗口验证再放大。
class="kw">virtual class="type">int Type(class="type">void) class="kw">const { class="kw">return defNeuronMLMHAttentionOCL; }class=class="str">"cmt">///< Identificator of class.@class="kw">return Type of class class=class="str">"cmt">//--- methods for working with files class="kw">virtual class="type">bool Save(class="type">int class="kw">const file_handle); class=class="str">"cmt">///< Save method @param[in] file_handle handle of file @class="kw">return logical result of operation class="kw">virtual class="type">bool Load(class="type">int class="kw">const file_handle); class=class="str">"cmt">///< Load method @param[in] file_handle handle of file @class="kw">return logical result of operation }; CNeuronMLMHAttentionOCL::CNeuronMLMHAttentionOCL(class="type">void) : iLayers(class="num">0), iHeads(class="num">0), iWindow(class="num">0), iWindowKey(class="num">0), iUnits(class="num">0) { QKV_Tensors=new CCollection(); QKV_Weights=new CCollection(); S_Tensors=new CCollection(); AO_Tensors=new CCollection(); FF_Tensors=new CCollection(); FF_Weights=new CCollection(); } CNeuronMLMHAttentionOCL::~CNeuronMLMHAttentionOCL(class="type">void) { if(CheckPointer(QKV_Tensors)!=POINTER_INVALID) class="kw">delete QKV_Tensors; if(CheckPointer(QKV_Weights)!=POINTER_INVALID) class="kw">delete QKV_Weights; if(CheckPointer(S_Tensors)!=POINTER_INVALID) class="kw">delete S_Tensors; if(CheckPointer(AO_Tensors)!=POINTER_INVALID) class="kw">delete AO_Tensors; if(CheckPointer(FF_Tensors)!=POINTER_INVALID) class="kw">delete FF_Tensors; if(CheckPointer(FF_Weights)!=POINTER_INVALID) class="kw">delete FF_Weights; } class="type">bool CNeuronMLMHAttentionOCL::Init(class="type">uint numOutputs,class="type">uint myIndex,COpenCLMy *open_cl,class="type">uint window,class="type">uint window_key,class="type">uint heads,class="type">uint units_count,class="type">uint layers,ENUM_OPTIMIZATION optimization_type) { if(!CNeuronBaseOCL::Init(numOutputs,myIndex,open_cl,window*units_count,optimization_type)) class="kw">return class="kw">false; class=class="str">"cmt">//--- iWindow=fmax(window,class="num">1); iWindowKey=fmax(window_key,class="num">1); iUnits=fmax(units_count,class="num">1); iHeads=fmax(heads,class="num">1); iLayers=fmax(layers,class="num">1); class="type">uint num=class="num">3*iWindowKey*iHeads*iUnits; class=class="str">"cmt">//Size of QKV tensor class="type">uint qkv_weights=class="num">3*(iWindow+class="num">1)*iWindowKey*iHeads; class=class="str">"cmt">//Size of weights&class="macro">#x27; matrix of QKV tensor class="type">uint scores=iUnits*iUnits*iHeads; class=class="str">"cmt">//Size of Score tensor class="type">uint mh_out=iWindowKey*iHeads*iUnits; class=class="str">"cmt">//Size of multi-heads self-attention class="type">uint out=iWindow*iUnits; class=class="str">"cmt">//Size of our tensor
Transformer 张量缓冲的堆内存预分配
在 MT5 里手搓 Transformer 类模型，第一道坎不是矩阵乘，而是把每层要用到的中间张量提前在堆上开好。下面这段初始化逻辑按层循环，每层又跑两遍（d<2 大概率对应前向与反向或双流），把 QKV、注意力分数、多头输出、注意力输出、前馈层缓冲依次挂进各自的动态数组。 关键尺寸先算清楚：w0=(iWindowKey+1)*iHeads*iWindow 是 W0 张量体积，ff_1=4*(iWindow+1)*iWindow 是第一前馈层权矩阵尺寸，ff_2=(4*iWindow+1)*iWindow 是第二前馈层权矩阵尺寸。假设 iWindow=32、iHeads=8、iWindowKey=3，那么 w0=4*8*32=1024，ff_1=4*33*32=4224，ff_2=(128+1)*32=4128——这些数直接决定你 EA 的内存占用峰值。 [CODE] uint w0=(iWindowKey+1)*iHeads*iWindow; //W0 张量尺寸 uint ff_1=4*(iWindow+1)*iWindow; //第一前馈层权矩阵尺寸 uint ff_2=(4*iWindow+1)*iWindow; //第二前馈层权矩阵尺寸 for(uint i=0; i<iLayers; i++) { CBufferDouble *temp=NULL; for(int d=0; d<2; d++) { //--- 初始化 QKV 张量 temp=new CBufferDouble(); if(CheckPointer(temp)==POINTER_INVALID) return false; if(!temp.BufferInit(num,0)) return false; if(!QKV_Tensors.Add(temp)) return false; //--- 初始化 scores temp=new CBufferDouble(); if(CheckPointer(temp)==POINTER_INVALID) return false; if(!temp.BufferInit(scores,0)) return false; if(!S_Tensors.Add(temp)) return false; //--- 初始化多头注意力输出 temp=new CBufferDouble(); if(CheckPointer(temp)==POINTER_INVALID) return false; if(!temp.BufferInit(mh_out,0)) return false; if(!AO_Tensors.Add(temp)) return false; //--- 初始化注意力输出 temp=new CBufferDouble(); if(CheckPointer(temp)==POINTER_INVALID) return false; if(!temp.BufferInit(out,0)) return false; if(!FF_Tensors.Add(temp)) return false; //--- 初始化前馈层 1 temp=new CBufferDouble(); if(CheckPointer(temp)==POINTER_INVALID) [/CODE] 逐行拆一下：w0/ff_1/ff_2 三个 uint 只是记账用尺寸，不分配内存；外层 for 走 iLayers 层，内层 for 走 d=0,1 两次。每次 new CBufferDouble() 都在堆上开一块，BufferInit(num,0) 按 num 长度清零——num、scores、mh_out、out 分别是各张量的元素数，必须在此之前算好。任何一步 CheckPointer 失效或 BufferInit 失败就 return false，调用方得自己释放已挂入数组的 temp，否则 EA 跑久了内存泄漏会把终端拖崩。 开 MT5 把这段塞进你的 CIneuralNet 类构造里，先打日志看 iLayers=6、iWindow=32 时 AO_Tensors 总项数是否是 6*2=12。外汇和贵金属杠杆高，这类自定义推理 EA 未经充分回测直接上实盘，滑点和重算延迟可能让信号失效，风险自负。
class="type">uint w0=(iWindowKey+class="num">1)*iHeads*iWindow; class=class="str">"cmt">//Size W0 tensor class="type">uint ff_1=class="num">4*(iWindow+class="num">1)*iWindow; class=class="str">"cmt">//Size of weights&class="macro">#x27; matrix class="num">1-st feed forward layer class="type">uint ff_2=(class="num">4*iWindow+class="num">1)*iWindow; class=class="str">"cmt">//Size of weights&class="macro">#x27; matrix class="num">2-nd feed forward layer for(class="type">uint i=class="num">0; i<iLayers; i++) { CBufferDouble *temp=NULL; for(class="type">int d=class="num">0; d<class="num">2; d++) { class=class="str">"cmt">//--- Initialize QKV tensor temp=new CBufferDouble(); if(CheckPointer(temp)==POINTER_INVALID) class="kw">return class="kw">false; if(!temp.BufferInit(num,class="num">0)) class="kw">return class="kw">false; if(!QKV_Tensors.Add(temp)) class="kw">return class="kw">false; class=class="str">"cmt">//--- Initialize scores temp=new CBufferDouble(); if(CheckPointer(temp)==POINTER_INVALID) class="kw">return class="kw">false; if(!temp.BufferInit(scores,class="num">0)) class="kw">return class="kw">false; if(!S_Tensors.Add(temp)) class="kw">return class="kw">false; class=class="str">"cmt">//--- Initialize multi-heads attention out temp=new CBufferDouble(); if(CheckPointer(temp)==POINTER_INVALID) class="kw">return class="kw">false; if(!temp.BufferInit(mh_out,class="num">0)) class="kw">return class="kw">false; if(!AO_Tensors.Add(temp)) class="kw">return class="kw">false; class=class="str">"cmt">//--- Initialize attention out temp=new CBufferDouble(); if(CheckPointer(temp)==POINTER_INVALID) class="kw">return class="kw">false; if(!temp.BufferInit(out,class="num">0)) class="kw">return class="kw">false; if(!FF_Tensors.Add(temp)) class="kw">return class="kw">false; class=class="str">"cmt">//--- Initialize Feed Forward class="num">1 temp=new CBufferDouble(); if(CheckPointer(temp)==POINTER_INVALID)
◍ Transformer 前向张量与权重的初始化落点
这段初始化逻辑跑在模型构建阶段，核心是把每一层的 Feed Forward 张量、QKV 权重和前馈权重逐一挂到容器里，任何一步分配失败就直接 return false 中断，保证后续推理不会拿到空指针。 循环里对最后一层做了特判：i==iLayers-1 时只把 Output 或 Gradient 缓冲塞进 FF_Tensors 然后 continue，其余层则按 out 长度初始化并预留 4*out 的临时缓冲（BufferInit 第二参数填 0 表示初值清零）。 权重侧分三块：QKV_Weights 按 qkv_weights 预留并循环 GenerateWeight() 填充；FF_Weights 对应 w0 长度；另一块前馈权重按 ff_1 预留。你在 MT5 里改网络结构时，若想加宽注意力头，优先调 qkv_weights 与 ff_1 这两个量，能直接看到 Reserve 内存占用变化。 外汇与贵金属行情下跑这类自定义神经网络指标属于高风险实验，过拟合与实时重算延迟都可能导致信号失真，建议先在策略测试器用历史数据验证权重初始化稳定性。
class="kw">return class="kw">false; if(!temp.BufferInit(class="num">4*out,class="num">0)) class="kw">return class="kw">false; if(!FF_Tensors.Add(temp)) class="kw">return class="kw">false; class=class="str">"cmt">//--- Initialize Feed Forward class="num">2 if(i==iLayers-class="num">1) { if(!FF_Tensors.Add(d==class="num">0 ? Output : Gradient)) class="kw">return class="kw">false; class="kw">continue; } temp=new CBufferDouble(); if(CheckPointer(temp)==POINTER_INVALID) class="kw">return class="kw">false; if(!temp.BufferInit(out,class="num">0)) class="kw">return class="kw">false; if(!FF_Tensors.Add(temp)) class="kw">return class="kw">false; } class=class="str">"cmt">//--- Initialize QKV weights temp=new CBufferDouble(); if(CheckPointer(temp)==POINTER_INVALID) class="kw">return class="kw">false; if(!temp.Reserve(qkv_weights)) class="kw">return class="kw">false; for(class="type">uint w=class="num">0; w<qkv_weights; w++) { if(!temp.Add(GenerateWeight())) class="kw">return class="kw">false; } if(!QKV_Weights.Add(temp)) class="kw">return class="kw">false; class=class="str">"cmt">//--- Initialize Weights0 temp=new CBufferDouble(); if(CheckPointer(temp)==POINTER_INVALID) class="kw">return class="kw">false; if(!temp.Reserve(w0)) class="kw">return class="kw">false; for(class="type">uint w=class="num">0; w<w0; w++) { if(!temp.Add(GenerateWeight())) class="kw">return class="kw">false; } if(!FF_Weights.Add(temp)) class="kw">return class="kw">false; class=class="str">"cmt">//--- Initialize FF Weights temp=new CBufferDouble(); if(CheckPointer(temp)==POINTER_INVALID) class="kw">return class="kw">false; if(!temp.Reserve(ff_1)) class="kw">return class="kw">false;
「前馈与注意力权重的初始化落点」
这段逻辑紧接前面的结构定义，把两层前馈网络（ff_1、ff_2 神经元数）的权重容器逐个塞进 FF_Weights，每层都先 Reserve 出空间再循环填 GenerateWeight() 的返回值，任何一次 Add 失败就直接 return false 中断。 随后按优化方式分流：SGD 只跑 1 轮（d<1），其他优化器跑 2 轮（d<2）。每轮里先给 QKV_Weights 挂一个用 qkv_weights 初始化的缓冲，再往 FF_Weights 追加 w0、ff_1、ff_2 三个零初始化缓冲，指针无效或 BufferInit 失败同样立即退出。 在 MT5 里跑这套，你可以把 ff_1、ff_2 改成 16/8 之类的小值先验证内存分配是否按预期翻倍；外汇与贵金属品种上接这种模型前请记住杠杆交易的高风险，权重初始化只是起点，过拟合倾向得靠样本外数据压住。
for(class="type">uint w=class="num">0; w<ff_1; w++) { if(!temp.Add(GenerateWeight())) class="kw">return class="kw">false; } if(!FF_Weights.Add(temp)) class="kw">return class="kw">false; class=class="str">"cmt">//--- temp=new CBufferDouble(); if(CheckPointer(temp)==POINTER_INVALID) class="kw">return class="kw">false; if(!temp.Reserve(ff_2)) class="kw">return class="kw">false; for(class="type">uint w=class="num">0; w<ff_1; w++) { if(!temp.Add(GenerateWeight())) class="kw">return class="kw">false; } if(!FF_Weights.Add(temp)) class="kw">return class="kw">false; class=class="str">"cmt">//--- for(class="type">int d=class="num">0; d<(optimization==SGD ? class="num">1 : class="num">2); d++) { temp=new CBufferDouble(); if(CheckPointer(temp)==POINTER_INVALID) class="kw">return class="kw">false; if(!temp.BufferInit(qkv_weights,class="num">0)) class="kw">return class="kw">false; if(!QKV_Weights.Add(temp)) class="kw">return class="kw">false; temp=new CBufferDouble(); if(CheckPointer(temp)==POINTER_INVALID) class="kw">return class="kw">false; if(!temp.BufferInit(w0,class="num">0)) class="kw">return class="kw">false; if(!FF_Weights.Add(temp)) class="kw">return class="kw">false; class=class="str">"cmt">//--- Initialize FF Weights temp=new CBufferDouble(); if(CheckPointer(temp)==POINTER_INVALID) class="kw">return class="kw">false; if(!temp.BufferInit(ff_1,class="num">0)) class="kw">return class="kw">false; if(!FF_Weights.Add(temp)) class="kw">return class="kw">false; temp=new CBufferDouble(); if(CheckPointer(temp)==POINTER_INVALID) class="kw">return class="kw">false; if(!temp.BufferInit(ff_2,class="num">0))