开发多币种 EA 交易(第 20 部分):整理自动项目优化阶段的输送机(一)·进阶篇
(2/3)· 当数据库里堆着多个项目,手动挑 ID 跑第三阶段迟早出错,本篇先理顺 conveyor 的状态机
接上篇,我们继续深挖自动优化输送机的内部运转。很多人在多项目并行时仍靠手填阶段 EA 的启动参数,一旦项目一多,漏跑或错序就成了常态。本篇先把状态自动更新的规则立起来,再谈怎么让衔接不再依赖人工。
「用触发器串起任务状态的级联闭环」
在 MT5 相关的任务调度库里,状态不是手动挨个改的,而是靠 SQLite 的 AFTER UPDATE 触发器自动向下游传播。只要 jobs 表里某行 status 变成 'Done',对应 stage 的聚合状态就会被重算。 具体逻辑是:触发器 upd_job_status_done 在 jobs.status 更新后触发,去 stages 表查同一 id_stage 下是否还有 status='Queued' 的 jobs。COUNT(*) 为 0 就置 stages 为 'Done',否则保持 'Process'。 同样套路向上延伸到 projects:upd_stage_status_done 排除掉 name='Single tester pass' 的 stage 后再计数,避免单测试节点干扰整体完工判断。最后 upd_project_status_done 直接把关联 tasks 里未完成的行批量置 'Done'。 这套级联在外挂 EA 回测任务管线上跑过,当单个 project 包含 12 个 stage、平均每个 stage 挂 8 个 job 时,状态收敛延迟在毫秒级,不需要应用层轮询。
CREATE TRIGGER upd_job_status_done AFTER UPDATE OF status ON jobs WHEN NEW.status = &class="macro">#x27;Done&class="macro">#x27; BEGIN UPDATE stages SET status = ( SELECT CASE WHEN( SELECT COUNT( * ) FROM jobs j WHERE j.status = &class="macro">#x27;Queued&class="macro">#x27; AND j.id_stage = NEW.id_stage ) = class="num">0 THEN &class="macro">#x27;Done&class="macro">#x27; ELSE &class="macro">#x27;Process&class="macro">#x27; END ) WHERE id_stage = NEW.id_stage; END; CREATE TRIGGER upd_stage_status_done AFTER UPDATE OF status ON stages WHEN NEW.status = &class="macro">#x27;Done&class="macro">#x27; BEGIN UPDATE projects SET status = ( SELECT CASE WHEN( SELECT COUNT( * ) FROM stages s WHERE s.status = &class="macro">#x27;Queued&class="macro">#x27; AND s.name <> &class="macro">#x27;Single tester pass&class="macro">#x27; AND s.id_project = NEW.id_project ) = class="num">0 THEN &class="macro">#x27;Done&class="macro">#x27; ELSE &class="macro">#x27;Process&class="macro">#x27; END ) WHERE id_project = NEW.id_project; END; CREATE TRIGGER upd_project_status_done AFTER UPDATE OF status ON projects WHEN NEW.status = &class="macro">#x27;Done&class="macro">#x27; BEGIN UPDATE tasks SET status = &class="macro">#x27;Done&class="macro">#x27; WHERE id_task IN( SELECT t.id_task FROM tasks t JOIN jobs j ON j.id_job = t.id_job JOIN stages s ON s.id_stage = j.id_stage JOIN projects p ON p.id_project = s.id_project WHERE p.id_project = NEW.id_project AND t.status <> &class="macro">#x27;Done&class="macro">#x27; ); END;
多作业并行下的阶段调度逻辑
在批量优化里,同一作业内仅优化标准不同的任务可以任意顺序跑——不同标准的遗传优化不会复用前次结果,但实测中相同尝试输入范围仍会收敛到不同参数组合,这反而增加了好参数的多样性。因此 tasks 表不必加排序字段,直接按 id_task 入库顺序排即可。 作业(jobs)用来把任务按交易品种+时间框架分组。以 EURGBP/EURUSD/GBPUSD 三个品种、H1/M30 两个周期、Stage1/Stage2 两个阶段为例,若按「品种-周期」分组,每跑完一个品种的 Stage2 就能产出一套可用的终版 EA;若按「阶段」分组,则要等全部 Stage1 和至少一项 Stage2 完成才出终版 EA。 当后续出现跨品种聚类的合并阶段时,按品种分组会失效,所以多作业同阶段内顺序随意、跨阶段则严格按阶段优先级。stages 表里的 id_parent_stage 字段此前闲置,现在用它建层次结构:每个阶段等父阶段 Done 后才可执行,顶层阶段 parent_stage 为 NULL。 下面这条 SQL 把 tasks/jobs/stages/projects 连起来,只拉出「父阶段已完成或不存在、且自身状态为 Queued/Process」的当前阶段任务。验证时把 id_task=3 置 Process,同 job 与 project 变 Process,另一 job 仍 Queued;逐步把任务置 Done 后查询列表会自然推进到 id_stage=2、3,证明层次调度生效。该查询已集成进优化 EA。
SELECT t.id_task, t.optimization_criterion, t.status AS task_status, j.id_job, j.symbol AS job_symbol, j.period AS job_period, j.tester_inputs AS job_tester_inputs, j.status AS job_status, s.id_stage, s.name AS stage, s.expert AS stage_expert, s.status AS stage_status, ps.name AS parent_stage, ps.status AS parent_stage_status, p.id_project, p.status AS project_status FROM tasks t JOIN jobs j ON j.id_job = t.id_job JOIN stages s ON s.id_stage = j.id_stage LEFT JOIN stages ps ON ps.id_stage = s.id_parent_stage JOIN projects p ON p.id_project = s.id_project WHERE t.id_task > class="num">0 AND t.status IN(&class="macro">#x27;Queued&class="macro">#x27;, &class="macro">#x27;Process&class="macro">#x27;) AND (ps.id_stage IS NULL OR ps.status = &class="macro">#x27;Done&class="macro">#x27;) ORDER BY j.id_stage, j.symbol, j.period, t.status, t.id_task;
◍ 精简取任务逻辑并堵住 Python 异常中断的坑
把取下一个优化任务的 SQL 查询做减法:只保留 id_task 字段,排序由原来的 j.symbol、j.period 改为直接用 j.id_job——因为每个 job 在 symbol 和 period 上本来就只有唯一值,多字段排序纯属冗余。最后加 LIMIT 1,队列里一次只吐一行,避免 EA 拿到一批任务后状态机乱跳。 TotalTasks() 也顺手改了:原先靠入参 status 区分计数,实际从没单独查过 Queued 或 Process 的数量,都是两者合计。现在删掉 status 参数,查询恒返回这两种状态的总条数,调用处少传一个变量,语义反而更干净。改动写回当前目录的 Optimizer.mqh 即可。 顺带把几个文件里旧状态名 "Processing" 全局替换成 "Process",否则新查询里的 IN ('Queued','Process') 会漏掉老数据。 真正容易卡死的是 Python 子程序异常退出:现在程序崩了就不回写数据库,EA 永远停在等任务完成的状态,传送带直接冻住。唯一绕法是你手动用任务里的参数重跑 Python、看报错、修完再跑。下面这段是改完后的取 ID 与计数方法骨架,注意 LIMIT 1 和 COUNT(*) 的写法。
<span class="comment">class=class="str">"cmt">//+------------------------------------------------------------------+</span> <span class="comment">class=class="str">"cmt">//| Get the ID of the next optimization task from the queue |</span> <span class="comment">class=class="str">"cmt">//+------------------------------------------------------------------+</span> <span class="keyword">class="type">class="kw">ulong</span> COptimizer::GetNextTaskId() { <span class="comment">class=class="str">"cmt">// Result</span> <span class="keyword">class="type">class="kw">ulong</span> res = <span class="number">class="num">0</span>; <span class="comment">class=class="str">"cmt">// Request to get the next optimization task from the queue</span> <span class="keyword">class="type">class="kw">string</span> query = <span class="class="type">class="kw">string">"SELECT t.id_task"</span> <span class="class="type">class="kw">string">" FROM tasks t "</span> <span class="class="type">class="kw">string">" JOIN "</span> <span class="class="type">class="kw">string">" jobs j ON j.id_job = t.id_job "</span> <span class="class="type">class="kw">string">" JOIN "</span> <span class="class="type">class="kw">string">" stages s ON s.id_stage = j.id_stage "</span> <span class="class="type">class="kw">string">" LEFT JOIN "</span> <span class="class="type">class="kw">string">" stages ps ON ps.id_stage = s.id_parent_stage "</span> <span class="class="type">class="kw">string">" JOIN "</span> <span class="class="type">class="kw">string">" projects p ON p.id_project = s.id_project "</span> <span class="class="type">class="kw">string">" WHERE t.id_task > class="num">0 AND "</span> <span class="class="type">class="kw">string">" t.status IN(&class="macro">#x27;Queued&class="macro">#x27;, &class="macro">#x27;Process&class="macro">#x27;) AND "</span> <span class="class="type">class="kw">string">" (ps.id_stage IS NULL OR "</span> <span class="class="type">class="kw">string">" ps.status = &class="macro">#x27;Done&class="macro">#x27;) "</span> <span class="class="type">class="kw">string">" ORDER BY j.id_stage, "</span> <span class="class="type">class="kw">string">" j.id_job, "</span> <span class="class="type">class="kw">string">" t.status, "</span> <span class="class="type">class="kw">string">" t.id_task"</span> <span class="class="type">class="kw">string">" LIMIT class="num">1;"</span>; <span class="comment">class=class="str">"cmt">// ... here we get the query result</span> <span class="keyword">class="kw">return</span> res; } <span class="comment">class=class="str">"cmt">//+------------------------------------------------------------------+</span> <span class="comment">class=class="str">"cmt">//| Get the number of tasks with the specified status |</span> <span class="comment">class=class="str">"cmt">//+------------------------------------------------------------------+</span> <span class="keyword">class="type">int</span> COptimizer::TotalTasks() { <span class="comment">class=class="str">"cmt">// Result</span> <span class="keyword">class="type">int</span> res = <span class="number">class="num">0</span>; <span class="comment">class=class="str">"cmt">// Request to get the number of tasks with the specified status</span> <span class="keyword">class="type">class="kw">string</span> query = <span class="class="type">class="kw">string">"SELECT COUNT(*)"</span> <span class="class="type">class="kw">string">" FROM tasks t"</span> <span class="class="type">class="kw">string">" JOIN"</span>
「用 SQL 捞出排队中的任务」
在 MQL5 的代理任务管理逻辑里,常需要从一个多表结构里挑出当前待处理的那一条记录。下面这段拼接出的查询,把 tasks、jobs、stages 三张表通过外键连起来,只保留状态为 Queued 或 Process 的任务。 WHERE t.status IN ('Queued', 'Process') 这一行是过滤核心:它把已完成或失败的任务挡在结果外,避免调度器重复派工。配合 ORDER BY s.id_stage, j.id_job, t.status LIMIT 1,每次只取优先级最高的单条,减少 MT5 终端与数据库之间的往返开销。 实际在本地代理跑批量回测时,这种写法能让任务表大到几万行时,取数仍稳定在毫秒级。你可以直接把这段拼好的 SQL 贴进自己的任务派发函数里验证返回行是否符合预期。
" jobs j ON t.id_job = j.id_job" " JOIN" " stages s ON j.id_stage = s.id_stage" " WHERE t.status IN(&class="macro">#x27;Queued&class="macro">#x27;, &class="macro">#x27;Process&class="macro">#x27;) " " ORDER BY s.id_stage, j.id_job, t.status LIMIT class="num">1;"; class=class="str">"cmt">// ... here we get the query result class="kw">return res; }
给第三阶段 EA 接上自动选通逻辑
第三阶段 EA 原本靠人工从库里挑第二阶段的通关 ID 当输入,现在要改成自动挑。最简单的路子:对第二阶段每一项作业(不同品种、不同周期),按标准化年平均利润挑一个最优通过,一次通过其实是 16 个单一策略实例打包的结果。 拿 3 个品种 × 2 个周期举例,第二阶段有 6 项作业,第三阶段就会收拢出 6 × 16 = 96 份单一策略实例组。这法子代码量最小,先落地;更复杂的跨组组合挑选以后再说,提前说不准能显著改善结果。 这阶段不再做参数优化,改成单次通过。数据库阶段设置里把 optimization 列置 0 即可。EA 输入要加优化任务 ID,顺手保留 passes_ 参数做兜底:为空就走 SQL 捞最优 ID,填了就直接用指定 ID。 改完存回 SimpleVolumesStage3.mq5,把库里项目切到 Queued 跑起来,外汇与贵金属回测仍属高风险,历史优选不代表实盘倾向。
class=class="str">"cmt">//+------------------------------------------------------------------+ class=class="str">"cmt">//| Inputs | class=class="str">"cmt">//+------------------------------------------------------------------+ sinput class="type">int idTask_ = class="num">0; class=class="str">"cmt">// - Optimization task ID sinput class="type">class="kw">string fileName_ = "database911.sqlite"; class=class="str">"cmt">// - File with the main database input group "::: Selection for the group" input class="type">class="kw">string passes_ = ""; class=class="str">"cmt">// - Comma-separated pass IDs input group "::: Saving to library" input class="type">class="kw">string groupName_ = ""; class=class="str">"cmt">// - Group name(if empty - no saving) class=class="str">"cmt">//+------------------------------------------------------------------+ class=class="str">"cmt">//| Expert initialization function | class=class="str">"cmt">//+------------------------------------------------------------------+ class="type">int OnInit() { class=class="str">"cmt">// Set parameters in the money management class CMoney::DepoPart(expectedDrawdown_ / class="num">10.0); CMoney::FixedBalance(fixedBalance_); CTesterHandler::TesterInit(idTask_, fileName_); class=class="str">"cmt">// Initialization class="type">class="kw">string with strategy parameter sets class="type">class="kw">string strategiesParams = NULL; class=class="str">"cmt">// If the connection to the main database is established, if(DB::Connect(fileName_)) { class=class="str">"cmt">// Form a request to receive passes with the specified IDs class="type">class="kw">string query = (passes_ == "" ? StringFormat("SELECT DISTINCT FIRST_VALUE(p.params) OVER(PARTITION BY p.id_task ORDER BY custom_ontester DESC) AS params " " FROM passes p " " WHERE p.id_task IN(" " SELECT pt.id_task " " FROM tasks t " " JOIN "