利用 curl 解析 HTML·进阶篇
(2/3)· 常规 WebRequest 拿不到页面数据时,用 curl 封装 DLL 做同步解析的进阶路径
◍ 用 CCurlExec 把网页塞进内存缓冲区
在 MT5 的 EA 或指标工程里做联网抓取,先建一个 CCurlExec 类专门和 libcurl 打交道。头文件里直接 include curl.h 最省事,我习惯丢进 stdafx.h 统一编,避免每个 cpp 重复引。 回调类型用 typedef 起个别名 callback,签名就是 curl 标准的 (void*, size_t, size_t, void*),第四个参数放接收者指针。内存结构 MSTRUCT 只装一个 vector<char> 缓冲加一个 size 计数,文件结构 FSTRUCT 则靠 Windows API 的 GetTempFileNameA 在现场拼临时文件名,省得自己管重名。 WriteMemoryCallback 是默认落盘内存的函数:realsize = size * nmemb 算出本次字节数,把 contents 指针强转成 char* 后 insert 进 vector,同时累加 mem->size。开发者没自定义回调时就走这套,逻辑直白但够用。 真正发请求的是 GetFiletoMem:先 curl_global_init 再 curl_easy_init 拿两个句柄,option 里至少设 URL、关掉证书校验(SSL_VERIFYPEER/VERIFYHOST 都置 0)、绑 ERRORBUFFER 和超时,再把 WriteMemoryCallback 绑到 CURLOPT_WRITEFUNCTION。最后 curl_easy_perform 一跑,HTML 就全在 MSTRUCT.membuff 里了。 把 CURLOPT_VERBOSE 打开能看到每次 DNS、握手、重定向的明细,调不通的时候比瞎猜快得多。外汇和贵金属行情接口常带重定向,这块不打开容易在本地以为抓到了实则 302 空响应,实盘试错成本高、风险大。
class="macro">#include <curl\curl.h> typedef class="type">size_t (*callback)(class="type">void*, class="type">size_t, class="type">size_t, class="type">void*); typedef class="kw">struct MemoryStruct { vector<class="type">char> membuff; class="type">size_t size = class="num">0; } MSTRUCT, *PMSTRUCT; typedef class="kw">struct FileStruct { std::class="type">class="kw">string CalcName() { class="type">char cd[MAX_PATH]; class="type">char fl[MAX_PATH]; srand(unsigned(std::time(class="num">0))); ::GetCurrentDirectoryA(MAX_PATH, cd); ::GetTempFileNameA(cd, "_cUrl", std::rand(), fl); class="kw">return std::class="type">class="kw">string(fl); } std::class="type">class="kw">string filename; FILE* stream = nullptr; } FSTRUCT, *PFSTRUCT; class="type">size_t CCurlExec::WriteMemoryCallback(class="type">void * contents, class="type">size_t size, class="type">size_t nmemb, class="type">void * userp) { class="type">size_t realsize = size * nmemb; PMSTRUCT mem = (PMSTRUCT)userp; vector<class="type">char>tmp; class="type">char* data = (class="type">char*)contents; tmp.insert(tmp.end(), data, data + realsize); if (tmp.size() <= class="num">0) class="kw">return class="num">0; mem->membuff.insert(mem->membuff.end(), tmp.begin(), tmp.end() ); mem->size += realsize; class="kw">return realsize; } class="type">bool CCurlExec::GetFiletoMem(class="kw">const class="type">char* pUri) { CURL *curl; CURLcode res; res = curl_global_init(CURL_GLOBAL_ALL); if (res == CURLE_OK) { curl = curl_easy_init(); if (curl) { curl_easy_setopt(curl, CURLOPT_URL, pUri); curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, 0L); curl_easy_setopt(curl, CURLOPT_SSL_VERIFYHOST, 0L); curl_easy_setopt(curl, CURLOPT_ERRORBUFFER, m_errbuf);
用 libcurl 拉取外部数据的超时与回调设定
在 MT5 里通过 CCurlExec 类做 HTTP 请求时,网络层的健壮性基本靠几个_setopt 撑着。上面这段同时出现在两个成员函数里,说明无论是默认内存写入还是自定义 GetFile 回调,基础参数都保持一致。 超时方面给了两组硬数字:CURLOPT_TIMEOUT 设 20L,代表整体传输最多撑 20 秒;CURLOPT_CONNECTTIMEOUT 设 60L,连接阶段允许磨 60 秒。外汇与贵金属行情接口在流动性稀薄的时段可能连得慢,60 秒连接窗口能避免过早放弃,但 20 秒传输上限意味着大体积历史数据可能截断,调参时得按实际响应体量权衡。 SSL 验证在 GetFile 里被显式关掉:CURLOPT_SSL_VERIFYPEER 与 CURLOPT_SSL_VERIFYHOST 都设 0L。这能绕过自签证书问题,但明文传输通道被中间人嗅探的概率上升,接入第三方信号源时有数据被篡改的可能,属高风险操作。 写入逻辑分两条路:默认用 WriteMemoryCallback 配 &m_mchunk 把响应塞进内存块;GetFile 则把外部传进来的 pFunc 和 pTarget 挂上,方便直接落盘或解析。DEBUG 宏下会开 VERBOSE 打印握手细节,实盘编译记得关掉以免日志爆仓。
curl_easy_setopt(curl, CURLOPT_TIMEOUT, 20L); curl_easy_setopt(curl, CURLOPT_CONNECTTIMEOUT, 60L); curl_easy_setopt(curl, CURLOPT_USERAGENT, "libcurl-agent/class="num">1.0"); curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1L); class=class="str">"cmt">//redirects class="macro">#ifdef __DEBUG__ curl_easy_setopt(curl, CURLOPT_VERBOSE, 1L); class="macro">#endif curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, WriteMemoryCallback); curl_easy_setopt(curl, CURLOPT_WRITEDATA, &m_mchunk); res = curl_easy_perform(curl); if (res != CURLE_OK) PrintCurlErr(m_errbuf, res); curl_easy_cleanup(curl); }class=class="str">"cmt">// if (curl) curl_global_cleanup(); } else PrintCurlErr(m_errbuf, res); class="kw">return (res == CURLE_OK)? true: class="kw">false; } class="type">bool CCurlExec::GetFile(class="kw">const class="type">char * pUri, callback pFunc, class="type">void * pTarget) { CURL *curl; CURLcode res; res = curl_global_init(CURL_GLOBAL_ALL); if (res == CURLE_OK) { curl = curl_easy_init(); if (curl) { curl_easy_setopt(curl, CURLOPT_URL, pUri); curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, 0L); curl_easy_setopt(curl, CURLOPT_SSL_VERIFYHOST, 0L); curl_easy_setopt(curl, CURLOPT_ERRORBUFFER, m_errbuf); curl_easy_setopt(curl, CURLOPT_TIMEOUT, 20L); curl_easy_setopt(curl, CURLOPT_CONNECTTIMEOUT, 60L); curl_easy_setopt(curl, CURLOPT_USERAGENT, "libcurl-agent/class="num">1.0"); curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1L); class="macro">#ifdef __DEBUG__ curl_easy_setopt(curl, CURLOPT_VERBOSE, 1L); class="macro">#endif curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, pFunc); curl_easy_setopt(curl, CURLOPT_WRITEDATA, pTarget);
「收尾清理与执行结果判定」
这段片段处在 HTTP 请求函数的末尾,负责真正发出请求并回收 libcurl 资源。先调用 curl_easy_perform(curl) 执行之前配置好的句柄,若返回码不等于 CURLE_OK 就通过 PrintCurlErr 把 m_errbuf 里的错误文本和 res 码打出来,方便在 MT5 专家日志里定位是哪一步断了。 无论成功失败,只要 curl 句柄存在就必须 curl_easy_cleanup(curl) 释放,否则多次调用会泄漏内存;函数最外层再 curl_global_cleanup() 收掉全局环境。注意 else 分支里直接 PrintCurlErr 是针对 curl_easy_init 或全局初始化失败的情形,此时根本没有 easy 句柄可用。 最终返回用三元表达式:(res == CURLE_OK) ? true : false,把 C 枚举码转成布尔交给调用方。实盘里若该函数返回 false,说明这一轮向外网服务(例如小布盯盘的后端)的推送没达成,外汇与贵金属行情瞬息,这类失败需有重试或本地缓存兜底,相关品种波动风险依旧偏高。
res = curl_easy_perform(curl); if (res != CURLE_OK) PrintCurlErr(m_errbuf, res); curl_easy_cleanup(curl); }class=class="str">"cmt">// if (curl) curl_global_cleanup(); } else PrintCurlErr(m_errbuf, res); class="kw">return (res == CURLE_OK) ? true : class="kw">false; }
◍ 从 URI 抠出落地文件名
CCurlExec 类里负责把网络地址拆出本地落盘名的逻辑,核心在 ParseUri 这一层。它先用 curl_url() 建句柄,再向 CURLUPART_PATH 要路径片段,依据路径末尾是不是 '/' 决定文件名:结尾带斜杠就默认补 index.html,否则把路径按分隔符拆开取最后一段当文件名。 代码里有个硬门槛值得注意:编译期就卡 curl 7.62.0 以上版本,低于这个号直接 #error 退出,说明老版本 URL 解析 API 不在考虑范围。实际跑的时候如果 pUri 传空指针,函数立刻返回空串,不会往下碰句柄。 除了落盘,类还留了两种内存缓冲访问方式——std::vector<char> 和 std::string,具体用哪个看后面 html 解析器怎么选。当前项目没用到的属性和方法就不必深究。 curl 底层是线程安全的,但本例全走 curl_easy_init 这族同步接口;名字带 easy 的都只能同步,异步要走 curl_multi_init 那套。异步调用不复杂但代码冗长,这里先不碰,后面可能再回头说。
std::<span class="keyword">class="type">class="kw">string</span> CCurlExec::ParseUri(<span class="keyword">class="kw">const</span> <span class="keyword">class="type">char</span>* pUri) { <span class="preprocessor">class="macro">#if </span>!CURL_AT_LEAST_VERSION(<span class="number">class="num">7</span>, <span class="number">class="num">62</span>, <span class="number">class="num">0</span>) <span class="preprocessor">class="macro">#error </span><span class="class="type">class="kw">string">"this example requires curl class="num">7.62.class="num">0 or later"</span> <span class="keyword">class="kw">return</span> std::<span class="keyword">class="type">class="kw">string</span>(); <span class="preprocessor">class="macro">#endif </span>CURLU *h = curl_url(); <span class="keyword">if</span> (!h) { cerr << <span class="class="type">class="kw">string">"curl_url(): out of memory"</span> << endl; <span class="keyword">class="kw">return</span> std::<span class="keyword">class="type">class="kw">string</span>(); } std::<span class="keyword">class="type">class="kw">string</span> szres{}; <span class="keyword">if</span> (pUri == nullptr) <span class="keyword">class="kw">return</span> szres; <span class="keyword">class="type">char</span>* path; CURLUcode res; res = curl_url_set(h, CURLUPART_URL, pUri, <span class="number">class="num">0</span>); <span class="keyword">if</span> ( res == CURLE_OK) { res = curl_url_get(h, CURLUPART_PATH, &path, <span class="number">class="num">0</span>); <span class="keyword">if</span> ( res == CURLE_OK) { std::vector <<span class="keyword">class="type">class="kw">string</span>> elem; std::<span class="keyword">class="type">class="kw">string</span> pth = path; <span class="keyword">if</span> (pth[pth.length() - <span class="number">class="num">1</span>] == <span class="class="type">class="kw">string">&class="macro">#x27;/&class="macro">#x27;</span>) { szres = <span class="class="type">class="kw">string">"index.html"</span>; } <span class="keyword">else</span> { Split(pth, elem); cout << elem[elem.size() - <span class="number">class="num">1</span>] << endl; szres = elem[elem.size() - <span class="number">class="num">1</span>]; } curl_free(path); }<span class="comment">class=class="str">"cmt">// if (curl_url_get(h, CURLUPART_PATH, &path, class="num">0) == CURLE_OK)</span> }<span class="comment">class=class="str">"cmt">// if (curl_url_set(h, CURLUPART_URL, pUri, class="num">0) == CURLE_OK)</span> <span class="keyword">class="kw">return</span> szres; } std::vector<<span class="keyword">class="type">char</span>> std::<span class="keyword">class="type">class="kw">string</span>
用 Gumbo 把波动率网页拆成树
做 MT5 外接网页数据,第一步是选 HTML 解析器。C++ 里 Gumbo 是 Google 出的 HTML5 解析器,纯树构建、不附带额外逻辑,已通过 html5lib-0.95 全部测试,底层验证基于超过 25 亿个真实抓取页面,对错的输入数据也会拦掉。代价是性能一般,且只给树、不给提取封装——我们正好按原样用,不套第三方 wrapper。
接 mataf.net 的波动率页,目标数据在 <table id="volTable"> 里,每行 id="row_AUDCHF" 这类属性带货币对名,一行五列,跳过前两列,后三列分别是点值、货币和百分比三种波动率。搜索拆三层:先按 id 找表,再按 row_前缀找行,最后读指定列文本转 double。
网站改版会拖垮整段树搜索代码,这是紧耦合的硬伤;但函数少、逻辑直,真改起来也就动几个递归。下面的核心调用把 HTML 缓冲丢给 gumbo_parse,拿到 root 后递归找表,出错统一返 -1。
别把解析器当万能胶
Gumbo 只建树不提取,逗号做小数点的页面(如该站波动率表)要自己替换后再转 double,否则数值会直接错乱。外汇与贵金属数据抓取涉及第三方站点稳定性,本身属高风险操作,实盘前务必在模拟环境跑通。
class="macro">#include <gumbo\gumbo.h> GumboOutput* output = gumbo_parse(class="kw">input); class=class="str">"cmt">// ... do something with output ... gumbo_destroy_output(&options, output); typedef enum { byPips = class="num">2, byCurr = class="num">3, byPerc = class="num">4 } VOLTYPE; class="type">class="kw">double CVolatility::FindData(class="kw">const std::class="type">class="kw">string& szHtml, class="kw">const std::class="type">class="kw">string& pair, VOLTYPE vtype) { if (pair.empty()) class="kw">return -class="num">1; m_pair = pair; TOUPPER(m_pair); m_column = vtype; GumboOutput* output = gumbo_parse(szHtml.c_str() ); class="type">class="kw">double res = FindTable(output->root); class="kw">const GumboOptions mGumboDefaultOptions = { &malloc_wrapper, &free_wrapper, NULL, class="num">8, class="kw">false, -class="num">1, GUMBO_TAG_LAST, GUMBO_NAMESPACE_HTML }; gumbo_destroy_output(&mGumboDefaultOptions, output); class="kw">return res; }class=class="str">"cmt">// class="type">void CVolatility::FindData(class="type">char * pHtml) class="type">class="kw">double CVolatility::FindTable(GumboNode * node) { class="type">class="kw">double res = -class="num">1; if (node->type != GUMBO_NODE_ELEMENT) { class="kw">return res; } GumboAttribute* ptable; if ( (node->v.element.tag == GUMBO_TAG_TABLE) && \ (ptable = gumbo_get_attribute(&node->v.element.attributes, "id") ) && \ (m_idtable.compare(ptable->value) == class="num">0) ) { GumboVector* children = &node->v.element.children; GumboNode* pchild = nullptr; for (unsigned i = class="num">0; i < children->length; ++i) {
「从解析出的表格行里抠出波动率数值」
前面把网页 DOM 树翻到底,定位到 tbody 后就要在行一级继续匹配。FindTableRow 用 "row_" + m_pair 拼出目标 id,比如当前货币对是 EURUSD 就找 id="row_EURUSD" 的 tr 节点,命中才进 GetVolatility。 GetVolatility 干的事很直接:遍历该行下的 td,用计数器 j 跳过前面的单元格,当 j 自增后等于 m_column 时,那个 td 里的内容就是我们要的波动率。若整行没匹配到目标列,函数返回 -1 表示抓取失败。 实际接行情源时,m_column 对应的往往是「今日波动点数」那一列,从 0 开始数。若你换的网页模板列序不同,改这一个下标就能让整条解析链输出正确数字,外汇与贵金属波动抓取属高风险自动化,数值异常时优先怀疑列偏移而非行情本身。
pchild = class="kw">static_cast<GumboNode*>(children->data[i]); if (pchild && pchild->v.element.tag == GUMBO_TAG_TBODY) { class="kw">return FindTableRow(pchild); } class=class="str">"cmt">//for (class="type">int i = class="num">0; i < children->length; ++i) } else { for (unsigned class="type">int i = class="num">0; i < node->v.element.children.length; ++i) { res = FindTable(class="kw">static_cast<GumboNode*>(node->v.element.children.data[i])); if ( res != -class="num">1) class="kw">return res; }class=class="str">"cmt">// for (unsigned class="type">int i = class="num">0; i < node->v.element.children.length; ++i) } class="kw">return res; }class=class="str">"cmt">//class="type">void CVolatility::FindData(GumboNode * node, class="kw">const std::class="type">class="kw">string & szHtml) class="type">class="kw">double CVolatility::FindTableRow(GumboNode* node) { std::class="type">class="kw">string szRow = "row_" + m_pair; GumboAttribute* prow = nullptr; GumboNode* child_node = nullptr; GumboVector* children = &node->v.element.children; for (unsigned class="type">int i = class="num">0; i < children->length; ++i) { child_node = class="kw">static_cast<GumboNode*>(node->v.element.children.data[i]); if ( (child_node->v.element.tag == GUMBO_TAG_TR) && \ (prow = gumbo_get_attribute(&child_node->v.element.attributes, "id")) && \ (szRow.compare(prow->value) == class="num">0)) { class="kw">return GetVolatility(child_node); } }class=class="str">"cmt">// for (unsigned class="type">int i = class="num">0; i < node->v.element.children.length; ++i) class="kw">return -class="num">1; }class=class="str">"cmt">// class="type">class="kw">double CVolatility::FindVolatility(GumboNode * node) class="type">class="kw">double CVolatility::GetVolatility(GumboNode* node) { class="type">class="kw">double res = -class="num">1; GumboNode* child_node = nullptr; GumboVector* children = &node->v.element.children; class="type">int j = class="num">0; for (unsigned class="type">int i = class="num">0; i < children->length; ++i) { child_node = class="kw">static_cast<GumboNode*>(node->v.element.children.data[i]); if (child_node->v.element.tag == GUMBO_TAG_TD && j++ == (class="type">int)m_column) {