phpcurl使用privoxy代理访问https://www.google.com/search?q=xxx
curl配置平淡无奇,长时间运行发现一个严重问题,内存泄露!不论用单线程和多线程都无法避免!是curl访问https站点的时候有bug!
内存泄露可以通过linux的top命令发现,使用php函数memory_get_usage()不会发现。
经过反复调试找到解决办法,curl配置添加如下几项解决问题:
[CURLOPT_HTTPPROXYTUNNEL] = true; [CURLOPT_SSL_VERIFYPEER] = false; [CURLOPT_SSL_VERIFYHOST] = false;
CURLOPT_HTTPPROXYTUNNEL具体说明stackoverflow上有,直接贴原文:
Without CURLOPT_HTTPPROXYTUNNEL
Without CURLOPT_HTTPPROXYTUNNEL
: You just use the proxy address/port as a destination of your HTTP request. The proxy will read the HTTP headers of your query, forward your request to the destination (with your HTTP headers) and then write the response to you.
Example steps :
HTTP GET /index.html
sent to1.1.1.1 (proxy)
1.1.1.1
receive request and parse header for getting the final destination of your HTTP request.1.1.1.1
forward your query and headers towww.site.com (destination in request headers)
.1.1.1.1
write back to you the response receive fromwww.site.com
With CURLOPT_HTTPPROXYTUNNEL
With CURLOPT_HTTPPROXYTUNNEL
: You ask the proxy to open a direct binary connection (like HTTPS, called a TCP Tunnel
) directly to your destination by doing a CONNECT
HTTP request. When the tunnel is ok, the proxy write you back a HTTP/1.1 200 Connection established
. When it received your browser start to query the destination directly : The proxy does not parse HTTP headers and theoretically does not read tunnel datas, it just forward it, thats why it is called a tunnel !
Example steps :
HTTP CONNECT
sent to1.1.1.1
1.1.1.1
receiveHTTP CONNECT
and get the ip/port of your final destination (header
field ofHTTP CONNECT
).1.1.1.1
open aTCP Socket
by doing aTCP handshake
to your destination2.22.63.73:80 (ip/port of www.site.com)
.1.1.1.1
Make a tunnel by piping yourTCP Socket
to theTCP Socket
opened to2.22.63.73:80
and then write you backHTTP/1.1 200 Connection established
witch means that your client can now make your query throw theTCP Tunnel
(TCP datas received will be transmited directly to server and vice versa).
http://stackoverflow.com/questions/12288956/what-is-the-curl-option-curlopt-httpproxytunnel-means
楼主,我貌似遇到内存泄露了
我这里的表现为apache进程猛增,top命令查看发现几乎全是httpd,把内存全部吃光,ssh几乎登陆不上。。。
能帮我判断下以上特征是否是该BUG导致的呢?
是该BUG导致的可能性非常非常小。这个bug表现为一个单一的php进程内存不断增大(该php进程中使用curl类采集https网站)。
用curl访问java提供的一个服务 间歇性的 连接不上主机,网络应该没有问题
大部分时候都正常。
java服务的问题比较大