[{"data":1,"prerenderedAt":68},["ShallowReactive",2],{"article-82":3},{"code":4,"msg":5,"data":6,"count":14},200,"查询成功",{"id":7,"title":8,"keywords":9,"description":10,"category_id":11,"content":12,"body_html":13,"thumb_up":14,"clicks":15,"sort":14,"remark":16,"status":17,"is_open":17,"is_deleted":14,"is_top":14,"is_recommend":14,"create_time":18,"update_time":19,"image_id":20,"url":13,"member_id":14,"cate_name":21,"prev":22,"next":25,"tags":28,"words":40,"read_time":17,"comments":14,"cover":41,"relevant":42},82,"Pyhon爬虫Scrapy设置IP代理","Pyhon,Scrapy","在非设置代理的环境下，持续访问服务器，会被服务器判断为是一个爬虫，那么服务器维护人员就会禁止我们的IP访问，导致我们爬虫失效，代理IP就是为了防止该问题，如下整理的Scrapy爬虫框架中如何配置IP代理。\n",5,"在非设置代理的环境下，持续访问服务器，会被服务器判断为是一个爬虫，那么服务器维护人员就会禁止我们的IP访问，导致我们爬虫失效，代理IP就是为了防止该问题，如下整理的Scrapy爬虫框架中如何配置IP代理。\n\n**1、配置IP地址代理池**\n\n在`setting`配置文件中新增IP代理池\n\n```python\nIP_PROXY_POOL=[ \n    {\"addr\":\"web-proxy.tencent.com:8080\"}, \n    {\"addr\":\"219.228.126.86:8123\"}, \n    {\"addr\":\"61.152.81.193:9100\"}, \n]\n```\n\n**2、在中间件文件新增代理中间件**\n\n在``middlewares`.py`中新增`HttpProxyMiddleware`\n\n```python\nfrom study.settings import IP_PROXY_POOL\nimport random\nclass HttpProxyMiddleware(object):\n    def process_request(self, request, spider):\n        cur_ip=random.choice(IP_PROXY_POOL)\n        print(\"this is ip:\"+cur_ip[\"addr\"])\n        request.meta[\"proxy\"]=\"http:\u002F\u002F\"+cur_ip[\"addr\"]\n```\n\n**3、启用中间件**\n\n修改`setting`中的`DOWNLOADER_MIDDLEWARES `\n\n```py\nDOWNLOADER_MIDDLEWARES = {\n    'study.middlewares.HttpProxyMiddleware':543\n}\n```\n\n**4、使用效果**\n\n输出的log中可以看到效果\n\n```python\n2019-10-23 11:14:46 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023\nthis is ip:web-proxy.tencent.com:8080\n2019-10-23 11:14:47 [scrapy.core.engine] DEBUG: Crawled (404) \u003CGET http:\u002F\u002Fquotes.toscrape.com\u002Frobots.txt> (referer: None)\nthis is ip:web-proxy.tencent.com:8080\n```",null,0,512,"",1,"2019-10-23 11:39:42","2026-04-19 15:10:59",130,"Python",{"id":23,"title":24},81,"Win10中安装配置MongoDB",{"id":26,"title":27},83,"Yii2将当前请求执行的sql记录到日志中",[29,31,34,37],{"id":30,"name":21},27,{"id":32,"name":33},73,"Scrapy",{"id":35,"name":36},51,"爬虫",{"id":38,"name":39},74,"代理",796,"https:\u002F\u002Ftp.myong.top\u002Fstorage\u002Farticle\u002F83\u002F255da489c04d6be60af683b87dd2cd.jpg",[43,48,53,58,63],{"id":44,"title":45,"create_time":46,"description":47},95,"Django启用memcache缓存","2020-10-23 11:17:27","在动态网站中,用户所有的请求,服务器都会去数据库中进行相应的增,删,查,改,渲染模板,执行业务逻辑,最后生成用户看到的页面.当一个网站的用户访问量很大的时候,每一次的的后台操作,都会消耗很多的服务端资源,所以必须使用缓存来减轻后端服务器的压力.",{"id":49,"title":50,"create_time":51,"description":52},91,"用Python实现编程语言 20 年的动态排行榜","2020-06-24 16:06:54","爬取一下，自2001年5月至今，TIOBE 编程语言排行榜上编程语言的变化情况，看一下在接近20年的时间里，编程语言的热度是如何变化的。\r\n",{"id":54,"title":55,"create_time":56,"description":57},100,"re正则表达式","2021-07-10 18:09:52","正则表达式是一个特殊的字符序列，它能帮助你方便的检查一个字符串是否与某种模式匹配，本文罗列re正则表达式中，常用的匹配符号。",{"id":59,"title":60,"create_time":61,"description":62},34,"Python基础整理之列表常见操作","2019-02-10 16:53:55","列表是最常用的Python数据类型，它可以作为一个方括号内的逗号分隔值出现。列表的数据项不需要具有相同的类型",{"id":64,"title":65,"create_time":66,"description":67},63,"Python与mysqldump的数据库备份","2019-10-14 15:35:33","mysqldump是备份MySQL数据库的一种好工具。它相对于用phpmyadmin等备份工具更加快速，又避免受php.ini等的限制，在windows系统下还可以结合计划任务实现定时远程备份数据库",1783431659057]