[{"data":1,"prerenderedAt":61},["ShallowReactive",2],{"article-71":3},{"code":4,"msg":5,"data":6,"count":14},200,"查询成功",{"id":7,"title":8,"keywords":9,"description":10,"category_id":11,"content":12,"body_html":13,"thumb_up":14,"clicks":15,"sort":14,"remark":16,"status":17,"is_open":17,"is_deleted":14,"is_top":14,"is_recommend":14,"create_time":18,"update_time":19,"image_id":20,"url":13,"member_id":14,"cate_name":21,"prev":22,"next":25,"tags":28,"words":37,"read_time":38,"comments":14,"cover":39,"relevant":40},71,"Nginx反爬虫策略，禁止某些UA抓取网站","Nginx,反爬虫","目前网络上的爬虫非常多，有对网站收录有益的，比如百度蜘蛛（Baiduspider），也有不但不遵守robots规则对服务器造成压力，还不能为网站带来流量的无用爬虫，为防止网站有可能会被别人爬，通过配置Nginx, 我们可以拦截大部分爬虫 ",12,"目前网络上的爬虫非常多，有对网站收录有益的，比如百度蜘蛛（Baiduspider），也有不但不遵守robots规则对服务器造成压力，还不能为网站带来流量的无用爬虫，为防止网站有可能会被别人爬，通过配置Nginx, 我们可以拦截大部分爬虫 \n\n新增反爬虫策略文件：\n\n```bash\nvim \u002Fusr\u002Fwww\u002Fserver\u002Fnginx\u002Fconf\u002Fanti_spider.conf\n```\n\n文件内容\n\n```bash\n#禁止Scrapy等工具的抓取  \nif ($http_user_agent ~* (Scrapy|Curl|HttpClient)) {  \n     return 403;  \n}  \n \n#禁止指定UA及UA为空的访问  \nif ($http_user_agent ~ \"WinHttp|WebZIP|FetchURL|node-superagent|java\u002F|FeedDemon|Jullo|JikeSpider|Indy Library|Alexa Toolbar|AskTbFXTV|AhrefsBot|CrawlDaddy|Java|Feedly|Apache-HttpAsyncClient|UniversalFeedParser|ApacheBench|Microsoft URL Control|Swiftbot|ZmEu|oBot|jaunty|Python-urllib|lightDeckReports Bot|YYSpider|DigExt|HttpClient|MJ12bot|heritrix|EasouSpider|Ezooms|BOT\u002F0.1|YandexBot|FlightDeckReports|Linguee Bot|^$\" ) {  \n     return 403;               \n}  \n \n#禁止非GET|HEAD|POST方式的抓取  \nif ($request_method !~ ^(GET|HEAD|POST)$) {  \n    return 403;  \n}\n\n#屏蔽单个IP的命令是\n#deny 123.45.6.7\n#封整个段即从123.0.0.1到123.255.255.254的命令\n#deny 123.0.0.0\u002F8\n#封IP段即从123.45.0.1到123.45.255.254的命令\n#deny 124.45.0.0\u002F16\n#封IP段即从123.45.6.1到123.45.6.254的命令是\n#deny 123.45.6.0\u002F24\n \n# 以下IP皆为流氓\n#deny 58.95.66.0\u002F24;\n```\n\n配置使用\n\n在站点的server中引入\n\n```bash\n# 反爬虫\t\ninclude \u002Fusr\u002Fwww\u002Fserver\u002Fnginx\u002Fconf\u002Fanti_spider.conf\n```\n\n最后重启nginx\n\n校验是否有效\n\n模拟YYSpider\n\n```bash\nλ curl -X GET -I -A 'YYSpider' https:\u002F\u002Fwww.myong.top\nHTTP\u002F1.1 200 Connection established\n\nHTTP\u002F2 403\nserver: marco\u002F2.11\ndate: Fri, 20 Mar 2020 08:48:50 GMT\ncontent-type: text\u002Fhtml\ncontent-length: 146\nx-source: C\u002F403\nx-request-id: 3ed800d296a12ebcddc4d61c57500aa2\n```\n\n模拟百度Baiduspider\n\n```bash\nλ curl -X GET -I -A 'BaiduSpider' https:\u002F\u002Fwww.myong.top\nHTTP\u002F1.1 200 Connection established\n\nHTTP\u002F2 200\nserver: marco\u002F2.11\ndate: Fri, 20 Mar 2020 08:49:47 GMT\ncontent-type: text\u002Fhtml\nvary: Accept-Encoding\nx-source: C\u002F200\nlast-modified: Wed, 18 Mar 2020 13:16:50 GMT\netag: \"5e721f42-150ce\"\nx-request-id: e82999a78b7d7ea2e9ff18b6f1f4cc84\n```\n\n爬虫常见的User-Agent\n\n```bash\nFeedDemon             内容采集  \nBOT\u002F0.1 (BOT for JCE) sql注入  \nCrawlDaddy            sql注入  \nJava                  内容采集  \nJullo                 内容采集  \nFeedly                内容采集  \nUniversalFeedParser   内容采集  \nApacheBench           cc攻击器  \nSwiftbot              无用爬虫  \nYandexBot             无用爬虫  \nAhrefsBot             无用爬虫  \nYisouSpider           无用爬虫（已被UC神马搜索收购，此蜘蛛可以放开！）  \njikeSpider            无用爬虫  \nMJ12bot               无用爬虫  \nZmEu phpmyadmin       漏洞扫描  \nWinHttp               采集cc攻击  \nEasouSpider           无用爬虫  \nHttpClient            tcp攻击  \nMicrosoft URL Control 扫描  \nYYSpider              无用爬虫  \njaunty                wordpress爆破扫描器  \noBot                  无用爬虫  \nPython-urllib         内容采集  \nIndy Library          扫描  \nFlightDeckReports Bot 无用爬虫  \nLinguee Bot           无用爬虫\n```",null,0,645,"",1,"2020-03-20 16:59:46","2026-04-19 15:07:44",136,"Nginx",{"id":23,"title":24},70,"Laravel框架DB常见使用",{"id":26,"title":27},72,"LeetCode之整数反转",[29,31,34],{"id":30,"name":21},36,{"id":32,"name":33},35,"服务器",{"id":35,"name":36},65,"反爬虫",1808,4,"https:\u002F\u002Ftp.myong.top\u002Fstorage\u002Farticle\u002F5a\u002Fb205f7e4e0e5c670fba97db6ab7ceb.jpg",[41,46,51,56],{"id":42,"title":43,"create_time":44,"description":45},66,"Nginx解决访问图片显示404，403问题","2020-04-03 09:02:59","###### 问题描述\n\n项目中所有的资源放在`\u002Fwww\u002Fjianshu\u002Fpublic\u002Fstorage`中,希望通过url的方式将其展示出来，如下是配置\n\n```bash\neg: http:\u002F\u002Fdemo.test.com\u002Fstorage\u002Fimg\u002F1.png\n```\n\n```bash\nserver\n{\n    listen 80;\n    listen [::]:80;\n    server_name demo.test.com;\n    index index.php index.html in",{"id":47,"title":48,"create_time":49,"description":50},76,"Nginx配置优化","2019-12-29 14:37:11","\n###### 开启高效传输模式\n\n```bash\nhttp {\n\tinclude       mime.types;\n\t...\n\tdefault_type  application\u002Foctet-stream;\n\t...\n\tsendfile   on;\n\ttcp_nopush on;\n}\n#... 配置省略项\n```\n\n参数说明\n\n```bash\ninclude mime.types; #媒体类型,include 只是一个在当前文件中包含另一个文件内容的指令\ndefault_type appli",{"id":52,"title":53,"create_time":54,"description":55},60,"解决Nginx自动忽略header包含下划线参数方法","2019-07-05 15:14:57","该文章记录一次更换服务器后，web环境使用LNMP，部署项目时出现的问题以及解决方法，之前使用的LAMP环境。",{"id":57,"title":58,"create_time":59,"description":60},139,"Nginx 性能优化实战指南：从基础配置到高并发压测","2026-04-12 04:07:34","Nginx 是全球使用最广泛的 Web 服务器之一，以高性能、低资源占用著称。但默认配置往往保守，没有发挥出 Nginx 的全部潜力。本文系统梳理 2025 年 Nginx 性能优化的核心配置项，从基础参数调优、Gzip 压缩、缓存策略、安全防护到压测验证，配合完整配置示例，帮你将网站性能提升数倍。",1783431661400]