[{"data":1,"prerenderedAt":63},["ShallowReactive",2],{"article-32":3},{"code":4,"msg":5,"data":6,"count":14},200,"查询成功",{"id":7,"title":8,"keywords":9,"description":10,"category_id":11,"content":12,"body_html":13,"thumb_up":14,"clicks":15,"sort":14,"remark":16,"status":17,"is_open":17,"is_deleted":14,"is_top":14,"is_recommend":14,"create_time":18,"update_time":19,"image_id":20,"url":13,"member_id":14,"cate_name":21,"prev":22,"next":25,"tags":28,"words":34,"read_time":35,"comments":14,"cover":36,"relevant":37},32,"Python爬虫的基本思路","python,爬虫","人生苦短，我用Python；本文章主要整理一下Python爬虫程序基本逻辑思路。",5,"人生苦短，我用Python；本文章主要整理一下Python爬虫程序基本逻辑思路。以一个简单的爬虫程序为例\n\n1、确定目标网站\n\n```python\nhttps:\u002F\u002Fbh.sb\u002Fpost\u002Fcategory\u002Fmain\u002F\u003C\u002Fcode>\u003C\u002Fpre>\n```\n2、分析被爬去网站数据来源结构，即分析数据请求url格式\n\n```python\nurl = 'https:\u002F\u002Fbh.sb\u002Fpost\u002Fcategory\u002Fmain\u002Fpage\u002F2\u002F'\u003C\u002Fcode>\u003C\u002Fpre>\n```\n3、编写程序代码\n\n3.1、程序入口代码\n\n```python\nif __name__ == \"__main__\":\n    main()\u003C\u002Fcode>\u003C\u002Fpre>\n```\n3.2、模拟浏览器请求数据\n\n利用requests模拟浏览器\n\n```python\ndef get_html(url):\n    #添加headers头\n    headers = {\n        'User-Agent':'Mozilla\u002F5.0 (Windows NT 6.1; WOW64; rv:64.0) Gecko\u002F20100101 Firefox\u002F64.0'\n    }\n    response = requests.get(url, headers=headers);\n    response.encoding = 'utf-8'\n    if response.status_code == 200:\n        return response.text\n    return None\u003C\u002Fcode>\u003C\u002Fpre>\n```\n其中添加headers用来模拟火狐浏览器，参数User-Agent也可以是谷歌浏览器或者其他浏览器\n\n3.3、分析数据，通过解析方式，得到有用的数据\n\n```python\n#2、解析内容\ndef parse_html(html, csv):\n    soup = BeautifulSoup(html, 'lxml')\n    data = soup.find_all('article')\n    \n    for item in data:\n        #print(item.a.text)\n        title = item.a.get_text()\n        author = item.select('.text-muted.time')[0].get_text().split(' ')[0]\n        img_href = item.select('img')[0].attrs['src']\n        note = item.select('.note')[0].get_text()\n\n        li = [title, author,  note, img_href]\u003C\u002Fcode>\u003C\u002Fpre>\n```\n解析数据使用bs4,xPath等，这里使用bs4\n\n从数据中解析出title，author，img_href，note等信息，存在列表li中\n\n3.4、保存数据\n\n可以保存为文件或者保存到数据库中，该例子中保存到csv中\n\n```python\n#3、保存数据\ndef init_csv():\n    #歌单csv文件\n    csv_file = open('1.csv', 'w', newline='', encoding='utf-8-sig')\n    writer = csv.writer(csv_file)\n    writer.writerow(['标题', '作者',  '简介', '图片链接'])\n    return writer\u003C\u002Fcode>\u003C\u002Fpre>\n```\n最后一步进行数据分析以及可视化等。本文不做介绍\n",null,0,490,"",1,"2019-01-25 23:22:31","2026-04-19 15:15:42",135,"Python",{"id":23,"title":24},31,"整理Yii2中定义的常用路径别名",{"id":26,"title":27},33,"多线程爬去英雄联盟所有英雄的皮肤",[29,31],{"id":30,"name":21},27,{"id":32,"name":33},51,"爬虫",1009,2,"https:\u002F\u002Ftp.myong.top\u002Fstorage\u002Farticle\u002Fc5\u002F22a0e715e916985d7738d131d3b31b.jpg",[38,43,48,53,58],{"id":39,"title":40,"create_time":41,"description":42},95,"Django启用memcache缓存","2020-10-23 11:17:27","在动态网站中,用户所有的请求,服务器都会去数据库中进行相应的增,删,查,改,渲染模板,执行业务逻辑,最后生成用户看到的页面.当一个网站的用户访问量很大的时候,每一次的的后台操作,都会消耗很多的服务端资源,所以必须使用缓存来减轻后端服务器的压力.",{"id":44,"title":45,"create_time":46,"description":47},91,"用Python实现编程语言 20 年的动态排行榜","2020-06-24 16:06:54","爬取一下，自2001年5月至今，TIOBE 编程语言排行榜上编程语言的变化情况，看一下在接近20年的时间里，编程语言的热度是如何变化的。\r\n",{"id":49,"title":50,"create_time":51,"description":52},100,"re正则表达式","2021-07-10 18:09:52","正则表达式是一个特殊的字符序列，它能帮助你方便的检查一个字符串是否与某种模式匹配，本文罗列re正则表达式中，常用的匹配符号。",{"id":54,"title":55,"create_time":56,"description":57},34,"Python基础整理之列表常见操作","2019-02-10 16:53:55","列表是最常用的Python数据类型，它可以作为一个方括号内的逗号分隔值出现。列表的数据项不需要具有相同的类型",{"id":59,"title":60,"create_time":61,"description":62},63,"Python与mysqldump的数据库备份","2019-10-14 15:35:33","mysqldump是备份MySQL数据库的一种好工具。它相对于用phpmyadmin等备份工具更加快速，又避免受php.ini等的限制，在windows系统下还可以结合计划任务实现定时远程备份数据库",1783431668434]