[{"data":1,"prerenderedAt":61},["ShallowReactive",2],{"article-35":3},{"code":4,"msg":5,"data":6,"count":16},200,"查询成功",{"id":7,"title":8,"keywords":9,"description":10,"category_id":11,"content":12,"body_html":13,"thumb_up":14,"clicks":15,"sort":16,"remark":17,"status":14,"is_open":14,"is_deleted":16,"is_top":16,"is_recommend":16,"create_time":18,"update_time":19,"image_id":20,"url":13,"member_id":16,"cate_name":21,"prev":22,"next":25,"tags":28,"words":34,"read_time":35,"comments":16,"cover":36,"relevant":37},35,"xpath爬去boss直聘职位信息","python,xpath,爬虫","每年的年初都是招聘的旺季，想跳槽的小伙伴是否想了解一下相关行业使用的技术有哪些，哪些知识要学习的。该文件只要是爬去boss直聘的相关职位信息，程序运行时只需要输入查询的城市，职位，爬去页数就可以把数据保存到json文件中，用于分析。",5,"每年的年初都是招聘的旺季，想跳槽的小伙伴是否想了解一下相关行业使用的技术有哪些，哪些知识要学习的。\n\n该文件只要是爬去boss直聘的相关职位信息，程序运行时只需要输入查询的城市，职位，爬去页数就可以把数据保存到json\n\n文件中，用于分析。\n\n实现功能\n\n输入城市名，行政区名，职位，爬取的页面，将爬取数据保存为json文件，以便后期数据可视化分析，\n\n该文章不做数据可视化分析。\n\n该案例需要的库\n\n```python\nrequests,re,json,os,lxml\n```\n##### 一、目标网站\n\n```python\nhttps:\u002F\u002Fwww.zhipin.com\n```\n##### 二、分析网站\n\n1、根据城市查询某职位的信息\n\n```python\nhttps:\u002F\u002Fwww.zhipin.com\u002Fc101280600-p100103\n```\n分析可得 101280600 为查询城市的编号\n\n2、根据城市名加行政区查询某职位的信息\n\n```python\nhttps:\u002F\u002Fwww.zhipin.com\u002Fc101280600-p100103\u002Fb_宝安区\n```\n分析可得 b_XX 其中XX为城市行政区\n\n##### 三、查询到城市编号并保存为json文件\n\n1、分析所有的请求 找到城市编号的数据来源\n\n分析可得所有城市编号的数据来源链接为\n\n```python\nhttps:\u002F\u002Fwww.zhipin.com\u002Fcommon\u002Fdata\u002Fcity.json\n```\n城市编号爬取主要代码\n\n```python\ndef query_city():\n    '''\n    查询城市编号并保存\n    '''\n    url = 'https:\u002F\u002Fwww.zhipin.com\u002Fcommon\u002Fdata\u002Fcity.json'\n    data = json.loads(send_request(url))\n    city = {}\n    for item in data['data']['cityList']:\n        for it in item['subLevelModelList']:\n            city.update({it['name'] : it['code']})\n    with open('.\u002Fboss_city.json', 'w', encoding='utf-8') as f:\n        try:\n            json_str = json.dumps(city, indent=4, ensure_ascii=False)\n            f.write(json_str)\n        except Exception as e:\n            print('-----')\n            print(e)\n            pass\n```\n##### 四、程序编码\n\n1、加载程序用到的库以及全局变量定义\n\n```python\n# here put the import lib\nimport requests,re,json\nimport os\nfrom lxml import etree #使用xpath语法解析\nbase_url = \"https:\u002F\u002Fwww.zhipin.com\"\nheaders = {\n    \"User-Agent\": \"Mozilla\u002F5.0 (Windows NT 10.0; Win64; x64) AppleWebKit\u002F537.36 (KHTML, like Gecko) Chrome\u002F54.0.2840.99 Safari\u002F537.36\"\n}\n# 正则表达式：去掉标签中的\n 和 标签，便于使用xpath解析提取文本\nregx_obj = re.compile(r'\n|\u003C(em).*?>.*?\u003C!--\\1-->')\ncity_code = None #城市编号\ncity_region = None #城市行政区名\n```\n\n2、入口程序\n\n```python\ndef main():\n    #全局变量修改需要global\n    global city_code\n    global city_region\n\n    city_list = get_city_json()\n    city_name = input('请输入需要爬取的城市:')\n    city_code = city_list[city_name]\n \n    city_region = input('请输入需要爬取的城市行政区:')\n    work = input('请输入需要爬取的职业信息:')\n    pages = int(input('请输入需要爬取的页面数:'))\n\n    for page in range(1, pages + 1):\n        start_work(page, work)\n    \nif __name__ == \"__main__\":\n    main()\n```\n\n3、获取每页数据中详细页的链接\n\n```python\ndef detail_url(params):\n    '''\n    获取详细页的页面链接\n    '''\n\n    if city_region is None:\n        url_str = \"c%s-p100103\u002F?ka=sel-city-%s\"%(city_code,city_code)\n    else:\n       url_str = \"c%s-p100103\u002Fb_%s\"%(city_code,city_region) \n    \n    job_url = '\u002F'.join([base_url, url_str])\n\n    html = send_request(job_url, param=params)\n    # 列表页页面\n    html_obj = etree.HTML(html)\n    \n    # 提取详情页url地址\n    nodes = html_obj.xpath(\".\u002F\u002Fdiv[@class='info-primary']\u002F\u002Fa\u002F@href\")\n\n    for node in nodes:\n        url = '\u002F'.join([base_url, node])# 拼接成完整的url地址\n        print(url)\n        html = send_request(url)\n        html_obj = etree.HTML(html) \n        parse_data(html_obj)\n```\n4、xpath解析爬取的详细页数据\n\n```python\ndef parse_data(html_obj):\n    '''\n    xpath 解析相关的数据\n    '''\n    # 解析为HTML文档\n    item = {}\n    # 职位名\n    item['position'] = html_obj.xpath(\"\u002F\u002Fdiv[@class='job-primary detail-box']\u002Fdiv[@class='info-primary']\u002Fdiv[@class='name']\u002Fh1\u002Ftext()\")[0]\n        \n    # 发布者姓名\n    item['publisherName'] = html_obj.xpath(\"\u002F\u002Fdiv[@class='job-detail']\u002F\u002Fh2\u002Ftext()\")[0]\n        \n    # 发布者职位\n    item['publisherPosition'] = html_obj.xpath(\"\u002F\u002Fdiv[@class='detail-op']\u002F\u002Fp\u002Ftext()\")[0]\n        \n    # 薪水\n    item['salary'] = html_obj.xpath(\".\u002F\u002Fdiv[@class='info-primary']\u002F\u002Fspan[@class='salary']\u002Ftext()\")[0].strip()\n        \n    # 工作职责\n    item['responsibility'] = html_obj.xpath(\"\u002F\u002Fdiv[@class='job-sec']\u002F\u002Fdiv[@class='text']\u002Ftext()\")[0].strip()\n        \n    # 招聘要求\n    item['requirement'] = html_obj.xpath(\"\u002F\u002Fdiv[@class='job-primary detail-box']\u002Fdiv[@class='info-primary']\u002Fp\u002Ftext()\")[0]\n    \n    # 招聘企业\n    item['company'] = html_obj.xpath(\"\u002F\u002Fdiv[@class='sider-company']\u002Fdiv[@class='company-info']\u002Fa\u002F@title\")[0].strip()\n    \n    write_data(item) \n```\n5、保存解析的数据\n\n```python\ndef write_data(param):\n    '''\n    将数据写入文件中\n    '''\n    with open('.\u002Fjob.json', 'a+', encoding='utf-8') as f:\n        try:\n            job_json = json.dumps(param, indent=4, ensure_ascii=False)\n            f.write(job_json + ',')\n        except Exception as e:\n            print('-----')\n            print(e)\n            pass\n```\n \n",null,1,483,0,"","2019-02-15 18:15:43","2026-04-19 15:16:04",135,"Python",{"id":23,"title":24},34,"Python基础整理之列表常见操作",{"id":26,"title":27},36,"Python分析一波微信好友",[29,31],{"id":30,"name":21},27,{"id":32,"name":33},51,"爬虫",2490,6,"https:\u002F\u002Ftp.myong.top\u002Fstorage\u002Farticle\u002Fc5\u002F22a0e715e916985d7738d131d3b31b.jpg",[38,43,48,53,56],{"id":39,"title":40,"create_time":41,"description":42},95,"Django启用memcache缓存","2020-10-23 11:17:27","在动态网站中,用户所有的请求,服务器都会去数据库中进行相应的增,删,查,改,渲染模板,执行业务逻辑,最后生成用户看到的页面.当一个网站的用户访问量很大的时候,每一次的的后台操作,都会消耗很多的服务端资源,所以必须使用缓存来减轻后端服务器的压力.",{"id":44,"title":45,"create_time":46,"description":47},91,"用Python实现编程语言 20 年的动态排行榜","2020-06-24 16:06:54","爬取一下，自2001年5月至今，TIOBE 编程语言排行榜上编程语言的变化情况，看一下在接近20年的时间里，编程语言的热度是如何变化的。\r\n",{"id":49,"title":50,"create_time":51,"description":52},100,"re正则表达式","2021-07-10 18:09:52","正则表达式是一个特殊的字符序列，它能帮助你方便的检查一个字符串是否与某种模式匹配，本文罗列re正则表达式中，常用的匹配符号。",{"id":23,"title":24,"create_time":54,"description":55},"2019-02-10 16:53:55","列表是最常用的Python数据类型，它可以作为一个方括号内的逗号分隔值出现。列表的数据项不需要具有相同的类型",{"id":57,"title":58,"create_time":59,"description":60},63,"Python与mysqldump的数据库备份","2019-10-14 15:35:33","mysqldump是备份MySQL数据库的一种好工具。它相对于用phpmyadmin等备份工具更加快速，又避免受php.ini等的限制，在windows系统下还可以结合计划任务实现定时远程备份数据库",1783431668028]