[{"data":1,"prerenderedAt":64},["ShallowReactive",2],{"article-91":3},{"code":4,"msg":5,"data":6,"count":14},200,"查询成功",{"id":7,"title":8,"keywords":9,"description":10,"category_id":11,"content":12,"body_html":13,"thumb_up":14,"clicks":15,"sort":14,"remark":16,"status":17,"is_open":17,"is_deleted":14,"is_top":14,"is_recommend":14,"create_time":18,"update_time":19,"image_id":20,"url":13,"member_id":14,"cate_name":21,"prev":22,"next":25,"tags":28,"words":34,"read_time":35,"comments":36,"cover":37,"relevant":38},91,"用Python实现编程语言 20 年的动态排行榜","Python,爬虫,数据分析","爬取一下，自2001年5月至今，TIOBE 编程语言排行榜上编程语言的变化情况，看一下在接近20年的时间里，编程语言的热度是如何变化的。\r\n",5,"爬取一下，自2001年5月至今，TIOBE 编程语言排行榜上编程语言的变化情况，看一下在接近20年的时间里，编程语言的热度是如何变化的。\n###### 1、数据来源\n\n```bash\nhttps:\u002F\u002Fwww.tiobe.com\u002Ftiobe-index\u002F\n```\n\n###### 2、数据提取\n\n将原始数据从目标站点获取到，如下所如,站点部分源码\n\n```bash\nseries: [\n    \t{name : 'C',data : [[Date.UTC(2001, 5, 30), 20.24], [Date.UTC(2001, 6, 30), 20.77], [Date.UTC(2001, 7, 30), 20.75], [Date.UTC(2001, 8, 28), 20.77], [Date.UTC(2001, 9, 26), 19.75], [Date.UTC(2001, 10, 28), 19.21], [Date.UTC(2001, 11, 31), 20.14], [Date.UTC(2002, 0, 30), 18.83], [Date.UTC(2002, 1, 27), 19.89]]}\n    ]\n```\n\n解析数据核心代码\n\n```python\n#发送请求获取数据\nresponse = requests.get(self.url, headers=self.headers, proxies=self.proxies).text\n#获取总的全部数据\ntotal_content = ''.join(re.findall(r'series:(.*?)\\}\\);', response, re.DOTALL))\n#获取每种编程语言的数据\ntotal_content = re.findall(r'({.*?})', total_content, re.DOTALL)\n#print(total_content)\n\nwith open(self.table, 'w', newline='') as f:\n    self.writer = csv.DictWriter(f, ['name', 'value', 'date'])\n    self.writer.writeheader()\n\n    for content in total_content:\n        #获取编程语言的名字\n        name = ''.join(re.findall(r\"{name : '(.*?)'\", content, re.DOTALL))\n        #获取编程语言数据\n        data = re.findall(r\"\\[Date.UTC(.*?)\\]\", content, re.DOTALL)\n        for i in data:\n            i = i.replace(' ', '')\n            #提取时间\n            i = re.sub(r'[()]', '', i)\n            value = i.split(',')[-1]\n            date_list = i.split(',')[:3]\n            time = []\n            for index,j in enumerate(date_list):\n                if index != 0:\n                    if len(j) == 1:\n                        j = '0'+j\n                time.append(j)\n            self.writer.writerow({'name':name, 'value':value, 'date':'-'.join(time)})\n            print(name, value, '-'.join(time))\n```\n\n 对于抓取到的数据，进行正则表达式匹配，然后提取各个编程语言在不同时间段的热度数值，并保存到本地的文件中。 \n\n###### 3、清洗数据\n我们要完成的就是利用动态可视化的柱状图来观察各种编程语言随着时间的热度变化。我们先对数据进行清洗，获取编程语言的名字一起设置一个嵌套的字典，程序如下图所示：\n```python\ndata = pd.read_csv(self.table)\n#获取编程语言的名字\nnames = set(data['name'])\n#print(names)\n#获取编程语言的月份信息\nmonth_list = data[data['name'].isin(['Java'])]['date'].values\n#print(month_list)\n#创建嵌套列表，每个日期下保留每种编程语言的热度值\nper_month_lan_dict = {i:{j:0 for j in names} for i in month_list}\nfor month in month_list:\n    for name in names:\n        try:\n            per_month_lan_dict[month][name] = data[data['name'] == name][data['date'] == month]['value'].values[0]\n        except:\n            continue\nprint(per_month_lan_dict)\nreturn per_month_lan_dict\n```\n上述程序中的嵌套字典含义为每一个月份下的每种编程语言的热度值，其结构格式如下所示：\n```bash\n{“2020-1-12”:{“Java”:16, “C++”:14, “python”:10,…}, “2020-2-13”:{“Java”:16.3, “C++”:15.6, …},…}。\n```\n###### 4、设置图表\n为了在可视化过程中区分每一种编程语言，需要为柱状图中的每一柱都设置不同的颜色，同时，将嵌套字典按照月份的顺序进行排序，程序如下所示：\n```python\ncolors = ['k', 'r', 'sienna', 'yellow', 'g', 'aquamarine', 'dodgerblue', 'pink', 'b', 'darkviolet']\ncolor_dict = {\n    'C#':'k',\n    'Python':'r',\n    'C++':'sienna',\n    'JavaScript':'yellow',\n    'Visual Basic':'g',\n    'R':'aquamarine',\n    'C':'dodgerblue',\n    'Java':'pink',\n    'PHP':'b',\n    'SQL':'darkviolet'\n}\n```\n###### 5、动态显示\n最后，我们便可以对数据进行可视化的展示，程序如下图所示：\n```python\nax = plt.gca()\nfor data_item in data.items():\n    plt.cla()\n    temp = sorted(data_item[1].items(), key=lambda item: item[1])\n    x = [item[0] for item in temp]\n    color = [color_dict[i] for i in x]\n    y = [item[1] for item in temp]\n\t\n    plt.barh(range(1, 11), y, color = color)\n    plt.title(data_item[0], fontproperties='simhei', fontsize=16)\n    plt.yticks(range(1,11), list(x), fontproperties='simhei', fontsize=16)\n    plt.xticks(range(0, 30, 100))\n    for x,y in zip(range(1, 11), y):\n        plt.text(y + 0.1, x - 0.1, str(y))\n    plt.pause(0.1)\nplt.show()\n```\n\n###### 6、完整代码\n\n```python\n#!\u002Fusr\u002Fbin\u002Fenv python\n# -*- encoding: utf-8 -*-\n'''\n@File    :   DynamicChart.py\n@Time    :   2020\u002F06\u002F22 10:11:51\n@Author  :   MrYong \n@Version :   1.0\n@Contact :   m.yong@foxmail.com\n@License :   (C)Copyright 2018-2019, MR YONG\n@Desc    :   做了一个编程语言 20 年的动态排行榜！\n'''\n\n# here put the import lib\nimport requests\nimport pandas as pd\nimport re\nimport csv\nfrom collections import OrderedDict\nimport matplotlib.pyplot as plt\n\nclass Spider():\n    def __init__(self):\n        self.url = 'https:\u002F\u002Fwww.tiobe.com\u002Ftiobe-index\u002F'\n        self.headers = {\n            'User-Agent': 'Mozilla\u002F5.0 (Windows NT 10.0; WOW64) AppleWebKit\u002F537.36 (KHTML, like Gecko) Chrome\u002F63.0.3239.132 Safari\u002F537.36'\n        }\n        self.proxies = {\n            'http':'http:\u002F\u002Fweb-proxy.tencent.com:8080\u002F',\n            'https':'http:\u002F\u002Fweb-proxy.tencent.com:8080\u002F'\n        }\n        self.table = '.\u002Fsp_data.csv'\n    def run(self):\n        #self.getData()\n        data = self.analysisData()\n        self.draw(data)\n\n    def draw(self, data):\n        colors = ['k', 'r', 'sienna', 'yellow', 'g', 'aquamarine', 'dodgerblue', 'pink', 'b', 'darkviolet']\n        color_dict = {\n            'C#':'k',\n            'Python':'r',\n            'C++':'sienna',\n            'JavaScript':'yellow',\n            'Visual Basic':'g',\n            'R':'aquamarine',\n            'C':'dodgerblue',\n            'Java':'pink',\n            'PHP':'b',\n            'SQL':'darkviolet'\n        }\n\n        ax = plt.gca()\n        for data_item in data.items():\n            plt.cla()\n\n            temp = sorted(data_item[1].items(), key=lambda item: item[1])\n            x = [item[0] for item in temp]\n            color = [color_dict[i] for i in x]\n            y = [item[1] for item in temp]\n\n            plt.barh(range(1, 11), y, color = color)\n            plt.title(data_item[0], fontproperties='simhei', fontsize=16)\n            plt.yticks(range(1,11), list(x), fontproperties='simhei', fontsize=16)\n            plt.xticks(range(0, 30, 100))\n            for x,y in zip(range(1, 11), y):\n                plt.text(y + 0.1, x - 0.1, str(y))\n            plt.pause(0.1)\n        plt.show()\n\n    def sort_key(self, old_dict, reverse = False):\n        '''\n        对字典按key排序，并返回OrderedDict\n        '''\n        keys = sorted(old_dict.keys(), reverse=reverse)\n        #创建新的空字典\n        new_dict = OrderedDict()\n        for key in keys:\n            new_dict[key] = old_dict[key]\n        return new_dict\n\n    def analysisData(self):\n        data = pd.read_csv(self.table)\n        #获取编程语言的名字\n        names = set(data['name'])\n        #print(names)\n\n        #获取编程语言的月份信息\n        month_list = data[data['name'].isin(['Java'])]['date'].values\n        #print(month_list)\n\n        #创建嵌套列表，每个日期下保留每种编程语言的热度值\n        per_month_lan_dict = {i:{j:0 for j in names} for i in month_list}\n        for month in month_list:\n            for name in names:\n                try:\n                    per_month_lan_dict[month][name] = data[data['name'] == name][data['date'] == month]['value'].values[0]\n                except:\n                    continue\n        print(per_month_lan_dict)\n        return per_month_lan_dict\n\n    def getData(self):\n        #发送请求获取数据\n        response = requests.get(self.url, headers=self.headers, proxies=self.proxies).text\n        #获取总的全部数据\n        total_content = ''.join(re.findall(r'series:(.*?)\\}\\);', response, re.DOTALL))\n        #获取每种编程语言的数据\n        total_content = re.findall(r'({.*?})', total_content, re.DOTALL)\n        #print(total_content)\n\n        with open(self.table, 'w', newline='') as f:\n            self.writer = csv.DictWriter(f, ['name', 'value', 'date'])\n            self.writer.writeheader()\n\n            for content in total_content:\n                #获取编程语言的名字\n                name = ''.join(re.findall(r\"{name : '(.*?)'\", content, re.DOTALL))\n                #获取编程语言数据\n                data = re.findall(r\"\\[Date.UTC(.*?)\\]\", content, re.DOTALL)\n\n                for i in data:\n                    i = i.replace(' ', '')\n                    #提取时间\n                    i = re.sub(r'[()]', '', i)\n                    value = i.split(',')[-1]\n                    date_list = i.split(',')[:3]\n                    time = []\n                    for index,j in enumerate(date_list):\n                        if index != 0:\n                            if len(j) == 1:\n                                j = '0'+j\n                        time.append(j)\n                    \n                    self.writer.writerow({'name':name, 'value':value, 'date':'-'.join(time)})\n                    print(name, value, '-'.join(time))\n\nif __name__ == \"__main__\":\n    sp = Spider()\n    sp.run()\n```\n\n",null,0,837,"",1,"2020-06-24 16:06:54","2026-04-19 10:45:44",138,"Python",{"id":23,"title":24},90,"Redis常见使用场景",{"id":26,"title":27},92,"PHP对图片的处理",[29,31],{"id":30,"name":21},27,{"id":32,"name":33},51,"爬虫",4325,10,2,"https:\u002F\u002Ftp.myong.top\u002Fstorage\u002Farticle\u002F8b\u002Fa6c2722c9a3232b2c950943d7908b8.jpg",[39,44,49,54,59],{"id":40,"title":41,"create_time":42,"description":43},95,"Django启用memcache缓存","2020-10-23 11:17:27","在动态网站中,用户所有的请求,服务器都会去数据库中进行相应的增,删,查,改,渲染模板,执行业务逻辑,最后生成用户看到的页面.当一个网站的用户访问量很大的时候,每一次的的后台操作,都会消耗很多的服务端资源,所以必须使用缓存来减轻后端服务器的压力.",{"id":45,"title":46,"create_time":47,"description":48},100,"re正则表达式","2021-07-10 18:09:52","正则表达式是一个特殊的字符序列，它能帮助你方便的检查一个字符串是否与某种模式匹配，本文罗列re正则表达式中，常用的匹配符号。",{"id":50,"title":51,"create_time":52,"description":53},34,"Python基础整理之列表常见操作","2019-02-10 16:53:55","列表是最常用的Python数据类型，它可以作为一个方括号内的逗号分隔值出现。列表的数据项不需要具有相同的类型",{"id":55,"title":56,"create_time":57,"description":58},63,"Python与mysqldump的数据库备份","2019-10-14 15:35:33","mysqldump是备份MySQL数据库的一种好工具。它相对于用phpmyadmin等备份工具更加快速，又避免受php.ini等的限制，在windows系统下还可以结合计划任务实现定时远程备份数据库",{"id":60,"title":61,"create_time":62,"description":63},73,"城市天气爬去并绘制图表","2020-01-03 16:20:15","以爬去深圳城市七天天气为例，`BeautifulSoup`爬去天气数据，`echart`绘制最高气温与最低气温折线图。",1783431657359]