[{"data":1,"prerenderedAt":65},["ShallowReactive",2],{"article-106":3},{"code":4,"msg":5,"data":6,"count":14},200,"查询成功",{"id":7,"title":8,"keywords":9,"description":10,"category_id":11,"content":12,"body_html":13,"thumb_up":14,"clicks":15,"sort":14,"remark":13,"status":11,"is_open":11,"is_deleted":14,"is_top":11,"is_recommend":11,"create_time":16,"update_time":17,"image_id":18,"url":13,"member_id":14,"cate_name":19,"prev":20,"next":23,"tags":26,"words":36,"read_time":37,"comments":14,"cover":38,"relevant":39},106,"阿里开源数据同步组件Canal","canal,同步,数据库","最开始听说canal是从mysql与redis双写一致性解决方案，当时并没有太在意，最近由于需要实时同步数据，如果在代码对insert\u002Fupdate\u002Fdelete做拦截也可以实现，但对代码侵入性太大了，并且后期更改时容易有遗漏，风险太高，这时就又想到了canal，canal的好处在于对业务代码没有侵入，因为是基于监听binlog日志去进行同步数据，这个真的是太爽爽爽了。并且实时性也能做到准实时，这也是canal为什么这么流行，因为确实很多企业会用来做数据同步的方案。",1,"![image](https:\u002F\u002Ftp.myong.top\u002Fstorage\u002Farticle\u002F20231026\u002Fe2e805562a97f8bdff3a117811d69914.png)\n\n##### 简介\n阿里canal主要用途是基于 MySQL 数据库增量日志解析，提供增量数据订阅和消费\n\n早期阿里巴巴因为杭州和美国双机房部署，存在跨机房同步的业务需求，实现方式主要是基于业务 trigger 获取增量变更。从 2010 年开始，业务逐步尝试数据库日志解析获取增量变更进行同步，由此衍生出了大量的数据库增量订阅和消费业务。\n\n基于日志增量订阅和消费的业务包括\n```python\n数据库镜像\n数据库实时备份\n索引构建和实时维护(拆分异构索引、倒排索引等)\n业务 cache 刷新\n带业务逻辑的增量数据处理\n```\n\n当前的 canal 支持源端 MySQL 版本包括 5.1.x , 5.5.x , 5.6.x , 5.7.x , 8.0.x\n\n##### MySQL主备复制原理\n\n![image](https:\u002F\u002Ftp.myong.top\u002Fstorage\u002Farticle\u002F20231026\u002F4d7f2c14cd6b2cb1acc04f53ce1a5ad4.png)\n\n```python\nMySQL master 将数据变更写入二进制日志( binary log, 其中记录叫做二进制日志事件binary log events，可以通过 show binlog events 进行查看)\nMySQL slave 将 master 的 binary log events 拷贝到它的中继日志(relay log)\nMySQL slave 重放 relay log 中事件，将数据变更反映它自己的数据\n```\n\n##### canal 工作原理\n\n```pyhton\ncanal 模拟 MySQL slave 的交互协议，伪装自己为 MySQL slave ，向 MySQL master 发送dump 协议\nMySQL master 收到 dump 请求，开始推送 binary log 给 slave (即 canal )\ncanal 解析 binary log 对象(原始为 byte 流)\n```\n\n##### MySQL配置\n\n我们提前得有一台MySQL数据库，本文基于mysql 5.7，参考安装装程：window mysql 5.7安装教程\n在安装Canal之前，我们需要做2件事：\n1). 修改MySQL配置支持binlog，\n2). 创建canal用户\n###### 修改MySQL配置支持binlog\n\n修改my.ini，在[mysqld]下配置以下4项：\n```python\n[mysqld]\n# 打开binlog\nlog-bin=mysql-bin\n# 选择ROW(行)模式\nbinlog-format=ROW\n# 配置MySQL replaction需要定义，不要和canal的slaveId重复\nserver_id=1\n# 要监控的数据库名称\nbinlog-do-db=my-test\n```\n\n![image](https:\u002F\u002Ftp.myong.top\u002Fstorage\u002Farticle\u002F20231026\u002F403d30455dc36c755f0b7a83ae37e5f1.png)\n\n###### 创建canal用户\n```python\n1. 使用命令登录：mysql -u root -p\n2. 创建用户 用户名：canal 密码：canal@123456\ncreate user 'canal'@'%' identified by 'canal@123456';\n3. 授权 *.*表示所有库\ngrant SELECT, REPLICATION SLAVE, REPLICATION CLIENT on *.* to 'canal'@'%' identified by 'canal@123456';\n4.重启mysql\n5.查看是否打开binlog模式：show variables like 'log_bin';\n6.查看binlog日志文件列表：show binary logs;\n7.查看当前正在写入的binlog文件：show master status;\n```\n\n##### 下载安装canal\n官网下载：https:\u002F\u002Fgithub.com\u002Falibaba\u002Fcanal\u002Freleases\n\n![image](https:\u002F\u002Ftp.myong.top\u002Fstorage\u002Farticle\u002F20231026\u002F0047c273f540fb590fae6cebba00d6d2.jpg)\n\n\n###### 配置与mysql信息\n打开配置文件conf\u002Fexample\u002Finstance.properties，主要配置数据库地址和用户：\n```python\n# mysql数据库地址\ncanal.instance.master.address=127.0.0.1:3306\n# username\u002Fpassword\ncanal.instance.dbUsername=canal\ncanal.instance.dbPassword=canal@123456\n```\n###### 启动canal\n转到bin目录，cmd运行startup.bat\n注意需要有java环境\n\n![image](https:\u002F\u002Ftp.myong.top\u002Fstorage\u002Farticle\u002F20231026\u002F466441fa3925311ea494a498376fcd40.jpg)\n\n##### Python客户端\n下载官方提供的python客户端demo\n![image](https:\u002F\u002Ftp.myong.top\u002Fstorage\u002Farticle\u002F20231026\u002F5f36ecaebb9564e24109c9c302f49c7d.png)\n\n\n改造example.py，新增一下两个方法即可\n\n###### 解析拼接sql\n```python\ndef data_to_sql(data):\n    db = data['db']\n    table = data['table']\n    sql = ''\n    # insert\n    if data['event_type'] == 1:\n        dic_data = data['data']['after']\n        insert_value = []\n        for key in dic_data.keys():  \n            insert_value.append(f\"'{dic_data[key]}'\") \n        insert_value = ','.join(insert_value)\n        sql = f\"insert into {db}.{table} values ({insert_value});\"\n        return sql\n    # update\n    elif data['event_type'] == 2:\n        before_data = data['data']['before']\n        after_data = data['data']['after']\n        update_value = []\n        update_condition = []\n        for key in before_data.keys():\n            update_condition.append(key + f\"='{before_data[key]}'\") \n        update_condition = ' and '.join(update_condition)\n\n        for key in after_data.keys():\n            update_value.append(key + f\"='{after_data[key]}'\") \n        update_value = ','.join(update_value)\n\n        sql = f\"update {db}.{table} set {update_value} \\\n            where {update_condition};\"   \n        return sql  \n    # delete\n    else:\n        dic_data = data['data']['before']\n        delete_condition = []\n        for key in dic_data.keys():\n            delete_condition.append(key + f\"='{dic_data[key]}'\") \n        delete_condition = ' and '.join(delete_condition)\n        sql = f\"delete from {db}.{table} where {delete_condition};\"\n        return sql\n```\n\n###### 执行sql\n```python\ndef exec_sql(sql):\n    db = pymysql.connect(\n        host=\"127.0.0.1\", \n        user=\"root\", \n        password=\"root\", \n        database=\"testcanal\", \n        charset=\"utf8\",\n        port=3306\n    )\n    cursor = db.cursor()  # 使用cursor（）方法获取游标\n    cursor.execute(sql)  # 执行sql查询语句\n    db.commit()\n    cursor.close()  # 关闭游标\n    db.close()  # 关闭数据库连接\n    print('exec ok')\n\n```\n\n###### 执行py文件\n![image](https:\u002F\u002Ftp.myong.top\u002Fstorage\u002Farticle\u002F20231026\u002F4df1dd471dafcdd4071315459ef0c093.jpg)",null,0,599,"2023-10-26 22:43:32","2024-04-15 00:45:58",65,"技术杂谈",{"id":21,"title":22},105,"php项目中调用python脚本使用pdf和image之间相互转换",{"id":24,"title":25},107,"PHP 导出 Excel 的三种常用方法",[27,30,33],{"id":28,"name":29},26,"MySQL",{"id":31,"name":32},82,"同步",{"id":34,"name":35},83,"Canal",2971,7,"https:\u002F\u002Ftp.myong.top\u002Fstorage\u002Farticle\u002F20230407\u002Fda5dcd2a5c3f5ab09b50c7626eddd200.jpeg",[40,45,50,55,60],{"id":41,"title":42,"create_time":43,"description":44},89,"获取腾讯“邮我”的链接","2020-06-16 10:47:34","腾讯QQ邮箱提供了“邮我”组件，可以放在自己的网站上，让别人点击提供的图片或者链接就可以发Email过来，该文章分享的是如何获取到其中的链接。",{"id":46,"title":47,"create_time":48,"description":49},98,"本站点JavaScript相关特效使用方法整理","2021-04-22 18:38:53","本站的页面用的特效有，粒子线canvas-nest，动态彩带canvas-ribbon，鼠标点击特效以及音乐播放器，本文整理了特效的使用方法",{"id":51,"title":52,"create_time":53,"description":54},62,"内网使用Composer","2019-10-08 23:29:23","最近本地部署Laravel开发环境时，遇到公司的内网无法正常使用Composer下载Laravel，后来发现是要设置公司内网代理",{"id":56,"title":57,"create_time":58,"description":59},75,"Hexo-SEO优化开启静态文件压缩功能","2019-12-30 21:34:32","个人对HEXO搭建博客的SEO优化方案进行总结，从本地的文章结构到定期推送，再到SEO关键词优化做一个全面体系的汇总，如果有更好的方法可以私聊我。\n\n",{"id":61,"title":62,"create_time":63,"description":64},49,"了解域名SSL证书申请安装","2019-03-22 17:17:43","如何将自己的网站的访问协议http转成https协议，就得要求我们必须对域名进行SSl证书申请并应用。",1783431654621]