{"tag":"encoding","articles":{"blog\/technology\/emojis-in-doctrine":{"key":"blog\/technology\/emojis-in-doctrine","type":"article","published":true,"meta":{"createdAt":"2017-05-25T19:08:08+02:00","publishedAt":"2017-05-25T19:03:00+02:00","group":null,"category":"blog","subcategory":"technology","slug":"emojis-in-doctrine"},"content":{"en":{"slug":"emojis-in-doctrine","title":"Emojis in Doctrine \ud83d\ude0e","intro":"\u003Cp\u003EPutting emojis in your database should be a piece of cake, right? You\u2019ve had enough trouble with encodings in your lifetime, and now that we have the blessing of \u003Cstrong\u003EUTF-8\u003C\/strong\u003E, you\u2019re always so careful to use it everywhere, so you\u2019d expect all the characters to just finally work out of the box, right?\u003C\/p\u003E\n\u003Cp\u003EWell, I did expect that. But I\u2019ve recently realised I can only put \u003Cem\u003Esome\u003C\/em\u003E emojis (like \u201c\u2764\ufe0f\u201d) in by blog posts. Most of them were just lost or replaced with \u201c?\u201d by MySQL... Oh, those damn encodings again!\u003C\/p\u003E\n\u003Cp\u003EBut fortunately the solution is quite simple.\u003C\/p\u003E\u003Csvg xmlns=\u0022http:\/\/www.w3.org\/2000\/svg\u0022 style=\u0022display: none;\u0022\u003E\u003C\/svg\u003E","content":"\u003Cp\u003EPutting emojis in your database should be a piece of cake, right? You\u2019ve had enough trouble with encodings in your lifetime, and now that we have the blessing of \u003Cstrong\u003EUTF-8\u003C\/strong\u003E, you\u2019re always so careful to use it everywhere, so you\u2019d expect all the characters to just finally work out of the box, right?\u003C\/p\u003E\n\u003Cp\u003EWell, I did expect that. But I\u2019ve recently realised I can only put \u003Cem\u003Esome\u003C\/em\u003E emojis (like \u201c\u2764\ufe0f\u201d) in by blog posts. Most of them were just lost or replaced with \u201c?\u201d by MySQL... Oh, those damn encodings again!\u003C\/p\u003E\n\u003Cp\u003EBut fortunately the solution is quite simple.\u003C\/p\u003E\n\u003Cp\u003EInstead of the \u003Ccode\u003Eutf8\u003C\/code\u003E charset, you need to use its superset, \u003Ccode\u003Eutf8mb4\u003C\/code\u003E. This one handles 4-byte characters and that\u2019s exactly what we need. We just have to make both the database and \u003Ca href=\u0022http:\/\/www.doctrine-project.org\/projects\/orm.html\u0022 target=\u0022_blank\u0022 rel=\u0022noopener\u0022\u003E\u003Csvg class=\u0022icon\u0022\u003E\u003Cuse xlink:href=\u0022#light-link\u0022\u003E\u003C\/use\u003E\u003C\/svg\u003E Doctrine\u003C\/a\u003E use it.\u003C\/p\u003E\n\u003Cp\u003E\u003Ca href=\u0022https:\/\/www.phpmyadmin.net\/\u0022 target=\u0022_blank\u0022 rel=\u0022noopener\u0022\u003E\u003Csvg class=\u0022icon\u0022\u003E\u003Cuse xlink:href=\u0022#light-link\u0022\u003E\u003C\/use\u003E\u003C\/svg\u003E phpMyAdmin\u003C\/a\u003E or \u003Ca href=\u0022https:\/\/www.adminer.org\/\u0022 target=\u0022_blank\u0022 rel=\u0022noopener\u0022\u003E\u003Csvg class=\u0022icon\u0022\u003E\u003Cuse xlink:href=\u0022#light-link\u0022\u003E\u003C\/use\u003E\u003C\/svg\u003E Adminer\u003C\/a\u003E let you change the charset easily. Just go to edit table schema and switch the collation of relevant columns and tables to \u003Ccode\u003Eutf8mb4_unicode_ci\u003C\/code\u003E. Since you\u2019re going from \u003Ccode\u003Eutf8\u003C\/code\u003E to its superset, no data should become corrupted in the process. But just to play it safe, better create a \u003Cstrong\u003Ebackup\u003C\/strong\u003E before you start.\u003C\/p\u003E\n\u003Cp\u003EYou might encounter an error: \u201cSpecified key was too long; max key length is 767 bytes\u201d. That\u2019s because of some limitations of InnoDB regarding indices length. To make an index into \u003Ccode\u003Emb4\u003C\/code\u003E, you\u2019ll have to shorten the max length of a column to 191 (instead of max 255 for simple \u003Ccode\u003Eutf8\u003C\/code\u003E).\u003C\/p\u003E\n\u003Cp\u003EWhen it comes to Doctrine, you have to specify the collation and charset on the level of an entity\/table and on the level of the connection.\u003C\/p\u003E\n\u003Cp\u003EFor an entity, if you\u2019re using annotations:\u003C\/p\u003E\n\u003Cpre\u003E\u003Ccode class=\u0022hljs bash border\u0022\u003E\/**\n * @ORM\\Entity\n * @ORM\\Table(options={\u003Cspan class=\u0022hljs-string\u0022\u003E\u0022collate\u0022\u003C\/span\u003E=\u003Cspan class=\u0022hljs-string\u0022\u003E\u0022utf8mb4_unicode_ci\u0022\u003C\/span\u003E, \u003Cspan class=\u0022hljs-string\u0022\u003E\u0022charset\u0022\u003C\/span\u003E=\u003Cspan class=\u0022hljs-string\u0022\u003E\u0022utf8mb4\u0022\u003C\/span\u003E})\n *\/\n\u003C\/code\u003E\u003C\/pre\u003E\n\u003Cp\u003EOr if you\u2019re using XML:\u003C\/p\u003E\n\u003Cpre\u003E\u003Ccode class=\u0022hljs xml border\u0022\u003E\u003Cspan class=\u0022hljs-tag\u0022\u003E\u0026lt;\u003Cspan class=\u0022hljs-name\u0022\u003Eentity\u003C\/span\u003E \u003Cspan class=\u0022hljs-attr\u0022\u003Ename\u003C\/span\u003E=\u003Cspan class=\u0022hljs-string\u0022\u003E\u0022Acme\\DemoBundle\\Entity\\User\u0022\u003C\/span\u003E\u0026gt;\u003C\/span\u003E\n    \u003Cspan class=\u0022hljs-tag\u0022\u003E\u0026lt;\u003Cspan class=\u0022hljs-name\u0022\u003Eoptions\u003C\/span\u003E\u0026gt;\u003C\/span\u003E\n        \u003Cspan class=\u0022hljs-tag\u0022\u003E\u0026lt;\u003Cspan class=\u0022hljs-name\u0022\u003Eoption\u003C\/span\u003E \u003Cspan class=\u0022hljs-attr\u0022\u003Ename\u003C\/span\u003E=\u003Cspan class=\u0022hljs-string\u0022\u003E\u0022charset\u0022\u003C\/span\u003E\u0026gt;\u003C\/span\u003Eutf8mb4\u003Cspan class=\u0022hljs-tag\u0022\u003E\u0026lt;\/\u003Cspan class=\u0022hljs-name\u0022\u003Eoption\u003C\/span\u003E\u0026gt;\u003C\/span\u003E\n        \u003Cspan class=\u0022hljs-tag\u0022\u003E\u0026lt;\u003Cspan class=\u0022hljs-name\u0022\u003Eoption\u003C\/span\u003E \u003Cspan class=\u0022hljs-attr\u0022\u003Ename\u003C\/span\u003E=\u003Cspan class=\u0022hljs-string\u0022\u003E\u0022collate\u0022\u003C\/span\u003E\u0026gt;\u003C\/span\u003Eutf8mb4_unicode_ci\u003Cspan class=\u0022hljs-tag\u0022\u003E\u0026lt;\/\u003Cspan class=\u0022hljs-name\u0022\u003Eoption\u003C\/span\u003E\u0026gt;\u003C\/span\u003E\n    \u003Cspan class=\u0022hljs-tag\u0022\u003E\u0026lt;\/\u003Cspan class=\u0022hljs-name\u0022\u003Eoptions\u003C\/span\u003E\u0026gt;\u003C\/span\u003E\n\u003Cspan class=\u0022hljs-tag\u0022\u003E\u0026lt;\/\u003Cspan class=\u0022hljs-name\u0022\u003Eentity\u003C\/span\u003E\u0026gt;\u003C\/span\u003E\n\u003C\/code\u003E\u003C\/pre\u003E\n\u003Cp\u003EAs for the connection, go with:\u003C\/p\u003E\n\u003Cpre\u003E\u003Ccode class=\u0022hljs php border\u0022\u003E$conn = \\Doctrine\\DBAL\\DriverManager::getConnection(\n    [\n        \u003Cspan class=\u0022hljs-string\u0022\u003E\u0027driver\u0027\u003C\/span\u003E   =\u0026gt; \u003Cspan class=\u0022hljs-string\u0022\u003E\u0027pdo_mysql\u0027\u003C\/span\u003E,\n        \u003Cspan class=\u0022hljs-string\u0022\u003E\u0027host\u0027\u003C\/span\u003E     =\u0026gt; \u003Cspan class=\u0022hljs-string\u0022\u003E\u0027localhost\u0027\u003C\/span\u003E,\n        \u003Cspan class=\u0022hljs-string\u0022\u003E\u0027user\u0027\u003C\/span\u003E     =\u0026gt; \u003Cspan class=\u0022hljs-string\u0022\u003E\u0027user\u0027\u003C\/span\u003E,\n        \u003Cspan class=\u0022hljs-string\u0022\u003E\u0027password\u0027\u003C\/span\u003E =\u0026gt; \u003Cspan class=\u0022hljs-string\u0022\u003E\u0027password\u0027\u003C\/span\u003E,\n        \u003Cspan class=\u0022hljs-string\u0022\u003E\u0027dbname\u0027\u003C\/span\u003E   =\u0026gt; \u003Cspan class=\u0022hljs-string\u0022\u003E\u0027database\u0027\u003C\/span\u003E,\n        \u003Cspan class=\u0022hljs-string\u0022\u003E\u0027charset\u0027\u003C\/span\u003E  =\u0026gt; \u003Cspan class=\u0022hljs-string\u0022\u003E\u0027utf8mb4\u0027\u003C\/span\u003E,\n    ],\n    \u003Cspan class=\u0022hljs-keyword\u0022\u003Enew\u003C\/span\u003E \\Doctrine\\DBAL\\Configuration()\n);\n\u003C\/code\u003E\u003C\/pre\u003E\n\u003Cp\u003EOr in \u003Ca href=\u0022https:\/\/symfony.com\/\u0022 target=\u0022_blank\u0022 rel=\u0022noopener\u0022\u003E\u003Csvg class=\u0022icon\u0022\u003E\u003Cuse xlink:href=\u0022#light-link\u0022\u003E\u003C\/use\u003E\u003C\/svg\u003E Symfony\u003C\/a\u003E:\u003C\/p\u003E\n\u003Cpre\u003E\u003Ccode class=\u0022hljs yaml border\u0022\u003E\u003Cspan class=\u0022hljs-comment\u0022\u003E# app\/config\/config.yml\u003C\/span\u003E\n\u003Cspan class=\u0022hljs-attr\u0022\u003Edoctrine:\u003C\/span\u003E\n    \u003Cspan class=\u0022hljs-attr\u0022\u003Edbal:\u003C\/span\u003E\n        \u003Cspan class=\u0022hljs-attr\u0022\u003Echarset:\u003C\/span\u003E  \u003Cspan class=\u0022hljs-string\u0022\u003Eutf8mb4\u003C\/span\u003E\n\u003C\/code\u003E\u003C\/pre\u003E\n\u003Cp\u003EOr in \u003Ca href=\u0022https:\/\/micrus.avris.it\/\u0022 target=\u0022_blank\u0022 rel=\u0022noopener\u0022\u003E\u003Csvg class=\u0022icon\u0022\u003E\u003Cuse xlink:href=\u0022#light-link\u0022\u003E\u003C\/use\u003E\u003C\/svg\u003E Micrus\u003C\/a\u003E:\u003C\/p\u003E\n\u003Cpre\u003E\u003Ccode class=\u0022hljs yaml border\u0022\u003E\u003Cspan class=\u0022hljs-comment\u0022\u003E# app\/Config\/parameters.yml\u003C\/span\u003E\n\u003Cspan class=\u0022hljs-attr\u0022\u003Edatabase:\u003C\/span\u003E\n  \u003Cspan class=\u0022hljs-attr\u0022\u003Edriver:\u003C\/span\u003E \u003Cspan class=\u0022hljs-string\u0022\u003Epdo_mysql\u003C\/span\u003E\n  \u003Cspan class=\u0022hljs-attr\u0022\u003Ehost:\u003C\/span\u003E \u003Cspan class=\u0022hljs-number\u0022\u003E127.0\u003C\/span\u003E\u003Cspan class=\u0022hljs-number\u0022\u003E.0\u003C\/span\u003E\u003Cspan class=\u0022hljs-number\u0022\u003E.1\u003C\/span\u003E\n  \u003Cspan class=\u0022hljs-attr\u0022\u003Edbname:\u003C\/span\u003E \u003Cspan class=\u0022hljs-string\u0022\u003Edatabase\u003C\/span\u003E\n  \u003Cspan class=\u0022hljs-attr\u0022\u003Euser:\u003C\/span\u003E \u003Cspan class=\u0022hljs-string\u0022\u003Euser\u003C\/span\u003E\n  \u003Cspan class=\u0022hljs-attr\u0022\u003Epassword:\u003C\/span\u003E \u003Cspan class=\u0022hljs-string\u0022\u003Epassword\u003C\/span\u003E\n  \u003Cspan class=\u0022hljs-attr\u0022\u003Echarset:\u003C\/span\u003E \u003Cspan class=\u0022hljs-string\u0022\u003Eutf8mb4\u003C\/span\u003E\n\u003C\/code\u003E\u003C\/pre\u003E\n\u003Cp\u003EAnd that would be it (hopefully). Enjoy your new emoji support! \ud83d\ude0a\u003C\/p\u003E\u003Csvg xmlns=\u0022http:\/\/www.w3.org\/2000\/svg\u0022 style=\u0022display: none;\u0022\u003E\u003C\/svg\u003E","tags":["charset","databse","doctrine","emoji","encoding","mysql","unicode","utf8"],"hasMore":true,"image":null,"introLite":"\u003Cp\u003EPutting emojis in your database should be a piece of cake, right? You\u2019ve had enough trouble with encodings in your lifetime, and now that we have the blessing of \u003Cstrong\u003EUTF-8\u003C\/strong\u003E, you\u2019re always so careful to use it everywhere, so you\u2019d expect all the characters to just finally work out of the box, right?\u003C\/p\u003E\n\u003Cp\u003EWell, I did expect that. But I\u2019ve recently realised I can only put \u003Cem\u003Esome\u003C\/em\u003E emojis (like \u201c\u2764\ufe0f\u201d) in by blog posts. Most of them were just lost or replaced with \u201c?\u201d by MySQL... Oh, those damn encodings again!\u003C\/p\u003E\n\u003Cp\u003EBut fortunately the solution is quite simple.\u003C\/p\u003E","contentLite":"\u003Cp\u003EPutting emojis in your database should be a piece of cake, right? You\u2019ve had enough trouble with encodings in your lifetime, and now that we have the blessing of \u003Cstrong\u003EUTF-8\u003C\/strong\u003E, you\u2019re always so careful to use it everywhere, so you\u2019d expect all the characters to just finally work out of the box, right?\u003C\/p\u003E\n\u003Cp\u003EWell, I did expect that. But I\u2019ve recently realised I can only put \u003Cem\u003Esome\u003C\/em\u003E emojis (like \u201c\u2764\ufe0f\u201d) in by blog posts. Most of them were just lost or replaced with \u201c?\u201d by MySQL... Oh, those damn encodings again!\u003C\/p\u003E\n\u003Cp\u003EBut fortunately the solution is quite simple.\u003C\/p\u003E\n\u003Cp\u003EInstead of the \u003Ccode\u003Eutf8\u003C\/code\u003E charset, you need to use its superset, \u003Ccode\u003Eutf8mb4\u003C\/code\u003E. This one handles 4-byte characters and that\u2019s exactly what we need. We just have to make both the database and \u003Ca href=\u0022http:\/\/www.doctrine-project.org\/projects\/orm.html\u0022 target=\u0022_blank\u0022 rel=\u0022noopener\u0022\u003E Doctrine\u003C\/a\u003E use it.\u003C\/p\u003E\n\u003Cp\u003E\u003Ca href=\u0022https:\/\/www.phpmyadmin.net\/\u0022 target=\u0022_blank\u0022 rel=\u0022noopener\u0022\u003E phpMyAdmin\u003C\/a\u003E or \u003Ca href=\u0022https:\/\/www.adminer.org\/\u0022 target=\u0022_blank\u0022 rel=\u0022noopener\u0022\u003E Adminer\u003C\/a\u003E let you change the charset easily. Just go to edit table schema and switch the collation of relevant columns and tables to \u003Ccode\u003Eutf8mb4_unicode_ci\u003C\/code\u003E. Since you\u2019re going from \u003Ccode\u003Eutf8\u003C\/code\u003E to its superset, no data should become corrupted in the process. But just to play it safe, better create a \u003Cstrong\u003Ebackup\u003C\/strong\u003E before you start.\u003C\/p\u003E\n\u003Cp\u003EYou might encounter an error: \u201cSpecified key was too long; max key length is 767 bytes\u201d. That\u2019s because of some limitations of InnoDB regarding indices length. To make an index into \u003Ccode\u003Emb4\u003C\/code\u003E, you\u2019ll have to shorten the max length of a column to 191 (instead of max 255 for simple \u003Ccode\u003Eutf8\u003C\/code\u003E).\u003C\/p\u003E\n\u003Cp\u003EWhen it comes to Doctrine, you have to specify the collation and charset on the level of an entity\/table and on the level of the connection.\u003C\/p\u003E\n\u003Cp\u003EFor an entity, if you\u2019re using annotations:\u003C\/p\u003E\n\u003Cpre\u003E\u003Ccode\u003E\/**\n * @ORM\\Entity\n * @ORM\\Table(options={\u0022collate\u0022=\u0022utf8mb4_unicode_ci\u0022, \u0022charset\u0022=\u0022utf8mb4\u0022})\n *\/\u003C\/code\u003E\u003C\/pre\u003E\n\u003Cp\u003EOr if you\u2019re using XML:\u003C\/p\u003E\n\u003Cpre\u003E\u003Ccode\u003E\u0026lt;entity name=\u0022Acme\\DemoBundle\\Entity\\User\u0022\u0026gt;\n    \u0026lt;options\u0026gt;\n        \u0026lt;option name=\u0022charset\u0022\u0026gt;utf8mb4\u0026lt;\/option\u0026gt;\n        \u0026lt;option name=\u0022collate\u0022\u0026gt;utf8mb4_unicode_ci\u0026lt;\/option\u0026gt;\n    \u0026lt;\/options\u0026gt;\n\u0026lt;\/entity\u0026gt;\u003C\/code\u003E\u003C\/pre\u003E\n\u003Cp\u003EAs for the connection, go with:\u003C\/p\u003E\n\u003Cpre\u003E\u003Ccode\u003E$conn = \\Doctrine\\DBAL\\DriverManager::getConnection(\n    [\n        \u0027driver\u0027   =\u0026gt; \u0027pdo_mysql\u0027,\n        \u0027host\u0027     =\u0026gt; \u0027localhost\u0027,\n        \u0027user\u0027     =\u0026gt; \u0027user\u0027,\n        \u0027password\u0027 =\u0026gt; \u0027password\u0027,\n        \u0027dbname\u0027   =\u0026gt; \u0027database\u0027,\n        \u0027charset\u0027  =\u0026gt; \u0027utf8mb4\u0027,\n    ],\n    new \\Doctrine\\DBAL\\Configuration()\n);\u003C\/code\u003E\u003C\/pre\u003E\n\u003Cp\u003EOr in \u003Ca href=\u0022https:\/\/symfony.com\/\u0022 target=\u0022_blank\u0022 rel=\u0022noopener\u0022\u003E Symfony\u003C\/a\u003E:\u003C\/p\u003E\n\u003Cpre\u003E\u003Ccode\u003E# app\/config\/config.yml\ndoctrine:\n    dbal:\n        charset:  utf8mb4\u003C\/code\u003E\u003C\/pre\u003E\n\u003Cp\u003EOr in \u003Ca href=\u0022https:\/\/micrus.avris.it\/\u0022 target=\u0022_blank\u0022 rel=\u0022noopener\u0022\u003E Micrus\u003C\/a\u003E:\u003C\/p\u003E\n\u003Cpre\u003E\u003Ccode\u003E# app\/Config\/parameters.yml\ndatabase:\n  driver: pdo_mysql\n  host: 127.0.0.1\n  dbname: database\n  user: user\n  password: password\n  charset: utf8mb4\u003C\/code\u003E\u003C\/pre\u003E\n\u003Cp\u003EAnd that would be it (hopefully). Enjoy your new emoji support! \ud83d\ude0a\u003C\/p\u003E","words":419,"readTime":2,"lang":"en"}}}}}