• danA
    link
    fedilink
    arrow-up
    1
    ·
    2 hours ago

    Unfortunately there’s a huge number of MySQL servers that are using latin1 charset and collation, even when storing Unicode data. Surprisingly it “works” as long as your DB client treats the data as raw bytes (ie you don’t do any sort of string manipulation in MySQL), the same way that UTF-8 “works” in PHP even though native strings are just raw bytes.

    ,Obviously sorting and filtering won’t take any sort of case insensitivity / case folding into account though, which is an issue.

    I hit issues with this recently at work, because for some reason all our MySQL servers still use latin1 by default. Unicode characters were getting corrupted in JSON columns, because MySQL forces them to use utf8mb4 as per the JSON spec (JSON sent between multiple systems should always be encoded as UTF-8), and mixing “real UTF-8” with “fake UTF-8” (UTF-8 in latin1) doesnt end well.