Python Morsels: Unicode character encodings

Table of contents

1. All input starts as raw bytes
2. Encoding strings into bytes
3. Decoding bytes into strings
4. Specifying a character encoding when opening files
5. Be careful with your character encodings
6. Summary

All input starts as raw bytes

When you open a file in Python, the default mode is r or rt, for read text mode:

>>> withopen("my_file.txt")asf:... contents=f.read()...>>> f.mode'r'

>>> withopen("my_file.txt")asf:... contents=f.read()...>>> f.mode'r'

Meaning when we read our file, we'll get back strings that represent text:

>>> contents'This is a file ✨\n'

>>> contents'This is a file ✨\n'

But that's not what Python actually reads from disk.

If we open a file with the mode rb and read from our file we'll see what Python sees; that is bytes:

>>> withopen("my_file.txt",mode="rb")asf:... contents=f.read()...>>> contentsb'This is a file \xe2\x9c\xa8\n'>>> type(contents)<class 'bytes'>

>>> withopen("my_file.txt",mode="rb")asf:... contents=f.read()...>>> contentsb'This is a file \xe2\x9c\xa8\n'>>> type(contents)<class 'bytes'>

Bytes are what Python decodes to make strings.

Encoding strings into bytes

If you have a string …