Lesson 11 of 20 · 52 min · UG / professional
Unicode text vs bytes
UTF-8 for student names. Bytes for JPEG and sockets. Opening a text file with the wrong encoding is how Indian names die. Source files are UTF-8 too.
Class notice message
Strings are text you can shape.
You will be able to
- ✓ Say UTF-8 out loud for college text files and source files
- ✓ Treat bytes as for JPEG, sockets and blobs, not for student names
- ✓ Normalise a filing key without vandalising the ID-card spelling
Indian names die in the wrong encoding
UTF-8 is the default you should say out loud when you open rolls.csv. Windows may guess another encoding and turn a name into punctuation. Bytes are for gate.jpg, encrypted blobs and sockets. A CSV of attendance is text. On a laptop you write open(path, encoding="utf-8"). Source .py files are UTF-8 too — same agreement as the data.
If a vendor sends a legacy file, you document the encoding. You do not guess every morning. Unicode is the idea that one character is not always one byte. You do not need to memorise code points. You need to never open college text as ‘whatever the OS feels’.
Normalise for people, preserve for storage
.lower() is for a comparison key (RSIL vs rsil). The stored display name should keep the spelling the student used, in UTF-8. Do not uppercase a Marathi name ‘to be safe’ if the ID card does not.
len of a Python string here is characters. On a laptop, len of a bytes object is octets — do not mix them in one ==.
Words that matter
- UTF-8
- The encoding to use for almost all new text files and source.
- Bytes
- Raw 0–255 values — files, sockets, images.
- Encoding
- The agreement that turns bytes into characters.
Common mistakes
Avoid: open(path) with no encoding on a machine that is not UTF-8 by default.
Do this: encoding="utf-8" unless you have a documented vendor reason.
Run it step by step
Each box is a real program. Press Run, change a number, Run again — the output must follow your code.
1. Step 1 — compare a filing key, keep display
lower is for ==. The original string stays for the ID card.
Edit the example, press Run, then Build if you want a compile check.
build
Press Build to compile.
2. Step 2 — length of text is characters here
On a laptop, len of bytes can differ. Do not mix them.
Edit the example, press Run, then Build if you want a compile check.
build
Press Build to compile.
On a full Python install — open + encoding
Student names, notices, CSV headers. Say UTF-8. Read bytes only when the file is not text.
No pip — language + stdlib.
Text vs bytes at the file edge. Source files are UTF-8 as well.
Real library code (not run in this browser sandbox)
# -*- coding: utf-8 -*-
path = "rolls.csv"
with open(path, encoding="utf-8") as f:
text = f.read()
with open("gate.jpg", "rb") as f:
blob = f.read(16)
print("UTF-8 header", text.splitlines()[0] if text else "", "bytes", len(blob))
Example program — A row that must survive a file round-trip
Keys are ASCII. The name field is still a string you must encode as UTF-8 on disk.
Edit the example, press Run, then Build if you want a compile check.
build
Press Build to compile.
Your turn — Filing key
raw = "Labs". Print the lowercase form so labs appears.
Edit the example, press Run, then Build if you want a compile check.
build
Press Build to compile.
Self-assessment
Check your understanding before you mark the lesson complete.
Progress is stored in a browser cookie on this device.