First step was to verify that everything was UTF-8; the HTML header, the template (html) files, the (html) partial files etc. Then check that all config settings were set to UTF-8; web.config and umbracoSettings.config.
Still didn't work. I also tried manually switching encoding in chrome (tools->encoding) but it still didn't work.
The fix:
I needed to add the fileEncoding attribute to the globalization tag in web.config
before:
<globalization requestEncoding="UTF-8" responseEncoding="UTF-8" />
after:
<globalization requestEncoding="UTF-8" responseEncoding="UTF-8" fileEncoding="UTF-8" />
Ah, well...