// HACKER NEWS — CYBERSECURITY
Parsing the Infamous Japanese Postal CSV
Late last year I released posuto, a package presenting Japanese postal code
data in an easy-to-use format. It's based on data released by Japan
Post, which is infamous for being widely used but hard to parse.
This adorable character by Irasutoya is cute, but the raw postal CSV data is not.
I first became aware of the postal data when I entered my postal code in an
online form and it auto-completed my address as "XXX-borough (except the
following buildings)". I had no idea what that parenthetical was referring to,
so I looked for a common source of postal data, found the CSV, and found the
issue. It turns out the CSV file contains parenthetical notes for anyone
reading the CSV file and makes reference to the order of the rows.
This causes problems. The data is mainly useful one row at a time, where the
parenthetical is meaningless. Since CSV is a field-delimited format, there's
also no need for parentheticals - you could just add a note field.
This is only one of many issues with ken_all.csv. You can find people
complaining about it regularly on Twitter, and there was
even briefly a blog just collecting posts from all over the web
about it. A particularly amusing tweet describes people who expect
computers to bend to the will of humans being punished in Hell by having to
parse ken_all.csv forever.
The README for the file explains that lines with overly long fields will be
broken up into multiple lines. Specifically, if the neighborhood name is over
38 characters, or if the half-width katakana (half-width katakana)
pronunciation field is over 76 characters, the line will be split into two
lines. The overly-long neighborhood field will be continued and all other
fields will be duplicated. This is an abbreviated sample of what that looks
like:
The motivation for this is not explained. Maybe there was a fixed-width buffer
for storing a line somewhere thirty years ago. I used to process CSV and other
files from hundreds of different providers at an old job and I saw many
horrors, but I've never seen this particular formatting choice anywhere else.
It should also be noted that while the length limits are as stated, the
location where line breaks are inserted in long lines appears random, occurring
neither at the character limit nor at normal word boundaries.
It's worth noting not all the issues with the CSV are inherently technical;
postal codes are always complicated. The postal code with the most rows in the
CSV - a stunning 66 - is 〒452-0961, which refers to the Haruhi region of
Kiyosu City in Aichi Prefecture. This has that many lines because every
neighborhood gets a separate line. (This particular case may be related to Haruhi
having been the smallest town by area in Japan from 2006 until 2009, when it
was incorporated into Kiyosu City.)
In contrast, the longest continued line, using the line break rules above, is
the entry for 〒602-8368 or 〒602-8374, both with eight lines. These are both in
one of a few areas in Kyoto that uses a unique, bizarre system of
intersection-based addressing. The entry looks a bit like this:
I have used quoted fields here, but the actual CSV doesn't quote fields and
instead uses a different kind of comma.