It's not strange. This is not ASCII sorting, it's sorting based on
actual lexigraphical sorting rules for the language/locale, in other
words what you'd expect from any paper index. Non-alphanumeric
characters cannot be ordered lexigraphically, so they are ignored.
If you want ASCII sorting (based on ASCII value), use the C locale.
LC_COLLATE is the variable that controls sorting, so make sure
LC_COLLATE is "C" in that case (also make sure you unset LC_ALL, else
you won't be able to change LC_COLLATE).