It doesn't tokenize, it just splits; there's a difference -- if your tokens have embedded separators (consider, a space-delimited sequence of tokens, where tokens can have backslash-escaped ...