Date: 2012jun20 Update: 2026mar30 Language: Java Keywords: token, tokenize Q. Java: Convert a String into tokens A. Java has these classes/methods builtin:

String.split() // Simplest java.io.StreamTokenizer // Expects java style object like float, int java.util.StringTokenizer // By delimiting characters java.util.Scanner // Expects java style objects like float, int java.text.BreakIterator // For English words

If you want something fancier look for a "lexer". Or look for a parser for the language you are dealing with. eg JSON, XML, RSS etc have parsers. We have detailed articles about most of these: https://www.davekb.com/browse_programming_tips:java_string_split.txt https://www.davekb.com/browse_programming_tips:java_streamtokenizer_on_a_string.txt https://www.davekb.com/browse_programming_tips:java_stringtokenizer:txt https://www.davekb.com/browse_programming_tips:java_scanner_example:txt https://www.davekb.com/browse_programming_tips:java_split_into_words:txt