How to capture delimiters also as tokens in result

Question

Below is the code for creating tokens based on delimiters. This will give the tokens but will not consider delimiters in token result. If we want to include the delimiters also in result, what should be the approach.

StringTokenizer tr=
new StringTokenizer("Will you come, yes/no",",/");
while (tr.hasMoreElements( ))
System.out.println(tr.nextElement( )+",");

So output should be tokens as:

Will, ,you, ,come, ,yes,/,no,

(Instead of just will,you,come,yes,no,)

Answer 1

/ is not coming as one of the tokens because it is not surrounded by the delimiter. If you want it to be parsed as one of the tokens then you can surround it with delimiters.

Here is the code snippet:

public static void main (String[] args)
{
    String sample = "Will you come,yes/no";
    StringTokenizer tr = new StringTokenizer(sample.replace(" ",",").replace("/",",/,"),",");
    while (tr.hasMoreElements())
        System.out.println(tr.nextElement() + ",");
}

Output:

Will,
you,
come,
yes,
/,
no,