The tokenSize parameter is a query processor parameter that is used only during the query compilation.
It defines the size of the memory buffer allocated for every string literal that appears in the query.
This means that higher tokenSize values result in higher memory consumption by the query processor High tokenSize values should not affect performance (given there is enough memory on the machine to accommodate the higher memory consumption of the query processor)

Note that using a parameterized query by replacing the long literals with parameters, you will no longer need to set the tokenSize.