This would allow it to begin tokenising mid way through downloading content, rather than requiring it all to be in memory before beginning.