Add simple tokenizer

This commit is contained in:
2025-11-06 07:20:31 +01:00
commit 33c7349178
5 changed files with 307 additions and 0 deletions

4
test.txt Normal file
View File

@@ -0,0 +1,4 @@
hello world 123 "this is a string"
goodbye world 123