Add simple tokenizer

2025-11-06 07:20:31 +01:00
commit 33c7349178
5 changed files with 307 additions and 0 deletions
--- a/test.txt
+++ b/test.txt
@@ -0,0 +1,4 @@
+
+hello world 123 "this is a string"
+
+goodbye world 123