Download
Tokenize is a Julia package that serves a similar purpose and API as the tokenize module in Python but for Julia. This is to take a string or buffer containing Julia code, perform lexical analysis and return a stream of tokens.
Features
- Fast, it currently lexes all of Julia source files in ~0.25 seconds (580 files, 2 million Tokens)
- Round trippable, that is, from a stream of tokens the original string should be recoverable exactly
- Round trippable, that is, from a stream of tokens the original string should be recoverable exactly
- The function tokenize is the main entrypoint for generating Tokens
- Each Token is represented by where it starts and ends, what string it contains and what type it is
- Documentation available
Project Samples
License
MIT LicenseFollow Tokenize.jl
Other Useful Business Software
Go from Data Warehouse to Data and AI platform with BigQuery
BigQuery is more than a data warehouse—it's an autonomous data-to-AI platform. Use familiar SQL to train ML models, run time-series forecasts, and generate AI-powered insights with native Gemini integration. Built-in agents handle data engineering and data science workflows automatically. Get $300 in free credit, query 1 TB, and store 10 GB free monthly.
Rate This Project
Login To Rate This Project
User Reviews
Be the first to post a review of Tokenize.jl!
