Explore a Tokenizer: Korean and English Token IDs and Embeddings

Enter Korean or English text in the browser tool to inspect token pieces, token IDs and their input positions. Start with 안녕하세요, then select a token and calculate the 384-component vector for that position. The guide distinguishes those contextual vectors from the token ID and a whole-sentence embedding.

Token IDs are integers that distinguish pieces in the vocabulary list, and vectors are arrays of numbers that the model computes. Distinguishing between the two values ​​makes it easier to understand the process of converting a sentence into a model input. This article is a browser exercise for readers who are new to tokenization and embedding.

On this page

Enter sentences directly

Using the Tokenizer When you press , an input window opens and the basic sentence is analyzed. After changing the sentence, click tokenize. You can also change the input using the Korean, English, and mixed example buttons.

Enter Korean or English text to inspect token pieces and IDs.

Open the tokenizer

Your input is processed in this browser. The first run downloads about 17MB of tokenizer files. No API key or login is required.

Open the tool with the button. Your browser must support JavaScript and WebAssembly.

Input statements are processed by the visitor’s browser and are not sent to the server or external AI API for analysis. First, download the tokenizer file, about 17MB. Calculating actual vectors uses approximately 118MB of model and approximately 22MB of additional executable file, so please check the download size first in a mobile data environment.

Reading token fragments and IDs

The tokenizer breaks sentences into smaller units that fit the model’s vocabulary. A word can be a single token or it can be divided into several pieces. One Korean letter or one English word does not always equal one token. Results will vary depending on your separation rules and vocabulary. Hugging Face’s tokenization algorithm description shows the differences between methods.

Result item	meaning	What to check
token piece	String registered in the vocabulary list	It may be different from the original text cut out.
Token ID	An integer that differentiates each piece	Just because the IDs are close doesn’t mean they have similar meanings.
input location	Order of arrays passed to model	The same ID can appear in multiple locations.
special token	A fragment representing the input structure, such as start and end	Displays the number divided by text tokens in the body.

This tool uses the tokenizer of paraphrase-multilingual-MiniLM-L12-v2. ▁ is a word boundary mark in SentencePiece and can also appear before a sentence. <s> and </s> are special tokens that represent the beginning and end. Because of normalization or whitespace handling, the result of simply concatenating the fragments may not be exactly the same as the original text.

In this environment, 안녕하세요 is one piece ▁안녕하세요, and its ID is 107687. The input array including the start and end tokens is [0, 107687, 2]. Hello, world! is divided into four text tokens: ▁Hello, ,, ▁world, !, and the input array is [0, 35378, 4, 8999, 38, 2]. Both results were viewed in this article’s browser tool on October 22, 2026.

The displayed IDs and counts are read relative to this model. Since the fragments and IDs of the same sentence may vary when using tokenizers from different models, we do not apply this result when calculating the input length of other LLMs.

Vectors are computed after tokenization

The processing order is sentence → token fragment → token ID → model calculation → context vector by position. All you have to do is tokenize and you’ll get an ID. The vector button on the screen actually runs the learned model to obtain the number array for each input location.

The model has a learned input embedding table indexed by token ID. This exercise displays the final layer’s contextual vectors, which also reflect surrounding tokens and position. The output is named last_hidden_state and contains 384 components per position. It is distinct from a sentence embedding produced by mean pooling. See the original model description for the contextual output and pooling behavior.

When you select a token, you can read the full value at that position, and the graph displays the first 32 components. The vertical extent of the graph changes to match the selected vector, so there is no absolute value comparison with other tokens based solely on bar height. The screen value is displayed with 6 decimal places, and Stores the entire result as JSON. contains the model information, original text, token order, and the entire vector value.

The calculation uses the q8 quantized ONNX model. Since this is a model with reduced precision of the stored weights, the numbers may be slightly different from the original float32 model. Individual numbers in a vector do not directly correspond to a specific meaning, such as ‘positive score’ or ‘personnel score’.

Comparing the same token in different sentences

First, enter 안녕하세요, calculate the vector, and save it as JSON. Next, analyze 안녕하세요. 반갑습니다. to see if it has the same fragment and ID. Even if there is the same ID, the context vector may be different if the sentence and location are different.

This comparison looks at the name and ID of the fragment together with the input location. If two sentences do not have the same fragment, they are comparing different tokens. To compare semantic similarity across sentences, you need to construct separate sentence embeddings, and this tool is used to observe vectors for each position.

Execution scope and usage environment

• Input can be up to 500 characters (UTF-16). Some emojis count as two characters even though they appear as one character.
• The tokenization result displays the entire input. Vector calculations are limited to 128 tokens or less, including special tokens, in line with the model’s sentence practice range. Excess sentences are secretly trimmed and not calculated.
• You will need a browser that supports Web Worker and WebAssembly. The speed and feasibility of calculations will vary depending on the visitor’s device memory and browser environment.
• Program and model files are provided on this site. You can reuse it as long as your browser cache is maintained, and you can download it again after going into private mode or clearing your storage space.

This exercise is based on Transformers.js 3.8.1 and the Xenova transformation model’s fixed revision 2c4055b. For browser model execution, refer to the local file setting Transformers.js, , and for token ID processing, refer to tokenizer API document . The effective date is October 22, 2026.