← 返回 anthropic 的题目列表Longest-Match Tokenizer with Unknown Token Merging
类型:online_judge
Problem: Longest-Match Tokenizer
Given a vocabulary and an input string, implement a tokenizer. Scan the string from left to right. At each position, choose the longest vocabulary token that matches the current position.
If no vocabulary token matches, that character is part of an unknown token. Consecutive unknown characters must be merged into one <UNK:...> token.
Input
The first line contains integer V.
The next V lines each contain one token.
The last line contains the input string s.
Output
Print the tokenization result, separated by single spaces. Unknown spans should be printed as <UNK:span>.
Constraints
1 <= V <= 10^5
total vocabulary token length at most 10^6
1 <= len(s) <= 10^5
tokens and input may contain letters, digits, and common punctuation; sample inputs do not contain whitespace tokens
Example
Input
5
hello
hell
world
##s
!
helloworld!
Output
hello world !