bmel:truncateToTokens¶
Category: LLM & AI Observability · Returns: bmel:string
bmel:truncateToTokens(text: bmel:string, maxTokens: bmel:integer, tokenizer: bmel:string)
Description¶
Truncates a string to at most maxTokens tokens using the specified tokenizer, cutting at a word boundary where possible. Useful for capping prompt or context size before sending to an LLM, and for enforcing context-window budgets in telemetry frames. Uses the same tokenizer identifiers as bmel:tokenCount.
Arguments¶
| Parameter | Type | Required | Description |
|---|---|---|---|
text | bmel:string | ✅ | The input text to truncate. |
maxTokens | bmel:integer | ✅ | Maximum number of tokens to keep. |
tokenizer | bmel:string | ✅ | Tokenizer identifier. Supported values: 'cl100k_base', 'o200k_base', 'p50k_base', 'voyage', 'gemini', 'llama3', 'mistral'. |
Example¶
bmel:truncateToTokens({createChat:Request Payload}.$.message, 4096, 'cl100k_base')