Coverage for src / harness_utils / tokens / estimator.py: 50%

1"""Token estimation using simple heuristic. 

2 

3Uses chars/4 approximation which is fast and good enough for 

4pruning decisions. Actual token counts from LLM responses are 

5used for overflow detection. 

6""" 

7 

8 

9def estimate_tokens(text: str, chars_per_token: int = 4) -> int: 

10 """Estimate token count for text using simple heuristic. 

11 

12 Uses chars_per_token ratio (default 4) for fast estimation. 

13 This is good enough for pruning decisions. For accurate counts, 

14 use actual LLM response token counts. 

15 

16 Args: 

17 text: Text to estimate tokens for 

18 chars_per_token: Characters per token ratio (default: 4) 

19 

20 Returns: 

21 Estimated token count 

22 """ 

23 return max(0, round(len(text) / chars_per_token))