TextForge
The three text jobs every scraping and RAG agent has to do, as one keyless JSON API: strip HTML to clean markdown, split text into token-bounded chunks with overlap, and count tokens exactly.
Try it
Endpoints
| Method | Path | Body | Returns |
|---|---|---|---|
| POST | /extract | {"html": "..."} | markdown, title, tokens, reduction_pct |
| POST | /chunk | {"text": "...", "max_tokens": 512, "overlap_tokens": 64} | chunks[] with per-chunk token counts |
| POST | /tokens | {"text": "..."} | tokens, chars, words |
| GET | /health | — | liveness + tokenizer in use |
OpenAPI schema at openapi.json, interactive docs at /docs.
Why it exists
Chunking naively on character count splits sentences mid-word and blows past model context windows because
characters are not tokens. TextForge splits on paragraph, then sentence, then token boundaries, and never emits a
chunk over your max_tokens. Overlap is measured in tokens too, so retrieval context stays intact.
/extract drops script, style, nav, footer,
form and aside before converting, which typically cuts 85–95% of a scraped page's
bytes before it ever reaches your model.