huggingface-tokenizers
huggingface-tokenizers（Orchestra-Research/AI-Research-SKILLs）：Fast tokenizers optimized for research and production. Rust-based implementation tokenizes 1GB in <20 seconds. Supports BPE, WordPiece, and Unigram algorithms. Train custom vocabularies, track alignments, handle padding/truncation. Integrates seamlessly with transformers. Use when you need high-performance tokenization or custom tokenizer training.
GitHub 星
13k
评分
—
综合分
41.2863
来源: Orchestra-Research/AI-Research-SKILLs · 02-tokenization/huggingface-tokenizers/SKILL.md
npx skills add Orchestra-Research/AI-Research-SKILLs --skill huggingface-tokenizers
相关技能
| 01 | axolotlOrchestra-Research/AI-Research-SKILLs | 科学 | 13k | 4128.6 |
↗ | Orchestra-Research/AI-Research-SKILLs · 科学 · 4128.6 |
| 02 | neuroskill-bciNousResearch/hermes-agent | 科学 | 255k | 5406.3 |
↗ | NousResearch/hermes-agent · 科学 · 5406.3 |
| 03 | aeonK-Dense-AI/scientific-agent-skills | 科学 | 52k | 4720.1 |
↗ | K-Dense-AI/scientific-agent-skills · 科学 · 4720.1 |
| 04 | anndataK-Dense-AI/scientific-agent-skills | 科学 | 52k | 4720.1 |
↗ | K-Dense-AI/scientific-agent-skills · 科学 · 4720.1 |
| 05 | powerbi-modelinggithub/awesome-copilot | 科学 | 40k | 4603.6 |
↗ | github/awesome-copilot · 科学 · 4603.6 |