outline_detection detects where one Tibetan text ends and another begins. It offers two backends:
- Rule-based — pattern rules A–H derived from annotated boundary snippets (yig mgo ༄༅, section mark ༈, closing phrases, Sanskrit blessings, etc.), plus optional page-layout rules I–L for OCR
- mmBERT neural — a finetuned ModernBERT token classifier (
BDRC/Bo-Boundary-mmBert)
Both share the same predict() interface and return (position, confidence, rule) tuples. An annotation pipeline (annotate_text + postprocess_annotations) inserts <b> markers directly into raw text. An optional CRF sequence labeler is also available via outline-detect crf (requires the [crf] extra).
pip install -e .
pip install -e ".[crf]" # optional CRF extras (scikit-learn, sklearn-crfsuite)
# Install directly from GitHub
pip install git+https://github.com/OpenPecha/outline-detection.gitRequires Python 3.9+.
from outline_detection import detect_breakpoints
text = "...རྫོགས་སོ།། ༄༅། །next text..."
# Rule-based (default)
detect_breakpoints(text)
# {"breakpoints": [152, 410, ...]}
# mmBERT neural detector
detect_breakpoints(text, detector="mmbert")
# {"breakpoints": [149, 408, ...]}Options:
detect_breakpoints(text, profile="precision") # recall | balanced | precision (rule only)
detect_breakpoints(text, min_confidence=0.5) # override confidence threshold (rule only)
detect_breakpoints(text, detailed=True) # adds per-boundary confidence + rule label
# mmBERT options
detect_breakpoints(text, detector="mmbert", threshold=0.80)
detect_breakpoints(text, detector="mmbert", model_name_or_path="path/to/checkpoint")When detailed=True the return value gains a "details" key:
{
"breakpoints": [152],
"details": [{"index": 152, "confidence": 0.90, "rule": "A:yig_mgo"}]
}from outline_detection import RuleBasedDetector, MmBertDetector
# Rule-based
det = RuleBasedDetector(profile="balanced") # or profile="recall" / "precision"
boundaries = det.predict(text)
# [(152, 0.90, "A:yig_mgo"), (410, 0.82, "G:sbrul_shad"), ...]
# mmBERT (model loaded lazily on first predict() call)
det = MmBertDetector() # uses BDRC/Bo-Boundary-mmBert
boundaries = det.predict(text)
# [(149, 0.9312, "mmbert"), ...]
positions = det.predict_positions(text) # just offsets: [149, ...]from outline_detection.utils import annotate_text, postprocess_annotations
# Insert <b> markers at boundary positions
annotated = annotate_text(text, boundaries)
# Fix minor placement errors (shifts <b> past trailing shad, before ༄/༈, etc.)
annotated = postprocess_annotations(annotated)strip_boundaries and insert_boundaries are also available for round-tripping annotated corpora.
For OCR output that arrives as a sequence of pages, the optional page-layout rules (I–L) use line density instead of orthographic signals. They are off by default and inert on continuous (single-page) text:
detect_breakpoints(
ocr_text,
rule_i_empty_page=True, # an empty page marks a break
rule_j_sparse_tail=True, # dense page then two sparse pages
line_threshold=4, # T: "few lines" cutoff
page_delimiter="\f", # form feed (default), "blank"/"blankN", or regex
)See docs/rules.md for the full rule set.
| Profile | min_confidence |
merge_window |
Active rules |
|---|---|---|---|
recall |
0.30 | 25 chars | A, B, G, H |
balanced |
0.40 | 20 chars | A, B, G, H |
precision |
0.50 | 15 chars | A, G, H |
Rules C, D, and E are disabled by default (net-negative on the current corpus). Pass them explicitly:
RuleBasedDetector(rule_c=True, rule_d=True)| Rule | Signal | Confidence |
|---|---|---|
| A | Yig mgo opener (༄༅) at junction | 0.90 |
| B | Closing formula + strong structural break | 0.78–0.80 |
| C | Page/volume digit separator (off by default) | 0.40–0.70 |
| D | Opening formula without yig mgo (off by default) | 0.60–0.70 |
| E | Collection title header (off by default) | 0.50 |
| G | Section opener ༈ at junction | 0.45–0.82 |
| H | Sanskrit closing blessing | 0.72 |
| I–L | Page-layout / OCR line-density (off by default) | 0.45–0.75 |
| — | mmBERT neural prediction | model output |
The install provides an outline-detect command.
Detect boundaries (text in → JSON out):
outline-detect detect mytext.txt
# {"breakpoints": [152, 410]}
echo "..." | outline-detect detect - # read from stdin
outline-detect detect --text "རྫོགས་སོ།། ༄༅། །next" --pretty
outline-detect detect mytext.txt -o result.json
# Neural detector
outline-detect detect mytext.txt --detector mmbert
outline-detect detect mytext.txt --detector mmbert --threshold 0.80
outline-detect detect mytext.txt --detector mmbert --model path/to/checkpoint
# Detailed output (confidence + rule per boundary)
outline-detect detect mytext.txt --detailed --prettyAnnotate a raw file with <b> boundary markers:
outline-detect predict data/samples/INPUT.txt --profile balanced
outline-detect predict data/samples/INPUT.txt --detector mmbert
outline-detect predict data/samples/INPUT.txt --detector mmbert --model BDRC/Bo-Boundary-mmBert -o out.txtCRF (requires the [crf] extra):
# Full-corpus train with feature cache and post-train eval
outline-detect crf train data/breakpoints_context_snippets.json \
--save-model --features-cache reports/models/crf_features.pkl \
--eval-file data/breakpoints_context_snippets_unique.json
# Evaluate a saved model
outline-detect crf evaluate data/breakpoints_context_snippets_unique.json \
--model reports/models/boundary_crf.pkl --tolerance 15
outline-detect crf predict data/samples/INPUT.txt --model reports/models/boundary_crf.pklBoundaries in annotated JSON are marked with <b> (also accepts legacy </b>).
annotate_text(text, boundaries) inserts <b> at each predicted boundary position.
postprocess_annotations(text) then applies deterministic nudges:
- Right-shift: moves
<b>past trailing shad (།) clusters and closing parentheses. - Left-shift: moves
<b>before ༄ (yig mgo), ༈ (section opener), and split Tibetan prefix consonants.
Hugging Face Hub:
| Resource | Repo |
|---|---|
| Full snippets (82,560) | ganga4364/tibetan-outline-boundary-snippets-full |
| Unique benchmark (31,591) | ganga4364/tibetan-outline-boundary-snippets-unique |
| CRF full (production) | ganga4364/tibetan-outline-boundary-crf-full |
| CRF unbiased (honest eval) | ganga4364/tibetan-outline-boundary-crf-unbiased |
hf download ganga4364/tibetan-outline-boundary-snippets-unique --repo-type dataset
hf download ganga4364/tibetan-outline-boundary-crf-unbiased boundary_crf.pkl --local-dir ./reports/modelsevaluate, analyze, predict, and crf write under ./reports/ (relative to the working directory; gitignored except .gitkeep):
| Directory | Contents |
|---|---|
reports/evaluations/ |
rule_based_evaluation_*.md |
reports/analysis/ |
boundary_report_*.md / .json |
reports/diagnostics/ |
false_negatives.json |
reports/models/ |
CRF .pkl models |
reports/ |
predicted_boundaries.txt |
| Method | F1 |
|---|---|
| Rule-based (balanced) | 0.601 |
| CRF full | 0.571 |
| CRF unbiased | 0.555 |
Rule-based balanced reaches ~63% precision and ~57.5% recall. Primary active rules: A (yig mgo) and G (༈). See docs/evaluation.md for full comparison and regeneration commands.
- docs/terminology.md — Tibetan signals, markup, metrics
- docs/rules.md — Rules A–H (orthographic) and I–L (page layout)
- docs/workflow.md — Full step-by-step workflow
- docs/huggingface.md — Hub datasets and models
- docs/evaluation.md — Benchmark results
- docs/README.md — Doc index
- CHANGELOG.md — Release notes
├── pyproject.toml
├── src/
│ └── outline_detection/
│ ├── api.py # detect_breakpoints() high-level function
│ ├── cli.py # outline-detect CLI entry point
│ ├── config.py # mmBERT model defaults (MODEL_NAME, threshold, etc.)
│ ├── detector.py # RuleBasedDetector + MmBertDetector
│ ├── page_layout.py # OCR page segmentation for rules I–L
│ ├── crf.py # optional CRF train / evaluate / predict
│ ├── utils.py # annotate_text, postprocess_annotations, evaluation helpers
│ ├── evaluation.py # run_evaluation, run_prediction, reporting
│ └── analyzer.py # boundary pattern analysis
├── data/ # Annotated JSON corpora and samples/
├── docs/ # Static reference
├── scripts/ # Training, comparison, and Hub upload helpers
└── reports/ # Generated outputs (gitignored)
This package was created by Dharmaduta from specifications provided by the Buddhist Digital Resource Center (BDRC) for the BDRC Etext Corpus, with funding from the Khyentse Foundation.