Hinglish NER: Fine-Tuning vs In-Context Learning

A comprehensive empirical study comparing fine-tuned multilingual transformers (XLM-RoBERTa, mBERT) against zero-shot LLMs (GPT-4o, LLaMA 3.3 70B, LLaMA 3.1 8B) for Named Entity Recognition on Hinglish (Hindi-English code-mixed) text.

Full paper: nlp_final_project.pdf

Abstract

We fine-tune XLM-RoBERTa (110M params) and mBERT (170M params) on the full COMI-LINGUA Hinglish NER training set (12,151 examples, 8 entity types) and evaluate on all 4,829 test examples. We then compare against three LLMs under zero-shot prompting — GPT-4o, LLaMA 3.3 70B, and LLaMA 3.1 8B (via Groq API) — on a stratified 100-example subset of the same test set (subset used due to API cost/latency/rate-limit constraints). On this subset, LLaMA 3.1 8B reaches 95.7% accuracy (0.97 weighted F1) and LLaMA 3.3 70B reaches 94.8% accuracy (0.95 macro F1), both ahead of fine-tuned XLM-RoBERTa (89.7%) and mBERT (89.0%), while GPT-4o trails badly at 40.4% accuracy due to extreme under-prediction. The paper includes error analysis, confusion matrices, and dev-fold leakage checks.

Motivation

Hinglish (Hindi-English code-mixed, largely Roman-script) is common on Indian social media, but inconsistent transliteration, mid-sentence code-switching, and informal abbreviations break NER systems trained on formal monolingual text. This project asks: do we still need task-specific fine-tuning for Hinglish NER, or can zero-shot LLM prompting match it?

Dataset

COMI-LINGUA Hinglish NER subset — flat labeling scheme (no BIO), 8 entity types: O, PER, ORG, LOC, DATE, TIME, HASHTAG, MENTION.

Split	Examples	Tokens
Train	12,151	48,145
Validation	1,350	5,291
Test	4,829	25,246
Total	18,330	78,682

Model selection used 5-fold cross-validation on the training portion (not the provided validation split) to avoid leakage.

Results

Overall (Table 4 in paper)

Model	Accuracy	Macro F1	Weighted F1	Eval set
Majority class (all-O)	0.320	–	–	full test
XLM-RoBERTa (fine-tuned)	0.897	0.908	0.897	full test (4,829)
mBERT (fine-tuned)	0.890	0.894	0.890	full test (4,829)
LLaMA 3.1 8B (zero-shot)	0.957	0.681	0.967	stratified sample (100)
LLaMA 3.3 70B (zero-shot)	0.948	0.951	0.948	stratified sample (100)
GPT-4o (zero-shot)	0.404	0.325	0.350	stratified sample (100)

The LLaMA models' lower macro F1 is an artifact of the 100-example slice, in which the rare classes (TIME, HASHTAG, MENTION) never appear, which zeroes out their per-class F1 and drags the macro average down despite strong performance on the frequent classes.

Important caveat: fine-tuned models are evaluated on the full 4,829-example test set; LLMs are evaluated on a stratified 100-example subset only, due to API cost/latency/rate-limit constraints. These numbers are not on equal footing — see Limitations.

Per-entity F1 (Table 5 in paper)

Entity	XLM-R (full test)	mBERT (full test)	LLaMA 8B (100-ex)	GPT-4o (100-ex)
O	0.89	0.88	0.97	0.49
PER	0.95	0.94	0.99	0.21
ORG	0.86	0.87	0.97	0.27
LOC	0.87	0.85	0.91	0.28
DATE	0.90	0.91	0.93	0.69
TIME	0.82	0.87	0.00*	0.00*
HASHTAG	0.99	0.97	0.00*	0.00*
MENTION	0.96	0.94	0.00*	0.00*

* Rare classes had zero support in the 100-example LLM sample — not a true model failure, see caveat above.

Training cost (Table 17)

Model	Train time (CPU)	Params	Checkpoint size
XLM-RoBERTa-base	1.5 h	110M	440 MB
mBERT	3.0 h	170M	680 MB

Full per-class precision/recall/support tables, confusion matrices, and GPT-4o failure analysis are in the paper (Appendix A, Section 6).

Reproduce

git clone https://github.com/harshagarwalnyu/Hinglish-Ner-Empirical-Study.git
cd Hinglish-Ner-Empirical-Study
make install                 # pip install -r requirements.txt

1. Prepare data — place raw train.jsonl / dev.jsonl / test.jsonl (or a single JSONL) under data/raw/, then:

   make prep                  # src/data_prep.py --input data/raw --output data/processed --model xlm-roberta-base --add_script_id true

2. Fine-tune:

   make train-xlmr              # python -m src.train_flat_ner --model_name xlm-roberta-base --output_dir outputs/xlmr-flat
   make train-mbert             # python -m src.train_flat_ner --model_name bert-base-multilingual-cased --output_dir outputs/mbert-flat

3. Predict:

   make predict-test            # python -m src.predict_flat_ner --model_dir outputs/xlmr-flat --split test --out_path preds/xlmr_flat_test.jsonl

4. Evaluate / error analysis:

   make eval-test                # python -m src.eval_flat --pred_path preds/test_preds.jsonl --out_dir eval/test-metrics
   make errors                   # python -m src.error_slices --pred_path preds/test_preds.jsonl --top_k 30
   make stats                    # python scripts/label_stats.py data/processed

5. Run tests:

   make test                     # python -m pytest -q

LLM zero-shot evaluation (GPT-4o, LLaMA via Groq) used the prompting setup described in Section 4.2.3 of the paper; those calls were made manually/via API against the sampled subset and are not part of this repo's automated pipeline.

Repository layout

src/
  data_prep.py          # raw JSONL -> HF dataset, tokenization, BIO alignment, transliteration
  train_flat_ner.py      # fine-tunes XLM-R / mBERT for flat (non-BIO) token classification
  predict_flat_ner.py    # generates predictions on a split
  eval_flat.py            # metrics + confusion matrices
  error_slices.py         # top confusion pairs / error slicing
  utils.py
scripts/
  bootstrap_from_hf.py, label_report.py, label_stats.py, peek_entity_dense.py
  remote_deploy.sh, train_sbatch.sh   # optional remote/cluster helpers
tests/
  test_roman_filter.py, test_alignment.py, smoke_processed.py

Limitations

• LLM evaluation covers only 100 examples (API cost/latency/rate limits) vs. the full 4,829-example test set for fine-tuned models — not a like-for-like comparison.
• Single dataset (COMI-LINGUA); generalization to other Hinglish corpora is untested.
• Fine-tuning was CPU-only; GPU training may shift results.
• No LLM fine-tuning baseline was run (proprietary APIs don't expose it here).

See Section 8 of the paper for the full discussion, and Section 7.4 for broader-impact caveats — the headline "LLaMA beats fine-tuned models" result is reported on the sampled subset, and the paper is explicit that full-test-set LLM evaluation is future work.

Citations

If referencing this work, see the paper's own related-work citations (Conneau et al. 2020 — XLM-R; Devlin et al. 2019 — BERT; Singh et al. 2018 — Hindi-English code-mixed NER dataset; Solorio et al. 2014 — code-switching shared task; Wang et al. 2023 — GPT-NER) — full reference list in nlp_final_project.pdf.

Authors

Akshith Karthik, Ashmit Mukherjee, Harsh Agarwal, Lovnish Julka — NYU / NYU Abu Dhabi.

License

MIT — see LICENSE.