Center for Tamil Natural Language Processing Research

⚡️𝗙𝗶𝗻𝗲-𝗧𝘂𝗻𝗶𝗻𝗴 𝗜𝗻𝗱𝗶𝗰𝗡𝗘𝗥 𝗳𝗼𝗿 𝗦𝗿𝗶 𝗟𝗮𝗻𝗸𝗮𝗻 𝗧𝗮𝗺𝗶𝗹 𝗡𝗮𝗺𝗲𝗱 𝗘𝗻𝘁𝗶𝘁𝘆 𝗥𝗲𝗰𝗼𝗴𝗻𝗶𝘁𝗶𝗼𝗻

Transformer-based multilingual NLP systems have significantly improved Named Entity Recognition (NER) across many languages. However, low-resource language variants such as Sri Lankan Tamil still face substantial challenges due to limited domain-specific datasets and linguistic underrepresentation.

At CTNLPR, we fine-tuned 𝗮𝗶𝟰𝗯𝗵𝗮𝗿𝗮𝘁/𝗜𝗻𝗱𝗶𝗰𝗡𝗘𝗥 specifically for Sri Lankan Tamil using a custom annotated NER corpus.

𝗢𝗯𝗷𝗲𝗰𝘁𝗶𝘃𝗲

Improve entity recognition for:

• Sri Lankan Tamil linguistic patterns
• Local person, location, and organization names
• Morphology-aware contextual variations

𝗪𝗵𝘆 𝗦𝗿𝗶 𝗟𝗮𝗻𝗸𝗮𝗻 𝗧𝗮𝗺𝗶𝗹 𝗡𝗘𝗥 𝗶𝘀 𝗖𝗵𝗮𝗹𝗹𝗲𝗻𝗴𝗶𝗻𝗴

Most multilingual NER systems are trained primarily on:

• General web corpora
• Indian Tamil datasets
• Multilingual benchmark datasets
• Formal textual sources

When applied to Sri Lankan Tamil, they often struggle with:

• Regional naming conventions
• Local organization terminology
• Morphological suffix complexity
• OCR-induced token inconsistencies
• Subword tokenization fragmentation
• Ambiguous entity boundaries

These limitations directly affect downstream systems such as:

• Semantic Search
• Document Intelligence
• Knowledge Graph Construction
• Tamil Chatbots
• RAG Systems
• Government Document Processing

𝗠𝗼𝗱𝗲𝗹 𝗙𝗶𝗻𝗲-𝗧𝘂𝗻𝗶𝗻𝗴 𝗢𝘃𝗲𝗿𝘃𝗶𝗲𝘄

𝗕𝗮𝘀𝗲 𝗠𝗼𝗱𝗲𝗹

→ ai4bharat/IndicNER

𝗘𝗻𝘁𝗶𝘁𝘆 𝗧𝘆𝗽𝗲𝘀

• PERSON
• LOCATION
• ORGANIZATION

𝗞𝗲𝘆 𝗢𝗽𝘁𝗶𝗺𝗶𝘇𝗮𝘁𝗶𝗼𝗻𝘀

✅ Tamil-safe Tokenization
✅ Unicode Normalization
✅ BIO Tagging
✅ Proper Subword Label Alignment
✅ Morphology-aware Training
✅ OCR-aware Preprocessing

𝗧𝗲𝗰𝗵𝗻𝗶𝗰𝗮𝗹 𝗖𝗵𝗮𝗹𝗹𝗲𝗻𝗴𝗲𝘀

1️⃣ 𝗧𝗮𝗺𝗶𝗹 𝗧𝗼𝗸𝗲𝗻𝗶𝘇𝗮𝘁𝗶𝗼𝗻

Tamil is morphologically rich. Incorrect tokenization can cause:

• Broken entity spans
• Incorrect BIO labels
• Fragmented predictions

2️⃣ 𝗦𝘂𝗯𝘄𝗼𝗿𝗱 𝗟𝗮𝗯𝗲𝗹 𝗔𝗹𝗶𝗴𝗻𝗺𝗲𝗻𝘁

Transformer tokenizers frequently split Tamil words into multiple subword units.

Without proper alignment:

• Entity spans become corrupted
• BIO labels mismatch
• Training instability increases

3️⃣ 𝗢𝗖𝗥 𝗡𝗼𝗶𝘀𝗲

Tamil OCR systems still generate:

• Grapheme inconsistencies
• Merged tokens
• Invalid Unicode combinations
• Punctuation corruption

Therefore OCR-aware normalization was integrated before training.

𝗠𝗼𝗱𝗲𝗹 𝗘𝘃𝗮𝗹𝘂𝗮𝘁𝗶𝗼𝗻

𝗢𝘃𝗲𝗿𝗮𝗹𝗹 𝗣𝗲𝗿𝗳𝗼𝗿𝗺𝗮𝗻𝗰𝗲

• F1 Score → 0.650
• Precision → 0.602
• Recall → 0.707
• Accuracy → 96.04%

𝗘𝗻𝘁𝗶𝘁𝘆-𝘄𝗶𝘀𝗲 𝗙𝟭

• PERSON → 0.721
• LOCATION → 0.698
• ORGANIZATION → 0.484

PERSON and LOCATION categories achieved relatively strong performance, while ORGANIZATION entities remain the most challenging category.

𝗜𝗻𝗳𝗲𝗿𝗲𝗻𝗰𝗲 𝗘𝘅𝗮𝗺𝗽𝗹𝗲𝘀

𝗘𝘅𝗮𝗺𝗽𝗹𝗲 𝟭

Sentence:
"பாரதிதாசன் எழுதிய நூலை பாரதி பதிப்பகம் வெளியிட்டது."

Output:
👤 PERSON → பாரதிதாசன்
🏢 ORGANIZATION → பாரதி பதிப்பகம்

𝗘𝘅𝗮𝗺𝗽𝗹𝗲 𝟮

Sentence:
"வடமராட்சி தொழில்நுட்ப நிறுவனம் மாணவர்களை சேர்த்தது."

Output:
🏢 ORGANIZATION → வடமராட்சி தொழில்நுட்ப நிறுவனம்

𝗘𝘅𝗮𝗺𝗽𝗹𝗲 𝟯

Sentence:
"நவமணி கிராமம் வெள்ளத்தால் பாதிக்கப்பட்டது."

Output:
📍 LOCATION → நவமணி

𝗘𝘅𝗮𝗺𝗽𝗹𝗲 𝟰

Sentence:
"ஜே.ஏ.எஸ்.பீ. ஜயசிங்க நவமணி கிராமத்திற்கு சென்றார்."

Output:
👤 PERSON → ஜே.ஏ.எஸ்.பீ. ஜயசிங்க
📍 LOCATION → நவமணி

𝗞𝗲𝘆 𝗢𝗯𝘀𝗲𝗿𝘃𝗮𝘁𝗶𝗼𝗻

One of the most important findings from this work is:

"Better preprocessing and domain-specific data can be as important as model architecture."

For low-resource languages like Sri Lankan Tamil:

• High-quality annotations matter
• OCR normalization matters
• Tokenizer alignment matters
• Linguistic preprocessing matters

Large transformer architectures alone are not sufficient without carefully prepared language-specific datasets.

𝗔𝗽𝗽𝗹𝗶𝗰𝗮𝘁𝗶𝗼𝗻𝘀

• Tamil NER Systems
• Semantic Search
• RAG Pipelines
• OCR Information Extraction
• Knowledge Graph Construction
• Tamil Chatbots

This work is part of ongoing Tamil NLP research at CTNLPR aimed at building stronger NLP infrastructure for low-resource Tamil language technologies.

⚡️𝐃𝐞𝐬𝐢𝐠𝐧𝐢𝐧𝐠 𝐚 𝐁𝐢𝐥𝐢𝐧𝐠𝐮𝐚𝐥 𝐑𝐀𝐆 𝐒𝐲𝐬𝐭𝐞𝐦: 𝐂𝐫𝐨𝐬𝐬-𝐋𝐢𝐧𝐠𝐮𝐚𝐥 𝐃𝐞𝐧𝐬𝐞 𝐑𝐞𝐭𝐫𝐢𝐞𝐯𝐚𝐥 𝐟𝐨𝐫 𝐓𝐚𝐦𝐢𝐥–𝐄𝐧𝐠𝐥𝐢𝐬𝐡

In multilingual RAG systems, the key challenge is cross-lingual retrieval — enabling a query in Tamil to retrieve semantically relevant Tamil and English passages from a unified index (and vice versa), without translation pipelines or language-specific partitioning.

⚙️ Core Approach

We rely on multilingual dense encoders that project Tamil and English into a shared semantic vector space, allowing semantically aligned content across languages to be retrieved using standard similarity search.

🔬 Model Evaluation

We evaluated:
• Sentence Transformers (SBERT variants)
• Indic-specific models (IndicBERT, MuRIL)

Observed limitations:
• Weak Tamil–English alignment
• Inconsistent cross-lingual similarity distributions
• Lower recall in mixed-language retrieval

✅ Selected Model

→ intfloat/multilingual-e5-large

Reasons:
• Built on XLM-RoBERTa-large (multilingual pretraining)
• Trained with large-scale contrastive objectives (>1B pairs)
• Fine-tuned on retrieval benchmarks (MS MARCO, Mr.TyDi, MIRACL)
• Instruction-aware embedding (“query:” / “passage:” prefixes)

This results in strong cross-lingual ranking and alignment, especially for low-resource languages.

🧩 Indexing Strategy

We use a unified embedding + single index design:
• Chunk all documents (Tamil + English)
• Encode using the same model
• Store in one vector index
No language-based partitioning.

🔎 Retrieval Flow

1.Encode query (Tamil or English)
2 Perform ANN search (cosine similarity)
3.Retrieve top-k cross-lingual chunks
4.Pass to LLM for response synthesis

📊 Benchmark Signals (MRR / nDCG)

Across multilingual benchmarks and internal evaluations:
• MRR@10 ↑ → better early precision in cross-lingual retrieval
• nDCG@10 ↑ → improved ranking quality for mixed-language queries
• Recall@10 ↑ → higher retrieval coverage (Tamil ↔ English)
• More stable cosine similarity distributions across scripts

These gains are primarily driven by large-scale contrastive training + retrieval-specific fine-tuning.

💡 Key Insight

Cross-lingual RAG is not a database problem —it is an embedding alignment problem solved at training time.

🚀 Outcome

• Stronger cross-lingual ranking (Mean Reciprocal Rank/nDCG improvements)
• No translation overhead
• Single index, reduced system complexity
• Better knowledge coverage across languages

Multilingual retrieval becomes reliable when both languages share the same semantic space.