SDGi Text Classifier

Distributed multilingual policy document preprocessing and Sustainable Development Goal (SDG) topic classification with Python, Hadoop, HDFS, Spark RDDs, and Spark MLlib.

Overview

This project builds an end-to-end data engineering and machine learning pipeline for classifying English and French policy documents into SDG-aligned topic labels.

The system ingests institutional PDFs from the World Bank and United Nations, extracts text, performs language-aware preprocessing, stages the cleaned corpus in HDFS, runs Spark RDD word count and TF-IDF analysis, trains Spark MLlib classifiers on the labelled SDGi corpus, and applies the best model to 2,046 World Bank and UN documents.

The strongest model was an NGram Logistic Regression pipeline, which reached 0.8002 accuracy and 0.8074 F1 on the labelled SDGi test set.

Tech Stack

Layer	Tools
Ingestion	Python, requests, PyMuPDF
Text preprocessing	Unicode-aware regex, spaCy, English/French stopword handling
Distributed storage	Hadoop, HDFS, NameNode, DataNodes
Distributed compute	Spark RDDs, YARN workers, Docker Compose
Machine learning	Spark MLlib, TF-IDF, Naive Bayes, Logistic Regression, NGram Logistic Regression
Evaluation	Accuracy, weighted precision, weighted recall, F1, confusion-matrix-style counts, manual review

Problem

Policy organizations publish large volumes of long, multilingual PDF documents. These files contain useful information about poverty, health, education, climate, energy, governance, inequality, and partnerships, but they are difficult to classify manually at scale.

This project asks:

How can a distributed Hadoop-Spark pipeline preprocess multilingual World Bank and United Nations policy documents and classify them into SDG-aligned topic labels using Spark MLlib?

Data Sources

The project separates labelled training data from unlabelled inference data.

Source	Role	Use
SDGi Corpus	Labelled benchmark corpus	Train and evaluate SDG classifiers
World Bank Documents & Reports	Unlabelled institutional corpus	Classify after training
UN Digital Library	Unlabelled institutional corpus	Classify after training

Final inference corpus:

Source	Language	Preprocessed documents
World Bank	English	810
World Bank	French	501
United Nations	English	487
United Nations	French	248
Total	English/French	2,046

The World Bank and UN documents are inference targets, not benchmark labels. Model accuracy, precision, recall, and F1 are reported on the labelled SDGi test split.

Pipeline

1. Search World Bank and UN metadata using broad SDG-style queries.
2. Download matching PDFs and extract visible text with PyMuPDF.
3. Keep only documents with at least 1,000 extracted words.
4. Clean PDF artifacts, URLs, emails, page headers, document codes, layout noise, punctuation, and numbers.
5. Normalize Unicode while preserving French accents and meaningful apostrophes/hyphens.
6. Tokenize with a bilingual-safe regex so words such as developpement, l'electricite, and accented French terms are handled consistently.
7. Remove English and French stopwords separately.
8. Lemmatize with spaCy English and French models.
9. Upload the final preprocessed corpus to HDFS under /corpus/preprocessed.
10. Run Spark RDD word count, term frequency, and TF-IDF analysis.
11. Train Spark MLlib classifiers on SDGi labelled examples.
12. Apply the best model to the 2,046-document World Bank and UN corpus.
13. Manually review a balanced 100-document sample to interpret transfer performance.

Distributed Architecture

The final corpus is staged in HDFS:

hdfs://namenode:8020/corpus/preprocessed/*/language=*/*.txt

The Hadoop side uses:

Component	Role
NameNode	HDFS metadata
2 DataNodes	HDFS storage workers
ResourceManager	YARN scheduling
2 NodeManagers	YARN compute workers

The Spark side uses spark-master, spark-worker, and spark-client. Spark reads the HDFS corpus after the Spark containers are connected to the Hadoop Docker network and can resolve the namenode hostname.

MapReduce-Style Analysis

Spark RDDs implement the MapReduce-style analysis:

lines = sc.textFile(HDFS_PREPROCESSED)

word_counts = (
    lines.flatMap(lambda line: line.split())
         .map(lambda word: (word, 1))
         .reduceByKey(lambda a, b: a + b)
)

Top full-corpus terms included:

Term	Count
poverty	97,141
national	89,235
service	88,202
public	77,501
plan	72,673
country	71,190
social	69,466
development	67,282
project	66,515
energy	57,362

TF-IDF analysis was also used to identify more document-specific policy terms. A diagnosis-aware filter reduced OCR/PDF noise by requiring alphabetic tokens, minimum token length, vowels, known noise-term removal, and minimum document frequency.

MLlib Classification

The best-performing pipeline combines unigram and bigram TF-IDF features:

text -> RegexTokenizer -> NGram -> CountVectorizer -> IDF -> VectorAssembler -> LogisticRegression

Key settings:

Component	Setting
Unigram vocabulary	50,000
Bigram vocabulary	30,000
Minimum document frequency	2
Logistic Regression iterations	20

Model	Accuracy	F1
Naive Bayes	0.6913	0.6917
Logistic Regression	0.7282	0.7370
NGram Logistic Regression	0.8002	0.8074

Additional final metrics for the NGram Logistic Regression model:

Metric	Value
Accuracy	0.8002
Weighted precision	0.8316
Weighted recall	0.8002
F1	0.8074

Full-Corpus Inference

After training, the best saved Spark MLlib model was applied to all 2,046 preprocessed World Bank and UN documents in HDFS. Each output row contains a document ID, source path, predicted SDG, readable SDG name, and confidence score.

Example predictions:

Document	Source/language	Prediction	Confidence
32430130.txt	World Bank EN	SDG 12: Responsible Consumption and Production	0.9663
34045382.txt	World Bank FR	SDG 7: Affordable and Clean Energy	0.9999
D1000517.txt	World Bank EN	SDG 17: Partnerships for the Goals	0.9999
D10099733.txt	World Bank FR	SDG 6: Clean Water and Sanitation	0.9999

The largest predicted categories were:

Predicted SDG	Documents
SDG 4: Quality Education	364
SDG 17: Partnerships for the Goals	240
SDG 10: Reduced Inequalities	187
SDG 1: No Poverty	183
SDG 7: Affordable and Clean Energy	183
SDG 8: Decent Work and Economic Growth	142

These predictions should be interpreted as model-based labels, not official ground truth. The labelled SDGi test set is used for benchmark evaluation; the World Bank and UN corpus is used for inference.

Manual Review

Because the World Bank and UN corpus does not have official SDG labels in this project, I manually reviewed a balanced 100-document sample:

Sample group	Documents
World Bank English	25
UN English	25
World Bank French	25
UN French	25

Manual review result:

Judgment	Count	Meaning
Reasonable	56	Prediction matches the main visible topic
Partial / ambiguous	22	Prediction is defensible, but another SDG may be more central
Questionable	22	Prediction does not match well, or text is too noisy/procedural

Strict reasonable match rate: 56/100.

Broadly defensible rate, counting reasonable plus partial/ambiguous predictions: 78/100.

This supports using the classifier as a scalable first-pass SDG screening tool, with human review for ambiguous, noisy, or high-stakes labels.

Repository Structure

.
|-- README.md
|-- requirements.txt
|-- src/
|   |-- ingest_world_bank.py
|   |-- ingest_un.py
|   |-- ingest_eu.py
|   |-- TextPreprocessing.py
|   |-- map_reduce.py
|   `-- MLlib.py
|-- docs/
|   `-- assets/
|       |-- pipeline-overview.svg
|       |-- corpus-composition.svg
|       |-- model-comparison.svg
|       |-- sdg-distribution.svg
|       `-- manual-review-summary.svg
`-- Report and presentation/
    |-- SDGi_Text_Classifier_Report_Final_Vincent.pdf
    `-- SDGi_Text_Classifier_Presentation_SDG_Theme  -  Repaired.pptx

How to Run

Install Python dependencies:

python3 -m pip install -r requirements.txt
python3 -m spacy download en_core_web_sm
python3 -m spacy download fr_core_news_sm

Run ingestion from the project root:

python3 src/ingest_world_bank.py
python3 src/ingest_un.py

Run preprocessing:

python3 src/TextPreprocessing.py

Check the final preprocessed corpus count:

find data/source=*/preprocessed/language=* -name "*.txt" | wc -l

The expected full-corpus count for the final run is 2046.

For the distributed steps, start the Hadoop/HDFS containers, upload the preprocessed corpus into /corpus/preprocessed, start the Spark containers, connect Spark to the Hadoop Docker network, and run the commands documented in:

• src/map_reduce.py
• src/MLlib.py

Those files include the HDFS upload checks, PySpark shell settings, RDD word count, TF-IDF computation, MLlib model training, model evaluation, and full-corpus inference commands used for the final report.

Limitations

• PDF extraction and OCR noise can still affect some documents.
• SDG topics naturally overlap, especially for policy documents about climate, energy, poverty, inequality, finance, and institutions.
• The SDGi benchmark and the World Bank/UN corpus use different institutional language, so transfer predictions require caution.
• The World Bank and UN predictions are model-generated labels, not official human-verified labels.
• High confidence means strong model preference, not guaranteed correctness.

Future Work

• Add calibrated confidence thresholds for automatic review routing.
• Expand from single-label classification to true multi-label SDG classification.
• Add more robust OCR and table cleanup for difficult PDFs.
• Evaluate on a larger manually labelled World Bank/UN validation set.
• Export predictions to a searchable dashboard or data catalog.

Project Artifacts

• Final report
• Presentation deck
• MapReduce terminal evidence
• MLlib terminal evidence

References

• Skrynnyk, M., Disassa, G., Krachkov, A., & DeVera, J. (2024). SDGi Corpus: A Comprehensive Multilingual Dataset for Text Classification by Sustainable Development Goals.
• Apache Spark documentation for Spark RDDs and Spark MLlib.
• World Bank Documents & Reports API.
• United Nations Digital Library.