Preprocessing Utils

This repo contains python scripts for preprocessing of (Korean) NLP. Also, it contains aligner to build parallel corpora.

Name		Name	Last commit message	Last commit date
Latest commit History 33 Commits
data		data
.gitignore		.gitignore
README.md		README.md
align.py		align.py
align.ted.sh		align.ted.sh
build_dict.sh		build_dict.sh
combine_line.py		combine_line.py
detokenizer.py		detokenizer.py
line_separator.py		line_separator.py
post_tokenize.py		post_tokenize.py
refine.py		refine.py
regex.txt		regex.txt
run_all_joongang.py		run_all_joongang.py
run_all_ted.sh		run_all_ted.sh
run_preprocessing.sh		run_preprocessing.sh
run_preprocessing_clien.sh		run_preprocessing_clien.sh
setup.WSL.sh		setup.WSL.sh
tokenizer.py		tokenizer.py
word_mt.py		word_mt.py

Provide feedback