CorpusCatcher

CorpusCatcher is a corpus collection toolset. It can help you to build language or topic specific corpora from publicly available web resources. This can be very useful for many purposes, especially for data to build spell checkers.

If you are interested in CorpusCatcher, or are working on spell checkers, you might also be interested in Spelt.

Releases can be downloaded from here and (more recent) sources from here.

• README - Contains all you need to know about using the CorpusCatcher tools.

Git: https://github.com/translate/corpuscatcher.git

See the Advanced Topics section in the README for some notes on the code.

• Download page in encoding of page's HTTP header
• Use mime type from HTTP header to handle file appropriately
  • Support .doc and .pdf
• Improve filtering (filter out numbers, repeated punctuation marks)