2019-02-27

lazynlp: a library to scrape, clean, de-duplicate webpages to create massive datasets. It has instructions to download Reddit URLs, Gutenberg books, and Wikipedia. Using this library, you should be able to create text datasets >40GB. https://github.com/chiphuyen/lazynlp