Download
nonechucks is a library that provides wrappers for PyTorch's datasets, samplers and transforms to allow for dropping unwanted or invalid samples dynamically. What if you have a dataset of 1000s of images, out of which a few dozen images are unreadable because the image files are corrupted? Or what if your dataset is a folder full of scanned PDFs that you have to OCRize, and then run a language detector on the resulting text, because you want only the ones that are in English? Or maybe you have an AlternateIndexSampler, and you want to be able to move to dataset[6] after dataset[4] fails while attempting to load! PyTorch's data processing module expects you to rid your dataset of any unwanted or invalid samples before you feed them into its pipeline, and provides no easy way to define a "fallback policy" in case such samples are encountered during dataset iteration.
Features
- Dealing with bad samples
- Use Transforms as Filters!
- To install nonechucks, simply use pip
- The function of transorms in PyTorch is restricted to modifying samples
- PyTorch's data processing module expects you to rid your dataset of any unwanted or invalid sample
- Datasets, samplers, and transforms to allow for dropping unwanted or invalid samples dynamically
Project Samples
