Memory mapping & streaming
The ? Datasets library lets you use and process datasets that don't fit in RAM. Learn how it can do this with memory mapping and how to use the streaming feature. This video is part of the Hugging Face course: http://huggingface.co/course Open in colab to run the code samples: https://colab.research.google.com/github/huggingface/notebooks/blob/master/course/videos/memory_mapping_streaming.ipynb Related videos: - Loading a custom dataset — https://youtu.be/HyQgpJTkRdE - Slide and dice a dataset ? — https://youtu.be/tqfSFcPMgOI Don't have a Hugging Face account? Join now: http://huggingface.co/join Have a question? Checkout the forums: https://discuss.huggingface.co/c/course/20 Subscribe to our newsletter: https://huggingface.curated.co/
The ? Datasets library lets you use and process datasets that don't fit in RAM. Learn how it can do this with memory mapping and how to use the streaming feature. This video is part of the Hugging Face course: http://huggingface.co/course Open in colab to run the code samples: https://colab.research.google.com/github/huggingface/notebooks/blob/master/course/videos/memory_mapping_streaming.ipynb Related videos: - Loading a custom dataset — https://youtu.be/HyQgpJTkRdE - Slide and dice a dataset ? — https://youtu.be/tqfSFcPMgOI Don't have a Hugging Face account? Join now: http://huggingface.co/join Have a question? Checkout the forums: https://discuss.huggingface.co/c/course/20 Subscribe to our newsletter: https://huggingface.curated.co/