DAT171

File Management - Packing Small Files

Courses tagged with "DAT171"

Managing large amounts of data can be a challenging task. Processing large numbers of files incur heavy overhead of IO communications. This course explores several options such as using Apptainer Overlay and SQLite to pack and reduce a large number of files to few files, and hence, improving IO performance. Python scripts are used throughout the course.

Format: Virtual

Category: Data Science
Date: Mon, 15 Jan 2024 - 1:00 pm
Data Science Credits: 3

Managing large amounts of data can be a challenging task. Processing large numbers of files incur heavy overhead of IO communications. This course explores several options such as using Apptainer Overlay and SQLite to pack and reduce a large number of files to few files, and hence, improving IO performance. Python scripts are used throughout the course.

Format: On-line (Zoom).

Category: Data Science
Date: Mon, 28 Nov 2022 - 1:00 pm
Data Science Credits: 3