Topic outline

  • Collapse Expand

    General

    Collapse all Expand all

    • Compute Ontario Colloquium (26 Apr 2023)

      This week's colloquium: "Parallel Job Orchestration with GNU Parallel" by Ramses van Zon (SciNet).  The Compute Ontario Colloquia are weekly Zoom presentations on Advanced Research Computing, High Performance Computing, Research Data Management, and Research Software topics, delivered by staff from three Compute Ontario consortia (CAC, SciNet, SHARCNET) and guest speakers. The colloquia are one hour long and include time for questions. No registration is required. 

      Teacher: Ramses van Zon
      Date: Wed., 26 Apr. 2023 - 12:00 pm
      Events:

    • Abstract

      Many research computing projects involve workloads of large amounts of relatively short, independent computations.  Such workloads are a common reason for researchers to move to shared resources.  Despite being called an "embarrasingly parallel" problem, orchestrating large amounts of small computational jobs is surprisingly subtle, particularly when dealing with shared resources with their own usage policies and implementations.  To deal with this, many tools have been created, used for a while, and then abandoned. Here, we will focus on a tool for job orchestration that is very versatile and actively maintained, GNU Parallel, and show how it helps for a number of use cases.

    • Recording (YouTube) URL
    • Compute Ontario Colloquium Zoom Information Page
    • Presentation Slides File
    • Announcements Forum
  • Collapse Expand

    Video Channels of the Ontario Consortia