NYU High Performance Computing

Spark: Multi-node jobs with Singularity

Initializing search

NYU High Performance Computing

Introduction
Services
HPC Systems
HPC Systems
- Greene
  Greene
  - None
  - Hardware specs
  - Storage specs
  - Getting started
    Getting started
    
    Getting started
    
    AMD Nodes
  - Best practices
  - Software
    
    Software
    
    R packages with renv
    R packages with renv
    
    Renv
    
    RStudio + R + Python, reproducible way
    
    Python packages with virtual environments
    
    Conda environments (Python, R)
    
    Singularity with Miniconda
    
    Open OnDemand (OOD) with Conda/Singularity
    
    LLAMA on HPC
  - System status
- HPC Storage
  HPC Storage
  - Hardware specs
  - Best practices
  - Storage status
  - Data management
    
    Data management
    
    Data transfers
    
    Data transfers
    
    Transferring Cloud Storage data with rclone
    
    Globus
    
    Sharing data on HPC
    
    Research Project Space (RPS)
    
    Squash file system and Singularity
  - Datasets
- Cloud Computing
  Cloud Computing
- Retired clusters
Accessing HPC
Accessing HPC
- Getting and renewing an account
- External collaborators
Training and support
Training and support
- Support
- General HPC Topics
  General HPC Topics
- Tutorials
  Tutorials
- Resources for classes
  Resources for classes
  - JupyterHub
- Grants
- Supported research initiatives
  Supported research initiatives
- Provide feedback for our services
- Resolving common issues
- HPC projects registration - pilot
HPC projects
HPC projects
- Projects registration
About
About
- Cluster stats and usage data
  Cluster stats and usage data
  - Efficiency of user jobs
  - Resource consumption
- Acknowledgement statement
- Events
- Collaborations
- HPC policies
- Announcements and news
- Featured research
  Featured research
- Job opportunities

Spark: Multi-node jobs with Singularity

Previous Spark: Interactive standalone cluster

Next Linux tutorial

Made with Material for MkDocs