Slurm Workload Manager - Release Notes

# Release Notes

The following are the contents of the RELEASE_NOTES file as distributed with the Slurm source code for this release. Please refer to the NEWS include alongside the source as well for more detailed descriptions of the associated changes, and for bugs fixed within each maintenance release.

    RELEASE NOTES FOR SLURM VERSION 20.11

    IMPORTANT NOTES:
    If using the slurmdbd (Slurm DataBase Daemon) you must update this first.

    NOTE: If using a backup DBD you must start the primary first to do any
    database conversion, the backup will not start until this has happened.

    The 20.11 slurmdbd will work with Slurm daemons of version 19.05 and above.
    You will not need to update all clusters at the same time, but it is very
    important to update slurmdbd first and having it running before updating
    any other clusters making use of it.

    Slurm can be upgraded from version 19.05 or 20.02 to version 20.11 without loss
    of jobs or other state information. Upgrading directly from an earlier version
    of Slurm will result in loss of state information.

    If using SPANK plugins that use the Slurm APIs, they should be recompiled when
    upgrading Slurm to a new major release.

    NOTE: Slurmctld is now set to fatal in case of computing node configured with
          CPUs == #Sockets. CPUs has to be either total number of cores or threads.

    NOTE: The FastSchedule option has been removed. The FastSchedule=2 functionality
          (used for testing and development) is available as the new
          SlurmdParameters=config_overrides option.

    NOTE: Slurmdbd is now set to fatal if slurmdbd.conf file isn't owned by
          SlurmUser or it's mode is not set to 0600.

    NOTE: In the database the AllocGres and ReqGres columns have been removed as
          they are duplicates for those using AccountingStorageTRES.  If you use
          these columns please make sure to backup this information since it will
          be lost and set up AccountingStorageTRES with your GRES to be able to
          get equivalent information out of AllocTRES and ReqTRES after upgrade.

    NOTE: PMIx v1.1.4 and below are no longer supported.

    HIGHLIGHTS
    ==========
     -- The example systemd unit files have been changed to the "simple" type of
        operation, and the daemon will now run in the foreground within systemd
        instead of daemonizing itself.
     -- Log messages enabled by the various DebugFlags have been overhauled, and
        will all print at the verbose() level, and prepend the flag name that is
        associated with a given log message.
     -- A separate unversioned libslurm_pmi.so will be installed, and the libpmi.so
        that Slurm can (optionally) install will link to that rather than libslurm.
        This should resolve long-standing issues when building static OpenMPI
        libraries and later updating your Slurm release, thereby breaking the
        embedded libslurm.so. link in those OpenMPI libraries that were
        inherited from libpmi.so.
     -- accounting_storage/filetxt has been removed as an option.  Please consider
        using accounting_storage/slurmdbd as an alternative.
     -- setting of number of Sockets per node was standardized for configuration
        line with and without Boards=. Specifically in case of Boards=1 and #CPUs
        given the default value of Sockets will be set to #CPUs / #Cores / #Threads.
     -- Dynamic Future Nodes - slurmds started with -F[] will be
        associated with a nodename in Slurm that matches the same hardware
        configuration.
     -- SlurmctldParameters=cloud_reg_addrsa - Cloud nodes automatically get
        NodeAddr and NodeHostname set from slurmd registration.
     -- SlurmctldParameters=power_save[_min]_interval - Configure how often the
        power save module looks to do work.
     -- By default, a step started with srun will be granted exclusive (or non-
        overlapping) access to the resources assigned to that step. No other
        parallel step will be allowed to run on the same resources at the same
        time. This replaces one facet of the '--exclusive' option's behavior, but
        does not imply the '--exact' option described below. To get the previous
        default behavior - which allowed parallel steps to share all resources -
        use the new srun '--overlap' option.
     -- In conjunction to this non-overlapping step allocation behavior being the
        new default, there is an additional new option for step management
        '--exact', which will allow a step access to only those resources requested
        by the step. This is the second half of the '--exclusive' behavior.
        Otherwise, by default all non-gres resources on each node in the allocation
        will be used by the step, making it so no other parallel step will have
        access to those resources unless both steps have specified '--overlap'.
     -- --threads-per-core now influences task layout/binding, not just allocation.
     -- AutoDetect in gres.conf can now be specified for some nodes while not for
        others via the NodeName option.
     -- gres.conf - Add new MultipleFiles configuration entry to allow a single
        GRES to manage multiple device files simultaneously.
     -- Remove SallocDefaultCommand option.
     -- Add support for an "Interactive Step", designed to be used with salloc to
        launch a terminal on an allocated compute node automatically. Enable by
        setting "use_interactive_step" as part of LaunchParameters.
     -- Add IPv6 support. Must be explicitly enabled with EnableIPv6 in
        CommunicationParameters. IPv4 support can be disabled with DisableIPv4.
     -- Allow use of a target directory with "srun --bcast", and change the default
        filename to include the node name as well.
     -- Added a new --mail-type=INVALID_DEPEND option to salloc, sbatch, and srun.
     -- Differences between hardware (memory size, number of CPUs) discovered on
        node vs configured in slurm.conf will now throw an error only when the node
        state is set to drain. Previously it was done on every node registration,
        those messages were demoted to debug level.
     -- Added "scrontab", which permits crontab-compatible job scripts to be
        defined. These scripts will recurr automatically (at most) on the intervals
        described.
     -- Enable -lnodes=#:gpus=# in #PBS/qsub -l nodes syntax.
     -- Any user >= Operator can see any hidden partition by default, as SlurmUser
        or root already did.
     -- select/linear will now allocate up to nodes RealMemory when configured with
        SelectTypeParameters=CR_Memory and --mem=0 specified. Previous behavior was
        no memory accouted and no memory limits implied to job.
     -- slurmrestd - add API to interface with slurmdbd.
     -- Add --ntasks-per-gpu option.
     -- Add --gpu-bind=single option.
     -- Fix "scontrol takeover [backup]" hangs when specifying a backup > 1. All
        slurmctlds below the "backup" will be shutdown.
     -- The names of the functions in the cli_filter plugin are now prepended with
        the string "cli_filter_p_". For example, the function setup_defaults()
        was changed to cli_filter_p_setup_defaults().

    CONFIGURATION FILE CHANGES (see man appropriate man page for details)
    =====================================================================
     -- Removed "cpusets" option from TaskPluginParam. Please use task/cgroup.
     -- Removed MsgAggregationParams.
     -- Removed Layouts.
     -- Remove switch/generic plugin.
     -- The acct_gather_energy/cray_aries plugin has been renamed to
        acct_gather_energy/pm_counters.
     -- The JobCompLoc URL endpoint when the JobCompType=jobcomp/elasticsearch
        plugin is enabled is now fully configurable and the plugin no longer appends
        a hardcoded "/slurm/jobcomp" index and type suffix to it.
     -- Removed support for "default_gbytes" option in SchedulerParameters.

    COMMAND CHANGES (see man pages for details)
    ===========================================
     -- Make sacct get the UID from database instead of from the username and a
        system call. Add --use-local-uid option to sacct to use old behavior.
     -- The '%s' format in -e/-i/-o options to sbatch will expand to "batch" rather
        than "4294967294".
     -- squeue - added "pendingtime" as a option for --Format.
     -- sacct - AllocGres and ReqGres were removed. Alloc/ReqTres should be used
        instead.
     -- scontrol - added the "Reserved" license count to 'scontrol show licenses'.
     -- Add time specification: "now-" (i.e. subtract from the present)
     -- squeue - put sorted start times of "N/A" or 0 at the end of the list.
     -- Change "scontrol reboot ASAP" to use next_state=resume logic.
     -- scontrol - added an admin-settable "Comment" field to each Node.
     -- squeue and sinfo -O no longer repeat the last suffix specified.
     -- salloc now waits for PrologSlurmctld to finish before entering the shell.

    API CHANGES
    ===========
     -- slurm_ctl_conf_t has been renamed to slurm_conf_t.
     -- slurm_free_kvs_comm_set() has been renamed to slurm_pmi_free_kvs_comm_set(),
        slurm_get_kvs_comm_set() has been renamed to slurm_pmi_get_kvs_comm_set().
     -- slurm_job_step_layout_get() parameters has changed to use slurm_step_id_t
        see slurm.h for new implementation.  If not running hetsteps just put
        NO_VAL as the value for step_het_comp.
     -- slurm_job_step_stat() parameters has changed to use slurm_step_id_t
        see slurm.h for new implementation.  If not running hetsteps just put
        NO_VAL as the value for step_het_comp.
     -- slurm_job_step_get_pids() parameters has changed to use slurm_step_id_t
        see slurm.h for new implementation.  If not running hetsteps just put
        NO_VAL as the value for step_het_comp.
     -- slurmdb_selected_step_t has been renamed slurm_selected_step_t.
     -- slurm_sbcast_lookup() arguments have changed.  It now takes a populated
        slurm_selected_step_t instead of job_id, het_job_offset, step_id.
     -- Due to internal restructuring ahead of the 20.11 release, applications
        calling libslurm MUST call slurm_init(NULL) before any API calls.
        Otherwise the API call is likely to fail due to libslurm's internal
        configuration not being available.