mercury 2.2.0rc1

Summary

This version brings bug fixes and updates to our v2.0.0 release.

New features

• [NA OFI]
  • Choose addr format dynamically based on user preferences
  • Add support for IPv6
  • Add support for FI_SOCKADDR_IB
  • Add support for HPE cxi provider,
    init info format for cxi is:
    • NIC:PID (both or only one may be passed), NIC is cxi[0-9], PID is [0-510]
  • Use hwloc to select interface to use if NIC information is available
    (only supported by cxi at the moment)
  • Support device memory types and FI_HMEM for verbs and cxi providers
  • Update min required version to libfabric 1.9
  • Improve debug output to print verbose FI info of selected provider
• [NA UCX]
  • Use active messaging UCP_FEATURE_AM for unexpected messages (only), this
    allows for removal of address resolution and retry on first message to
    exchange connection IDs
  • Turn on mempool by default
  • Support device memory types
• [NA PSM]
  • Add mercury NA plugin for the qlogic/intel PSM interface
    • Also support PSM2 (Intel OmniPath) through the PSM NA plugin
• [NA]
  • Add na_addr_format init info
  • Update NA_Mem_register() API call to support memory types (e.g., CUDA, ROCm, ZE) and devices IDs
  • Add na_loc module for hwloc detection
  • Remove na_uint, na_int, na_bool_t and na_size_t types
  • Use separate versioning for library and update to v3.0.0
• [NA IP]
  • Refactor na_ip_check_interface() to only use getaddrinfo() and getifaddrs()
  • Add family argument to force detection of IPv4/IPv6 addresses
  • Add ip debug log
• [HG util]
  • Add mercury_byteswap.h for bswap macros
  • Add mercury_inet.h for htonll and ntohll routine
  • Add mercury_param.h to use sys/param.h or MIN/MAX macros etc
  • Use separate versioning for library and update to v3.0.0
• [HG bulk]
  • Add support for memory attributes through a new HG_Bulk_create_attr() routine (support CUDA, ROCm, ZE)
• [HG]
  • Remove MERCURY_ENABLE_STATS CMake option and use 'diag' log subsys instead
    • Modify behavior of stats field to turn on diagnostics
    • Refactor existing counters (used only if debug is on)
  • Add checksum levels that can be manually controlled at runtime (disabled by default, HG_CHECKSUM_NONE level)
  • Update to mchecksum v2.0

Bug fixes

• [NA OFI]
  • Switch tcp provider to FI_PROGRESS_MANUAL
  • Prevent empty authorization keys from being passed
  • New implementation of address management
    • Fix duplicate addresses on multithreaded lookups
    • Redefine address keys and raw addresses to prevent allocations
    • Use FI addr map to lookup by FI addr
    • Improve serialization and deserialization of addresses
  • Fix provider table and use EP proto
  • Refactor and clean up plugin initialization
    • Clean up ip and domain checking
    • Ensure interface name is not used as domain name for verbs etc
    • Use NA IP module and add missing NA_OFI_VERIFY_PROV_DOM for tcp provider
    • Rework handling of fi_info to open fabric/domain/endpoint
    • Separate fabric from domain and keep single domain per NA class
    • Refactor handling of scalable vs standard endpoints
  • Improve handling of retries after FI_EAGAIN return code
    • Abort retried ops after default 90s timeout
    • Abort ops to a target being retried after first NA_HOSTUNREACH error in CQ
• [NA UCX]
  • Fix potential error not returned correctly on conn_insert()
• [HG util]
  • Make sure we round up ms time conversion, this ensures that small timeouts
    do not result in busy spin.
  • Fix 'none' log level not recognized
  • Let mercury log print counters on exit when debug outlet is on
• [HG proc]
  • Prevent call to save_ptr()/restore_ptr() during HG_FREE

⚠️ Known Issues

• [NA OFI]
  • [tcp/verbs;ofi_rxm] Using more than 256 peers requires FI_UNIVERSE_SIZE to be set.
• [NA UCX]
  • NA_Addr_to_string() cannot be used on non-listening processes to convert a self-address to a string.

mercury 2.1.0

Summary

This version brings bug fixes and updates to our v2.0.0 release.

New features

• [NA UCX]
  • Add initial support for UCX. As opposed to other plugins, the UCX plugin is able through the ucx+all init string to decide on which protocol to use.
• [NA SM]
  • Update default addressing format to follow PID-ID instead of PID/ID
  • Allow for passing of arbitrary SM init URIs
  • Enable support for bulk handle address binding
  • Add sm_info_string field to HG init info, which allows for specific init URIs to be used for SM when auto_sm is enabled.
• [NA]
  • Add thread_mode to NA init options and add NA_THREAD_MODE_SINGLE to relax thread-safety requirements.
  • Add na_cb_info_recv_expected to return actual_buf_size.
  • Add na_cb_type_to_string() to convert callback enum type to printable string.
• [NA IP]
  • Add na_ip_check_interface() routine that can be used by plugins to select IP interface to use.
• [HG util]
  • Add hg_mem_header_alloc()/free() calls to allocate buffers with a preceding header.
  • Add thread annotation module for thread safety analysis.
  • Add mercury_mem_pool memory pool to facilitate allocation and memory registration of a pool of buffers.
  • Enable format argument checking on logging functions.
  • Add hg_time_from_ms() and hg_time_to_ms() for time conversion to ms.
• [HG bulk]
  • Return transfer size size through hg_cb_info and hg_cb_info_bulk.

Bug fixes

• [NA OFI]
  • Require at least v1.7.0 of libfabric.
  • Fix handling of completion queue events and completion of retried operations that fail.
  • Fix progress loop to reduce time calls.
  • Force per-region registration for all providers and remove deprecated FI_MR_SCALABLE type of registrations and global MR keys.
• [NA SM]
  • Refactoring and clean up of sends/cancelation/retries/rma/address keys.
  • Remove use of usernames from SM paths.
• [HG util]
  • Prevent use of CLOCK_MONOTONIC_COARSE on PPC platforms and default to CLOCK_MONOTONIC.
  • Fix debug logs that were not freed at exit.
  • Remove return value of mutex lock/unlock routines.
  • Fix log subsys to prevent setting duplicates.
  • Simplify handling of compiler attributes and add mercury_compiler_attributes.h module.
  • Remove hg_util_ integer types and use stdint.h.
  • Remove OpenPA dependency for atomics and use built-in atomics instead (requires gcc >= 4.7).
• [HG/HG util/NA]
  • Fix thread safety warnings and potential thread locking issues.
  • Fix log level set routines that were not enabling the underlying log sub-system.
  • Avoid reading system timers and optimize handling of timeouts.
• [HG bulk]
  • Fix erroneous call to NA_Mem_deregister() when handle is deserialized.
  • Correctly mark op as canceled if canceled from NA.
  • Clean up and simplify handling of NA error return codes in callback.
  • Minimal tracking of bulk handles that are not freed.
• [HG Core]
  • Fix error handling when NA send fails during an HG_Forward() operation.
  • Correctly map NA error return code back to HG error return code in user callback.
  • Correctly print HG handle debug information.
  • In short responses like ACKs, leave room at the front of a buffer for
    the NA header, and expect the header to be present.
  • Fix potential issue on context destroy where handles could have been reposted while finalizing if RPCs were still in the queue.
• [General]
  • Warning and static analysis issues were fixed.

⚠️ Known Issues

• [NA OFI]
  • [tcp/verbs;ofi_rxm] Using more than 256 peers requires FI_UNIVERSE_SIZE to be set.
  • [tcp;ofi_rxm] Remains unstable, use sockets as a fallback in case of issues.
    • Please note that libfabric v1.13.0 and v1.13.1 have address management issues with that transport. Please either downgrade to v1.12.1 (or earlier) or upgrade to v1.13.2 (or later).
• [NA UCX]
  • NA_Addr_to_string() cannot be used on non-listening processes to convert a self-address to a string.

mercury 2.1.0rc4

Summary

This version brings bug fixes and updates to our v2.0.0 release.

New features

• [NA UCX]
  • Add initial support for UCX. As opposed to other plugins, the UCX plugin is able through the ucx+all init string to decide on which protocol to use.
• [NA SM]
  • Update default addressing format to follow PID-ID instead of PID/ID
  • Allow for passing of arbitrary SM init URIs
  • Enable support for bulk handle address binding
  • Add sm_info_string field to HG init info, which allows for specific init URIs to be used for SM when auto_sm is enabled.
• [NA]
  • Add thread_mode to NA init options and add NA_THREAD_MODE_SINGLE to relax thread-safety requirements.
  • Add na_cb_info_recv_expected to return actual_buf_size.
  • Add na_cb_type_to_string() to convert callback enum type to printable string.
• [NA IP]
  • Add na_ip_check_interface() routine that can be used by plugins to select IP interface to use.
• [HG util]
  • Add hg_mem_header_alloc()/free() calls to allocate buffers with a preceding header.
  • Add thread annotation module for thread safety analysis.
  • Add mercury_mem_pool memory pool to facilitate allocation and memory registration of a pool of buffers.
  • Enable format argument checking on logging functions.
  • Add hg_time_from_ms() and hg_time_to_ms() for time conversion to ms.
• [HG bulk]
  • Return transfer size size through hg_cb_info and hg_cb_info_bulk.

Bug fixes

• [NA OFI]
  • Require at least v1.7.0 of libfabric.
  • Fix handling of completion queue events and completion of retried operations that fail.
  • Fix progress loop to reduce time calls.
  • Force per-region registration for all providers and remove deprecated FI_MR_SCALABLE type of registrations and global MR keys.
• [NA SM]
  • Refactoring and clean up of sends/cancelation/retries/rma/address keys.
  • Remove use of usernames from SM paths.
• [HG util]
  • Prevent use of CLOCK_MONOTONIC_COARSE on PPC platforms and default to CLOCK_MONOTONIC.
  • Fix debug logs that were not freed at exit.
  • Remove return value of mutex lock/unlock routines.
  • Fix log subsys to prevent setting duplicates.
  • Simplify handling of compiler attributes and add mercury_compiler_attributes.h module.
  • Remove hg_util_ integer types and use stdint.h.
  • Remove OpenPA dependency for atomics and use built-in atomics instead (requires gcc >= 4.7).
• [HG/HG util/NA]
  • Fix thread safety warnings and potential thread locking issues.
  • Fix log level set routines that were not enabling the underlying log sub-system.
  • Avoid reading system timers and optimize handling of timeouts.
• [HG bulk]
  • Fix erroneous call to NA_Mem_deregister() when handle is deserialized.
  • Correctly mark op as canceled if canceled from NA.
  • Clean up and simplify handling of NA error return codes in callback.
• [HG Core]
  • Fix error handling when NA send fails during an HG_Forward() operation.
  • Correctly map NA error return code back to HG error return code in user callback.
  • Correctly print HG handle debug information.
  • In short responses like ACKs, leave room at the front of a buffer for
    the NA header, and expect the header to be present.
  • Fix potential issue on context destroy where handles could have been reposted while finalizing if RPCs were still in the queue.
• [General]
  • Warning and static analysis issues were fixed.

⚠️ Known Issues

• [NA OFI]
  • [tcp/verbs;ofi_rxm] Using more than 256 peers requires FI_UNIVERSE_SIZE to be set.
  • [tcp;ofi_rxm] Remains unstable, use sockets as a fallback in case of issues.
    • Please note that libfabric v1.13.0 and v1.13.1 have address management issues with that transport. Please either downgrade to v1.12.1 (or earlier) or upgrade to v1.13.2 (or later).
• [NA UCX]
  • NA_Addr_to_string() cannot be used on non-listening processes to convert a self-address to a string.

mercury 2.1.0rc3

Summary

This version brings bug fixes and updates to our v2.0.0 release.

New features

• [NA UCX]
  • Add initial support for UCX. As opposed to other plugins, the UCX plugin is able through the ucx+all init string to decide on which protocol to use.
• [NA SM]
  • Update default addressing format to follow PID-ID instead of PID/ID
  • Allow for passing of arbitrary SM init URIs
  • Enable support for bulk handle address binding
  • Add sm_info_string field to HG init info, which allows for specific init URIs to be used for SM when auto_sm is enabled.
• [NA]
  • Add thread_mode to NA init options and add NA_THREAD_MODE_SINGLE to relax thread-safety requirements.
  • Add na_cb_info_recv_expected to return actual_buf_size.
  • Add na_cb_type_to_string() to convert callback enum type to printable string.
• [NA IP]
  • Add na_ip_check_interface() routine that can be used by plugins to select IP interface to use.
• [HG util]
  • Add hg_mem_header_alloc()/free() calls to allocate buffers with a preceding header.
  • Add thread annotation module for thread safety analysis.
  • Add mercury_mem_pool memory pool to facilitate allocation and memory registration of a pool of buffers.
  • Enable format argument checking on logging functions.
  • Add hg_time_from_ms() and hg_time_to_ms() for time conversion to ms.
• [HG bulk]
  • Return transfer size size through hg_cb_info and hg_cb_info_bulk.

Bug fixes

• [NA OFI]
  • Require at least v1.7.0 of libfabric.
  • Fix handling of completion queue events and completion of retried operations that fail.
  • Fix progress loop to reduce time calls.
  • Force per-region registration for all providers and remove deprecated FI_MR_SCALABLE type of registrations and global MR keys.
• [NA SM]
  • Refactoring and clean up of sends/cancelation/retries/rma/address keys.
  • Remove use of usernames from SM paths.
• [HG util]
  • Prevent use of CLOCK_MONOTONIC_COARSE on PPC platforms and default to CLOCK_MONOTONIC.
  • Fix debug logs that were not freed at exit.
  • Remove return value of mutex lock/unlock routines.
  • Fix log subsys to prevent setting duplicates.
  • Simplify handling of compiler attributes and add mercury_compiler_attributes.h module.
  • Remove hg_util_ integer types and use stdint.h.
  • Remove OpenPA dependency for atomics and use built-in atomics instead (requires gcc >= 4.7).
• [HG/HG util/NA]
  • Fix thread safety warnings and potential thread locking issues.
  • Fix log level set routines that were not enabling the underlying log sub-system.
  • Avoid reading system timers and optimize handling of timeouts.
• [HG bulk]
  • Fix erroneous call to NA_Mem_deregister() when handle is deserialized.
  • Correctly mark op as canceled if canceled from NA.
• [HG Core]
  • Fix error handling when NA send fails during an HG_Forward() operation.
  • Correctly print HG handle debug information.
  • In short responses like ACKs, leave room at the front of a buffer for
    the NA header, and expect the header to be present.
• [General]
  • Warning and static analysis issues were fixed.

⚠️ Known Issues

• [NA OFI]
  • [tcp/verbs;ofi_rxm] Using more than 256 peers requires FI_UNIVERSE_SIZE to be set.
  • [tcp;ofi_rxm] Remains unstable, use sockets as a fallback in case of issues.
    • Please note that libfabric v1.13.0 and v1.13.1 have address management issues with that transport. Please either downgrade to v1.12.1 (or earlier) or upgrade to v1.13.2 (or later).
• [NA UCX]
  • NA_Addr_to_string() cannot be used on non-listening processes to convert a self-address to a string.

mercury 2.1.0rc2

Summary

This version brings bug fixes and updates to our v2.0.0 release.

New features

• [NA UCX]
  • Add initial support for UCX. As opposed to other plugins, the UCX plugin is able through the ucx+all init string to decide on which protocol to use.
• [NA SM]
  • Update default addressing format to follow PID-ID instead of PID/ID
  • Allow for passing of arbitrary SM init URIs
  • Enable support for bulk handle address binding
  • Add sm_info_string field to HG init info, which allows for specific init URIs to be used for SM when auto_sm is enabled.
• [NA]
  • Add thread_mode to NA init options and add NA_THREAD_MODE_SINGLE to relax thread-safety requirements.
  • Add na_cb_info_recv_expected to return actual_buf_size.
  • Add na_cb_type_to_string() to convert callback enum type to printable string.
• [NA IP]
  • Add na_ip_check_interface() routine that can be used by plugins to select IP interface to use.
• [HG util]
  • Add hg_mem_header_alloc()/free() calls to allocate buffers with a preceding header.
  • Add thread annotation module for thread safety analysis.
  • Add mercury_mem_pool memory pool to facilitate allocation and memory registration of a pool of buffers.
  • Enable format argument checking on logging functions.
  • Add hg_time_from_ms() and hg_time_to_ms() for time conversion to ms.
• [HG bulk]
  • Return transfer size size through hg_cb_info and hg_cb_info_bulk.

Bug fixes

• [NA OFI]
  • Fix handling of completion queue events and completion of retried operations that fail.
  • Fix progress loop to reduce time calls.
• [NA SM]
  • Refactoring and clean up of sends/cancelation/retries/rma/address keys.
  • Remove use of usernames from SM paths.
• [HG util]
  • Prevent use of CLOCK_MONOTONIC_COARSE on PPC platforms and default to CLOCK_MONOTONIC.
  • Fix debug logs that were not freed at exit.
  • Remove return value of mutex lock/unlock routines.
  • Fix log subsys to prevent setting duplicates.
• [HG/HG util/NA]
  • Fix thread safety warnings and potential thread locking issues.
  • Fix log level set routines that were not enabling the underlying log sub-system.
  • Avoid reading system timers and optimize handling of timeouts.
• [HG bulk]
  • Fix erroneous call to NA_Mem_deregister() when handle is deserialized.
  • Correctly mark op as canceled if canceled from NA.
• [HG Core]
  • Correctly print HG handle debug information.
  • In short responses like ACKs, leave room at the front of a buffer for
    the NA header, and expect the header to be present.
• [General]
  • Warning and static analysis issues were fixed.

⚠️ Known Issues

• [NA OFI]
  • [tcp/verbs;ofi_rxm] Using more than 256 peers requires FI_UNIVERSE_SIZE to be set.
  • [tcp;ofi_rxm] Remains unstable, use sockets as a fallback in case of issues.
    • Please note that libfabric v1.13.0 and v1.13.1 have address management issues with that provider. We do not recommend upgrading to these versions at the moment.
• [NA UCX]
  • NA_Addr_to_string() cannot be used on non-listening processes to convert a self-address to a string.

mercury 2.1.0rc1

Summary

This version brings bug fixes and updates to our v2.0.0 release.

New features

• [NA UCX]
  • Add initial support for UCX. As opposed to other plugins, the UCX plugin is able through the ucx+all init string to decide on which protocol to use.
• [NA]
  • Add thread_mode to NA init options and add NA_THREAD_MODE_SINGLE to relax thread-safety requirements.
  • Add na_cb_info_recv_expected to return actual_buf_size.
  • Add na_cb_type_to_string() to convert callback enum type to printable string.
• [NA IP]
  • Add na_ip_check_interface() routine that can be used by plugins to select IP interface to use.
• [HG util]
  • Add hg_mem_header_alloc()/free() calls to allocate buffers with a preceding header.
  • Add thread annotation module for thread safety analysis.
  • Add mercury_mem_pool memory pool to facilitate allocation and memory registration of a pool of buffers.
  • Enable format argument checking on logging functions.
  • Add hg_time_from_ms() and hg_time_to_ms() for time conversion to ms.
• [HG bulk]
  • Return transfer size size through hg_cb_info and hg_cb_info_bulk.

Bug fixes

• [NA OFI]
  • Fix handling of completion queue events and completion of retried operations that fail.
  • Fix progress loop to reduce time calls.
• [HG util]
  • Prevent use of CLOCK_MONOTONIC_COARSE on PPC platforms and default to CLOCK_MONOTONIC.
  • Fix debug logs that were not freed at exit.
  • Remove return value of mutex lock/unlock routines.
  • Fix log subsys to prevent setting duplicates.
• [HG/HG util/NA]
  • Fix thread safety warnings and potential thread locking issues.
  • Fix log level set routines that were not enabling the underlying log sub-system.
• [HG bulk]
  • Fix erroneous call to NA_Mem_deregister() when handle is deserialized.
  • Correctly mark op as canceled if canceled from NA.
• [HG Core]
  • Correctly print HG handle debug information.
  • In short responses like ACKs, leave room at the front of a buffer for
    the NA header, and expect the header to be present.

⚠️ Known Issues

• [NA OFI]
  • [tcp/verbs;ofi_rxm] Using more than 256 peers requires FI_UNIVERSE_SIZE to be set.
  • [tcp;ofi_rxm] Remains unstable, use sockets as a fallback in case of issues.
• [NA UCX]
  • NA_Addr_to_string() cannot be used on non-listening processes to convert a self-address to a string.
  • Serialization of addresses is currently not supported and will be supported in future UCX releases.

mercury 2.0.1

Summary

This version brings a few bug fixes and updates to our v2.0.0 release.

New features

• Improve logging system and add HG_LOG_SUBSYS environment variable that can be used
  in combination with HG_LOG_LEVEL to select log sub-systems.
  • Add min_debug log level to keep debug traces. Traces get printed when an error occurs.
  • [HG] Add HG_Set_log_subsys().
• [NA] Add support for message size hints though max_unexpected_size and max_expected_size hints. Supported with OFI, BMI and MPI plugins.
• [NA BMI/SM/OFI] Support sends to self address.
• [HG/NA] Add HG_HOSTUNREACH/NA_HOSTUNREACH error codes.

Bug fixes

• [HG]
  • Add missing check for NULL addr passed to HG_Forward().
  • Remove unnecessary spinwait() and track handle in completion queue.
  • Fix handle refcount if HG_Respond() fails.
  • Remove race in HG_Trigger() optimization that was skipping signaling.
• [HG bulk]
  • Prevent virtual handle data to be sent eagerly.
  • Ensure underlying error codes from NA are returned back to user.
• [HG util]
  • Fix timeout passed to pthread_cond_timedwait() when
    CLOCK_MONOTONIC_COARSE is used.
  • Remove check for STDERR_FILENO.
  • Add best-effort C++ compatibility for atomics.
• [NA]
  • Ensure completion callback is called after OP ID is fully released.
• [NA BMI]
  • Rework and simplify NA BMI code and remove extra allocations.
• [NA SM]
  • Prevent potential race on bulk handle that was freed.
  • Fix release of invalid addresses.
  • Prevent race in address resolution.
• [NA OFI]
  • Allow libfabric to return canceled operations.
  • Yield to other threads when using PSM2.
  • Return and convert OFI error codes back to upper layers.
  • Ensure selected domain matches address format.
  • Prevent tcp protocol to be used on macOS.
  • Fix potential memory leak in na_ofi_provider_check().
  • Add addr pool to prevent addr allocation on unexpected recv.

⚠️ Known Issues

• [NA OFI]
  • [tcp/verbs;ofi_rxm] Using more than 256 peers requires FI_UNIVERSE_SIZE to be set.
  • [tcp;ofi_rxm] Remains unstable, use sockets as a fallback in case of issues.

mercury 2.0.0

Major features since 1.0.1 include:

• Support for immediate lookups through HG_Addr_lookup2() (note that the callback version of HG_Addr_lookup will be deprecated in a future release)
• Improved support of libfabric and support of new tcp provider
• Improved shared-memory plugin with full connection-less endpoints support
• Improved bulk interface with more efficient handling of I/O with small segment count
• Improved efficiency of mercury proc routines
• Improved polling mechanism
• Improved cancellation of operations and error handling
• Improved error / warning and debug logging
• Removed CMake options (MERCURY_USE_SM_ROUTING, MERCURY_ENABLE_POST_LIMIT, MERCURY_USE_SELF, MERCURY_USE_EAGER_BULK), options are now controlled at runtime, through HG_Init_opt().

Please see the accompanying changelog for a full list of features and bug fixes.

mercury 2.0.0rc3

v2.0.0rc3

mercury 2.0.0rc3

mercury 2.0.0rc2

v2.0.0rc2

mercury 2.0.0rc2