-
Notifications
You must be signed in to change notification settings - Fork 0
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Preparing v0.6 Release (#200) #1
Merged
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Increment YGM version number in CMakeLists.txt * Sends comm:cerr() to std::cerr instead of std::cout (#78) * feature/msg_tweak (#77) * Increment YGM version number in CMakeLists.txt * changed messages to use instead of hard-coded value. * Cleaned up cmake build, and moved the library target description to one place. Co-authored-by: Trevor Steil <[email protected]> * Defaults CMAKE_BUILD_TYPE to Release (#76) * Initial ygm::io::multi_output functionality (#75) * Initial ygm::io::multi_output functionality * Removes unnecessary closing of ofstreams in destructor * Checks prefix path provided to ygm::io::multi_output does not exist as a regular file, and changes filename variable name to subpath * Adds tests for ygm::io::multi_output * Adds ygm::io::daily_output and simple tests * Adds test to check correct files are created by ygm::io::multi_output * Fixing clean-up of test files written * Forces filename_prefix given to ygm::io::multi_output constructor to be a directory * Feature/ci boost 1.78 (#80) Adds Boost 1.78 to CI. * Initial ygm::array implementation (#79) * Initial ygm::array implementation * Adds safety checks of array sizes * Adds barrier after resizing completes * Changes name of async_put to async_set * Adds array::async_unary_op_update_value and helpers for commonly used binary and unary operators * Adds functions for getting YGM pointers to arrays and array sizes * Adding container maptrix and SpMV. (#68) * Adding maptrix apis and impls. * Adding async_visit API+impl. * Some more details. * Maptrix API changes. * Adding new maptrix design. * Adding SpMV - first take. * Adding Structure * Porting to develop. * Adding SpMV as a standalone function. * Adding pagerank in examples and other restructuring changes. * Maptrix Impl. * Adding for_all over row_id and col_id. * Moving to experimental directory * Delete assoc_vector, replaced with ygm map. * Delete assoc_vector_impl, replaced with ygm map_impl. * Delete maptrix.hpp, in experimental mode. * Delete adj_impl.hpp, in experimental. * Delete csc_impl.hpp, in experimental. * Delete csr_impl.hpp, in experimental. * Delete maptrix_impl.hpp, in experimental. * Delete spmv.hpp, in experimental. * Moved as a part of alg_spmv.hpp. * Moved as a part of alg_spmv.hpp. * Add nicer examples or, tests. * Brain storming spmv row. * Moved within containers. * Moved within containers. * Adding new changes. * Adding timing details. * . * Changing API to insert_if_missing_else_visit. * Fix. * Fix. * Adding norm check. * Added OpPlus OpTimes. * Adding OpTimes. * Adding fixes from pull-request. * Adding fixes. * Delete spmv_row, not supporting for now. * Adding references. * Fixing references. * Changing line parsing. * Changing to 32bit. * Cleaning up webgraph example to use as general SpMV example * Removing extra headers * Switching to webgraph_spmv.cpp as new alg_spmv.cpp * Removing maptrix_visit * Cleaning up alg_pagerank.cpp * Removing unused 2D hasher * Removes unneeded header and changes a function name in adj_impl.hpp * Removing unused code in SpMV * Removing extra headers in column_view_impl.hpp * Removes unused headers in maptrix_impl and passes default value to row_view and column_view * Removes unused headers in maptrix_impl and row_view_impl * clang-format on maptrix.hpp Co-authored-by: Ancy Sarah Sarah Tom <[email protected]> Co-authored-by: Trevor Steil <[email protected]> * Feature/buffer multi output (#81) * Manually buffers ygm::io::multi_output writing * Adds buffering to ygm::io::daily_output * Updates ygm::io output tests to use buffering * Adds ability for csv_parser to read fields as unsigned integers (#83) * ygm::comm::Layout class (#82) * initial pass on implementing the Layout class. ygm::comm::layout() returns a const reference. * Removed comments and cleaned up ygm::Layout internals. * Reordered ygm::comm::Layout member for consistency. * Added tests for ygm::Layout::is_strided() and ygm::Layout::is_local(), and added functions to get const refs to all local and strided ranks. * Refactored ygm::Layout -> ygm::detail::layout. * Adds missing functions to counting set (#84) * Adds missing function to get a YGM pointer to a counting set * Adds comm() function to counting_set * Fixes typo in counting_set.comm() * Adds test for counting_set's YGM pointer * Adds topk to counting set * Adds barrier to beginning of map's topk * Adds example of counting_set topk * Consolidated ci jobs into a single job with matrixed dispatch on gcc version and mpi types. Also caching boost to avoid downloading/untaring with every job. (#86) * updated CI triggers such that PRs to main and develop and pushes to feauture/** and hotfix/** branches trigger jobs. (#88) * Adds possibility to send during processing of receive queue (#91) * Adds additional opportunities to flush buffers while processing received messages * Tweaked contributing guidelines to reflect CI changes and fixed some … (#89) * Tweaked contributing guidelines to reflect CI changes and fixed some typos. * Update CONTRIBUTING.md Co-authored-by: Trevor Steil <[email protected]> * Fixes bug in ygm::io::line_parser based on the split between files occurring on a newline. (#93) * Added get_ygm_ptr() to disjoint sets (#92) Co-authored-by: Sudharshan Srinivasan <[email protected]> * Switches ygm::container::array to block partitioning (#95) * Switches ygm::container::array to block partitioning * Gives last rank fewer elements during resize to avoid issues in for_all * Removes ability to resize ygm::container::array. No longer sensible with block partitioning * Properly sets local block size of last rank when block is full-sized * gather data to single std::vector added (#94) * gather data to single std::vector added * cleaned up collective operation, fixed test case, added to all support * Update bag_impl.hpp m_res -> p_res removes unnecessary `mailbox` from `gatherer` lambda passes `outer_data` to `gatherer` by reference uses m_comm to build ygm_ptr (preferred method) * Update bag_impl.hpp Makes vector arguments to lambdas in gather_to_vector const Co-authored-by: Dozier <[email protected]> Co-authored-by: Trevor Steil <[email protected]> * Hotfix/remove local receive (#96) * Adds ygm_tag to comm::impl * Removes local_receive function * Feature/interrupt mask (#97) * Adds initial ygm::detail::interrupt_mask implementation * Adds test for ygm::detail::mask_interrupt * Adds detail::interrupt_mask to map_impl to prevent iterator invalidation during map visits * Adds missing include statement for interrupt_mask in map_impl * Fixes namespacing for interrupt_mask use in map_impl * Gets m_comm from pmap in map_impl::async_visit_group * Adds missing mpi_typeof function for floats (#99) * Adds comm() function to set and multiset (#100) * Updates version number given in example CMake snippet (#101) * Added small fixes to clear gcc12 compiler warnings. (#102) * Feature/single threaded overhaul (#103) Significant overhaul removing listener thread and MPI_THREAD_MULTIPLE. * Removal of the listener thread. Removes requirement for MPI_THREAD_MULTIPLE. Improves performance substantially for the triangle counting benchmark. * NR & NLNR routing are supported. * Added the ability to capture primitive values in the async lambda to avoid Cereal overhead. This improves performance a bit, but mostly it makes user code easier to read/debug. Caution: there are no safety rails to prevent capturing a local pointer (but does prevent local references). * Added a full environment setting system, and it now controls the major settings in the runtime. * comm::welcome() added a welcome banner that prints out the current configuration and MPI settings. * API minor change removed the optional parameter to comm::comm(…, int buffer_capacity) that controlled the buffer size. Now it must be controlled by the environment. * Added new mechanism to indicate remote dispatch functions via static initialization. * Removes x86intrin.h inclusion in comm_impl (#104) * Inserted a std::stringstream into ygm::comm::cout()-style functions so that multi-rank print operations are less likely to get garbled. (#107) * Feature/ci update (#108) * Adds Github action to run GCC-8 tests on Ubuntu 20.04 as it is unavailable on Ubuntu 22.04 * Updating version of checkout action to get rid of javascript warning in Github Actions * Updating version of cache action to get rid of javascript warning in Github Actions * Develop arrow (#98) * arrow parquet file reader * arrow parquet file reader test * Update arrow_parquet_stream_reader.cpp Fixes typo in schema_to_string name * Update arrow_parquet_parser.hpp Fixes typo in schema_to_string name * arrow parquet file reader - updated how files in a directory are read Co-authored-by: Trevor Steil <[email protected]> * Small updates to Parquet parser (#109) * Updates Parquet reader tests and examples * Removes commented include * Updates arrow_parquet_parser to only check files from rank 0 during construction * Arrow/Parquet CI (#110) * arrow parquet file reader * arrow parquet file reader test * Adds installation of Apache Arrow to Github CI * Using Arrow version 9.0 * Downloading Arrow 9.0.0 for CI * Trying Arrow 10 for CI * Downloading newest Apache Arrow and installing 9.0.0 in CI * Trying to download Apache Arrow 9.0.0 source instead * Trying version range for Arrow * Changes CI runner to install Arrow 10.0 and explicitly check for Arrow 8, 9, and 10 in CMake to work around issues with Arrow version compatibility. * Investigating output test_arrow_parquet_stream_reader to find cause of failure in CI * Adds guards around finding different versions of Arrow Co-authored-by: tahsinreza <[email protected]> * Feature/nlnr bcast (#105) * Adds NLNR broadcast * Removes debug statements * Removes old bcast code * Reorganizes examples (#112) * Bumps version number to 0.5 (#111) * Release Prep * Adds dummy header to bcast messages when routing is used to process messages properlyin handle_next_receive (#120) * Loops through test_comm once for each routing scheme (#119) * Added support for (key, value) lambdas while still supporting (kv_pair) lambdas for ygm::container::map (#123) * Changed local lambda signature of ygm::container::map to expect separate (key, value) pairs. * Made ygm::container::map's expected lambda signature backwards compatible. * introduced constexpr compile-time guard to check that remote lambdas ygm::container::map adhere to legacy signature expectations. * ygm::container::map can now accept remote lambdas whose signatures expect either pairs or separate key, value arguments. pair visitors with optional map pointer arguments and no visitor arguments MUST specify that the second argument is a pair to compile correctly. * Update map_visit_optional_arguments_legacy.cpp * Update map_impl.hpp --------- Co-authored-by: Trevor Steil <[email protected]> * adding inline definition for release_assert_fail in order to support mulit-object targets; fix for Issue #126 (#127) * Feature/routing consistency (#125) * Initial comm_router implementation * Removes comments, adds user access to a comm's router, and spells out assumptions on routing in comm_router.hpp * Changes bcast to use same remote channels as new NLNR * wip/reducing adapter (#129) * Adds ygm::container::map::async_reduce operation * Fixes bug in test_map * Initial reducing_adapter for ygm::container::map without reduction tree * Adds ygm::container::array::key_type as an alias to Index for use in ygm::container::reducing_adapters * Adds container_traits for inspecting YGM container types and moves always_false from map_impl to ygm/detail/ygm_traits.hpp for use in other contexts * Adds ygm::container::reducing_adapter for use with ygm::container::array using functionality from ygm/container/detail/container_traits.hpp to handle both types * Adds is_counting_set and provides tests for container_traits * Stores reduction operation in ygm::container::reducing_adapter * modified ygm::container::bag::for_all() to support separated (first, second) lambdas (#128) * created ygm::container::detail::bag_impl::for_all_pairs() to be used with pair bags. * created make_similar() functions for map and array containers that return empty containers with the same comm and default value (and size for arrays) * Added missing cereal include for std::pair * moved template metaprogramming boilerplate into its own header. * Added special functionality to bag::for_all() so that it can accept split (first, second) signatures. * removed vestigial header * Added compiler guard to bag::for_all() and added more helpful compiler error messages. * removed the make_similar function in favor of something more disciplined in the future. * removed make_similar from multimap * remove legacy pair signature support for ygm::container::map (#132) * removed support for pair arguments in local and remote map lambdas. * Added more informative compiler error messages for map lambda signature checks * Removed outdate comment text [skip ci] * minor cleanup of compiler error message [skip ci] * Added support for local lambdas with (value_type&) signatures for arr… (#133) * Added support for local lambdas with (value_type&) signatures for arrays with constexpr compiler guards. * Opens ygm::container::array::owner() call to public API and adds ygm::container::array::is_mine() operation to match other containers (#136) * Reducing Adapter Reduction Tree (#139) * Adds basic caching layer to reducing_adapter without multi-hop caching * Adds reduction tree to reducing_adapter * Adds sanity check that reducing_adapter cache slot is empty after flushing * Fixes bug where reducing_adapter cache was declared non-empty when a value was placed in the underlying container * Adds missing check of pthis in reducing_adapter * Moves reducing_adapter class to ygm::container::detail * Added compile-time guards to all local and remote container lambdas (#138) * placed compiler guards on remote array lambdas * Added compiler guards to local set lambdas * changed local lambda signatures of disjoint_set to separated [](const value_type &, const value_type &) format to match map. Also added compiler guards to disjoint_set local and remote lambdas. * Feature reduce by key (#144) * Added reduce_by_key and started new traits features. * Adds communicator collectives. (#143) * Adds communicator collectives. * fixed MPI_Comm. * bcast & is_same * Added rank-aware RNG wrapper. It can be modified with different rank/… (#140) * Added rank-aware RNG wrapper. It can be modified with different rank/seed strategies. * updated random namespace to mimic STL * fixed namespace device -> engine * removed shared_random_device. Will reintroduce when we add a post barrier callback concept * moved most random machinery into ygm::detail and made std::mt19937 the default random engine * Hotfix/recursive double receive (#147) * Adds .git to Git repos fetched in CMakeLists.txt (#150) * Added new set methods. (#151) * Update comm_impl.hpp (#152) Removed old debug asserts. * static_assert no longer triggers when using a lambda with disjoint_set::async_union_and_execute that requires a pointer to the disjoint set (#158) * Feature/bag shuffle (#130) * Added local and global shuffles to bag container * Added test case for shuffles * Cleaning up a little code * Updated global bag shuffle to accept RNG as well * Updated bag shuffles to utilize ygm:default_random_engine * Removed some comments * Adding template for RandomEngine and functions where no rng argument is passed * Finished templating shuffle functions properly * Bugfix/disjoint set logic (#159) * Overcomplicated disjoint set with path splitting storing parents and their ranks * Simplifies disjoint_set * Adds missing const in async_union_and_execute * Avoid running up tree when other_item is my_parent * Avoid running up tree when other_item is my_parent * Finishes initial path-splitting union-by-rank disjoint set implementation * Adds missing barrier at beginning of all_compress() * Bugfix/disjoint set optional pointer (#160) Stops static_assert from triggering when disjoint_set::async_union_and_execute lambda requires a pointer to the disjoint set * Feature/bag balancing (#146) * Rough draft for rebalencing bag function * WIP: Second iteration of bag rebalance code * Added second iteration of rebalance method. * Appended rebalance tests to test_bag.cpp * Updated rebalancing algorithm to reduce space complexity. Added local_pop(int n) function to pop multiple values at once. * Updated test to reflect bag rebalancing sizes being congruent with ygm arrays * static_assert no longer triggers when using a lambda with disjoint_set::async_union_and_execute that requires a pointer to the disjoint set (#158) * Feature/bag shuffle (#130) * Added local and global shuffles to bag container * Added test case for shuffles * Cleaning up a little code * Updated global bag shuffle to accept RNG as well * Updated bag shuffles to utilize ygm:default_random_engine * Removed some comments * Adding template for RandomEngine and functions where no rng argument is passed * Finished templating shuffle functions properly * Bugfix/disjoint set logic (#159) * Overcomplicated disjoint set with path splitting storing parents and their ranks * Simplifies disjoint_set * Adds missing const in async_union_and_execute * Avoid running up tree when other_item is my_parent * Avoid running up tree when other_item is my_parent * Finishes initial path-splitting union-by-rank disjoint set implementation * Adds missing barrier at beginning of all_compress() * Bugfix/disjoint set optional pointer (#160) Stops static_assert from triggering when disjoint_set::async_union_and_execute lambda requires a pointer to the disjoint set * Rough draft for rebalencing bag function * Merged with updated develop branch to prep for pull request. * Added second iteration of rebalance method. * Appended rebalance tests to test_bag.cpp * Updated rebalancing algorithm to reduce space complexity. Added local_pop(int n) function to pop multiple values at once. * Updated test to reflect bag rebalancing sizes being congruent with ygm arrays * Several small fixes to make rebalancing run with better space efficiency * Fixed async_insert vector pass by value error. Made small fix for accessing map elements to save memory. * Fixed O(p) time complexity issue when sending values for rebalance * Adding support for Arrow 12.0. (#162) * arrow parquet file reader * arrow parquet file reader * arrow parquet file reader * arrow parquet file reader * arrow parquet file reader * arrow parquet file reader * arrow parquet file reader * arrow parquet file reader * arrow parquet file reader * arrow parquet file reader test * Update arrow_parquet_stream_reader.cpp Fixes typo in schema_to_string name * Update arrow_parquet_parser.hpp Fixes typo in schema_to_string name * arrow parquet file reader - updated how files in a directory are read * merge develop into develop-arrow * adding support for Arrow 12.0 * merge develop into develop-arrow * Added missing for_all test for counting set (#165) Co-authored-by: Stephen Thaddaeus Youd <[email protected]> * added tagged bag container (#163) Added tagged_bag container * Feature/container traits (#161) Added ygm_container_type tags to containers. Additionally added compile time functions to check container type. * Move Constructor for Set (#166) * Adds test of std::vector of YGM sets * Fixes size check in test of vector of YGM sets * Adds move constructor to YGM set * Removes comm_impl and bag_impl (#167) * Removes comm::impl * Properly passes lambdas to pack_lambda operations and removes mention of comm::impl from interrupt_mask * Removes comm copy constructor * Changes comm in arrow_parquet_parser to a reference * Removes comm_impl.hpp includes from comments * Removes bag_impl * Added missing limits include (#168) * Feature/comm progress (#171) * Added comm::progress() and comm::wait_until(Fn). * Added comm::process_incoming() * Adding consume_all to set. (#169) * Adding consume_all to set. * Made adapter more clear. * Hotfix/multi output filesystem (#176) * Adds missing filesystem include to multi_output.hpp * Replaces csv_parser.hpp include in ndjson_parser.hpp with line_parser.hpp * Removes unnecessary ygm::io::detail namespace * Renamed 3 local comm member functions. (#173) * fix type of the get_ygm_ptr call in set.hpp (#178) * fix type of set get_ygm_ptr * add a test for the ygm set pointer * also fix the set pointer for multiset --------- Co-authored-by: Grace Jingying Li <[email protected]> * Hotfix/arrow_ci (#180) * Adding recent version of Arrow to CMakeLists.txt * make -j * Adds CMake flag to require a version of Arrow to be found, specifically for use in CI to make it obvious when Apache Arrow is not being found * Adds disjoint_set::clear() function (#183) * Feature/context aware progress (#182) * Check calling context in comm.local_progress() * Fix spelling error and increase test trip count * Add Parquet -> JSON Converter (#181) * Change parquet parser wrt schema Specifically, schema() returns physical type rather than logical type. * Add paquet -> json converter * Update Parquet JSON reader example * Arrow 14.0 (#185) Change the cmake file to support Apache Arrow v14.0 * Fixes array partitioning and bag rebalancing to be more balanced. (#187) * Copies all private variables in ygm::container::detail::array_impl copy constructor (#189) * Removes impl from ygm::container::array (#190) * Parallel Parquet File Reading (v2) (#188) * Parallel parquet reader v2 * Add comments * Add tests for parquet reader (#191) * Add tests for parquet reader - Add test for parquet2json convertor - Add additional parallel parquet reading test * Fix wrong indent * Brush up on parque reader test * Updates Regarding Arrow Parquet (#193) * Read values using std::optional in the ParquetToJson converter * Support Arrow v15 * Bugfix/disjoint set compress (#195) * Adds print statements to see if obvious bugs exist in disjoint_set::all_compress() * Removes race condition in disjoint_set::all_compress() by preparing all parent queries before sending any * Removing print statements from disjoint_set * Adds check of parent ranks before compressing disjoint sets * Adds check of parent ranks before compressing disjoint sets * Adds check of parent ranks before compressing disjoint sets * Avoid querying parent during disjoint_set::all_compress() when item is a root * Fixes missing reference for local_rep_query inside of update_rep_functor of disjoint_set::all_compress() * More detailed debugging output in all_compress() * Fixing logic for when a parent request comes for an item that had to query its own parent in current all_compress() round * Fixing logic for when a parent request comes for an item that had to query its own parent in current all_compress() round * Cleaning up all_compress() queries as they return * Cleaning up all_compress() queries as they return * Adding additional debug printing to disjoint_set::all_compress() * Avoids potential race condition where all_compress() is using whether parent of item being queried has returned in place of directly tracking whether item has found its root * Cleaning up disjoint_set::all_compress() code * Removing unused code * Add parquet2variant converter (#194) Co-authored-by: Keita Iwabuchi <[email protected]> * Bugfix/disjoint set compress (#196) * Adds missing async_visit function to disjoint_set * Fixes stopping criteria on while loop in disjoint_set::all_compress() * Fixes tracking of ongoing compression updates (#197) * Updating version number to 0.6 (#198) * Removes performance directory (#199) * Fixing Github actions to run on pull requests to master branch --------- Co-authored-by: Benjamin Priest <[email protected]> Co-authored-by: Roger Pearce <[email protected]> Co-authored-by: ancysarahtom <[email protected]> Co-authored-by: Ancy Sarah Sarah Tom <[email protected]> Co-authored-by: Sudharshan <[email protected]> Co-authored-by: Sudharshan Srinivasan <[email protected]> Co-authored-by: Ryan Dozier <[email protected]> Co-authored-by: Dozier <[email protected]> Co-authored-by: tahsinreza <[email protected]> Co-authored-by: tahsinreza <[email protected]> Co-authored-by: jleidel <[email protected]> Co-authored-by: Roger Pearce <[email protected]> Co-authored-by: Lance <[email protected]> Co-authored-by: youd3 <[email protected]> Co-authored-by: Stephen Thaddaeus Youd <[email protected]> Co-authored-by: Seth Bromberger <[email protected]> Co-authored-by: Grace <[email protected]> Co-authored-by: Grace Jingying Li <[email protected]> Co-authored-by: Preston Piercey <[email protected]> Co-authored-by: Keita Iwabuchi <[email protected]> Co-authored-by: Keita Iwabuchi <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Increment YGM version number in CMakeLists.txt
Sends comm:cerr() to std::cerr instead of std::cout (Hotfix/comm_cerr_to_std_cerr LLNL/ygm#78)
feature/msg_tweak (feature/msg_tweak LLNL/ygm#77)
Increment YGM version number in CMakeLists.txt
changed messages to use instead of hard-coded value.
Cleaned up cmake build, and moved the library target description to one place.
Defaults CMAKE_BUILD_TYPE to Release (Defaults CMAKE_BUILD_TYPE to Release LLNL/ygm#76)
Initial ygm::io::multi_output functionality (Initial ygm::io::multi_output functionality LLNL/ygm#75)
Initial ygm::io::multi_output functionality
Removes unnecessary closing of ofstreams in destructor
Checks prefix path provided to ygm::io::multi_output does not exist as a regular file, and changes filename variable name to subpath
Adds tests for ygm::io::multi_output
Adds ygm::io::daily_output and simple tests
Adds test to check correct files are created by ygm::io::multi_output
Fixing clean-up of test files written
Forces filename_prefix given to ygm::io::multi_output constructor to be a directory
Feature/ci boost 1.78 (Feature/ci boost 1.78 LLNL/ygm#80)
Adds Boost 1.78 to CI.
Initial ygm::array implementation (Initial ygm::array implementation LLNL/ygm#79)
Initial ygm::array implementation
Adds safety checks of array sizes
Adds barrier after resizing completes
Changes name of async_put to async_set
Adds array::async_unary_op_update_value and helpers for commonly used binary and unary operators
Adds functions for getting YGM pointers to arrays and array sizes
Adding container maptrix and SpMV. (Adding container maptrix and SpMV. LLNL/ygm#68)
Adding maptrix apis and impls.
Adding async_visit API+impl.
Some more details.
Maptrix API changes.
Adding new maptrix design.
Adding SpMV - first take.
Adding Structure
Porting to develop.
Adding SpMV as a standalone function.
Adding pagerank in examples and other restructuring changes.
Maptrix Impl.
Adding for_all over row_id and col_id.
Moving to experimental directory
Delete assoc_vector, replaced with ygm map.
Delete assoc_vector_impl, replaced with ygm map_impl.
Delete maptrix.hpp, in experimental mode.
Delete adj_impl.hpp, in experimental.
Delete csc_impl.hpp, in experimental.
Delete csr_impl.hpp, in experimental.
Delete maptrix_impl.hpp, in experimental.
Delete spmv.hpp, in experimental.
Moved as a part of alg_spmv.hpp.
Moved as a part of alg_spmv.hpp.
Add nicer examples or, tests.
Brain storming spmv row.
Moved within containers.
Moved within containers.
Adding new changes.
Adding timing details.
.
Changing API to insert_if_missing_else_visit.
Fix.
Fix.
Adding norm check.
Added OpPlus OpTimes.
Adding OpTimes.
Adding fixes from pull-request.
Adding fixes.
Delete spmv_row, not supporting for now.
Adding references.
Fixing references.
Changing line parsing.
Changing to 32bit.
Cleaning up webgraph example to use as general SpMV example
Removing extra headers
Switching to webgraph_spmv.cpp as new alg_spmv.cpp
Removing maptrix_visit
Cleaning up alg_pagerank.cpp
Removing unused 2D hasher
Removes unneeded header and changes a function name in adj_impl.hpp
Removing unused code in SpMV
Removing extra headers in column_view_impl.hpp
Removes unused headers in maptrix_impl and passes default value to row_view and column_view
Removes unused headers in maptrix_impl and row_view_impl
clang-format on maptrix.hpp
Feature/buffer multi output (Feature/buffer multi output LLNL/ygm#81)
Manually buffers ygm::io::multi_output writing
Adds buffering to ygm::io::daily_output
Updates ygm::io output tests to use buffering
Adds ability for csv_parser to read fields as unsigned integers (Adds ability for csv_parser to read fields as unsigned integers LLNL/ygm#83)
ygm::comm::Layout class (ygm::comm::Layout class LLNL/ygm#82)
initial pass on implementing the Layout class. ygm::comm::layout() returns a const reference.
Removed comments and cleaned up ygm::Layout internals.
Reordered ygm::comm::Layout member for consistency.
Added tests for ygm::Layout::is_strided() and ygm::Layout::is_local(), and added functions to get const refs to all local and strided ranks.
Refactored ygm::Layout -> ygm::detail::layout.
Adds missing functions to counting set (Adds missing functions to counting set LLNL/ygm#84)
Adds missing function to get a YGM pointer to a counting set
Adds comm() function to counting_set
Fixes typo in counting_set.comm()
Adds test for counting_set's YGM pointer
Adds topk to counting set
Adds barrier to beginning of map's topk
Adds example of counting_set topk
Consolidated ci jobs into a single job with matrixed dispatch on gcc version and mpi types. Also caching boost to avoid downloading/untaring with every job. (Consolidated ci jobs into a single job with matrixed dispatch on gcc … LLNL/ygm#86)
updated CI triggers such that PRs to main and develop and pushes to feauture/** and hotfix/** branches trigger jobs. (More granular CI triggers LLNL/ygm#88)
Adds possibility to send during processing of receive queue (Adds possibility to send during processing of receive queue LLNL/ygm#91)
Adds additional opportunities to flush buffers while processing received messages
Tweaked contributing guidelines to reflect CI changes and fixed some … (Tweaked contributing guidelines to reflect CI changes and fixed some … LLNL/ygm#89)
Tweaked contributing guidelines to reflect CI changes and fixed some typos.
Update CONTRIBUTING.md
Fixes bug in ygm::io::line_parser based on the split between files occurring on a newline. (Fixes newline bug in ygm::io::line_parser LLNL/ygm#93)
Added get_ygm_ptr() to disjoint sets (Added get_ygm_ptr() to disjoint sets LLNL/ygm#92)
Switches ygm::container::array to block partitioning (Switches ygm::container::array to block partitioning LLNL/ygm#95)
Switches ygm::container::array to block partitioning
Gives last rank fewer elements during resize to avoid issues in for_all
Removes ability to resize ygm::container::array. No longer sensible with block partitioning
Properly sets local block size of last rank when block is full-sized
gather data to single std::vector added (gather data to single std::vector added LLNL/ygm#94)
gather data to single std::vector added
cleaned up collective operation, fixed test case, added to all support
Update bag_impl.hpp
m_res -> p_res
removes unnecessary
mailbox
fromgatherer
lambda passesouter_data
togatherer
by referenceuses m_comm to build ygm_ptr (preferred method)
Makes vector arguments to lambdas in gather_to_vector const
Hotfix/remove local receive (Hotfix/remove local receive LLNL/ygm#96)
Adds ygm_tag to comm::impl
Removes local_receive function
Feature/interrupt mask (Feature/interrupt mask LLNL/ygm#97)
Adds initial ygm::detail::interrupt_mask implementation
Adds test for ygm::detail::mask_interrupt
Adds detail::interrupt_mask to map_impl to prevent iterator invalidation during map visits
Adds missing include statement for interrupt_mask in map_impl
Fixes namespacing for interrupt_mask use in map_impl
Gets m_comm from pmap in map_impl::async_visit_group
Adds missing mpi_typeof function for floats (Adds missing mpi_typeof function for floats LLNL/ygm#99)
Adds comm() function to set and multiset (Adds comm() function to set and multiset LLNL/ygm#100)
Updates version number given in example CMake snippet (Updates version number given in example CMake snippet LLNL/ygm#101)
Added small fixes to clear gcc12 compiler warnings. (Added small fixes to clear gcc12 compiler warnings. LLNL/ygm#102)
Feature/single threaded overhaul (Feature/single threaded overhaul LLNL/ygm#103)
Significant overhaul removing listener thread and MPI_THREAD_MULTIPLE.
Removal of the listener thread. Removes requirement for MPI_THREAD_MULTIPLE. Improves performance substantially for the triangle counting benchmark.
NR & NLNR routing are supported.
Added the ability to capture primitive values in the async lambda to avoid Cereal overhead. This improves performance a bit, but mostly it makes user code easier to read/debug. Caution: there are no safety rails to prevent capturing a local pointer (but does prevent local references).
Added a full environment setting system, and it now controls the major settings in the runtime.
comm::welcome() added a welcome banner that prints out the current configuration and MPI settings.
API minor change removed the optional parameter to comm::comm(…, int buffer_capacity) that controlled the buffer size. Now it must be controlled by the environment.
Added new mechanism to indicate remote dispatch functions via static initialization.
Removes x86intrin.h inclusion in comm_impl (Removes x86intrin.h inclusion in comm_impl LLNL/ygm#104)
Inserted a std::stringstream into ygm::comm::cout()-style functions so that multi-rank print operations are less likely to get garbled. (Inserted a std::stringstream into ygm::comm::cout()-style functions s… LLNL/ygm#107)
Feature/ci update (Feature/ci update LLNL/ygm#108)
Adds Github action to run GCC-8 tests on Ubuntu 20.04 as it is unavailable on Ubuntu 22.04
Updating version of checkout action to get rid of javascript warning in Github Actions
Updating version of cache action to get rid of javascript warning in Github Actions
Develop arrow (Develop arrow LLNL/ygm#98)
arrow parquet file reader
arrow parquet file reader test
Update arrow_parquet_stream_reader.cpp
Fixes typo in schema_to_string name
Fixes typo in schema_to_string name
arrow parquet file reader - updated how files in a directory are read
Small updates to Parquet parser (Small updates to Parquet parser LLNL/ygm#109)
Updates Parquet reader tests and examples
Removes commented include
Updates arrow_parquet_parser to only check files from rank 0 during construction
Arrow/Parquet CI (Arrow/Parquet CI LLNL/ygm#110)
arrow parquet file reader
arrow parquet file reader test
Adds installation of Apache Arrow to Github CI
Using Arrow version 9.0
Downloading Arrow 9.0.0 for CI
Trying Arrow 10 for CI
Downloading newest Apache Arrow and installing 9.0.0 in CI
Trying to download Apache Arrow 9.0.0 source instead
Trying version range for Arrow
Changes CI runner to install Arrow 10.0 and explicitly check for Arrow 8, 9, and 10 in CMake to work around issues with Arrow version compatibility.
Investigating output test_arrow_parquet_stream_reader to find cause of failure in CI
Adds guards around finding different versions of Arrow
Feature/nlnr bcast (Feature/nlnr bcast LLNL/ygm#105)
Adds NLNR broadcast
Removes debug statements
Removes old bcast code
Reorganizes examples (Reorganizes examples LLNL/ygm#112)
Bumps version number to 0.5 (Bumps version number to 0.5 LLNL/ygm#111)
Release Prep
Adds dummy header to bcast messages when routing is used to process messages properlyin handle_next_receive (Bugfix/bcast with routing LLNL/ygm#120)
Loops through test_comm once for each routing scheme (Feature/test routing LLNL/ygm#119)
Added support for (key, value) lambdas while still supporting (kv_pair) lambdas for ygm::container::map (Added support for (key, value) lambdas while still supporting (kv_pair) lambdas for ygm::container::map LLNL/ygm#123)
Changed local lambda signature of ygm::container::map to expect separate (key, value) pairs.
Made ygm::container::map's expected lambda signature backwards compatible.
introduced constexpr compile-time guard to check that remote lambdas ygm::container::map adhere to legacy signature expectations.
ygm::container::map can now accept remote lambdas whose signatures expect either pairs or separate key, value arguments. pair visitors with optional map pointer arguments and no visitor arguments MUST specify that the second argument is a pair to compile correctly.
Update map_visit_optional_arguments_legacy.cpp
Update map_impl.hpp
adding inline definition for release_assert_fail in order to support mulit-object targets; fix for Issue Building multi-object targets triggers linker failure LLNL/ygm#126 (adding inline definition for release_assert_fail; Fix for Issue #126 LLNL/ygm#127)
Feature/routing consistency (Feature/routing consistency LLNL/ygm#125)
Initial comm_router implementation
Removes comments, adds user access to a comm's router, and spells out assumptions on routing in comm_router.hpp
Changes bcast to use same remote channels as new NLNR
wip/reducing adapter (feature/reducing adapter LLNL/ygm#129)
Adds ygm::container::map::async_reduce operation
Fixes bug in test_map
Initial reducing_adapter for ygm::container::map without reduction tree
Adds ygm::container::array::key_type as an alias to Index for use in ygm::container::reducing_adapters
Adds container_traits for inspecting YGM container types and moves always_false from map_impl to ygm/detail/ygm_traits.hpp for use in other contexts
Adds ygm::container::reducing_adapter for use with ygm::container::array using functionality from ygm/container/detail/container_traits.hpp to handle both types
Adds is_counting_set and provides tests for container_traits
Stores reduction operation in ygm::container::reducing_adapter
modified ygm::container::bag::for_all() to support separated (first, second) lambdas (modified ygm::container::bag::for_all() to support separated (first, second) lambdas LLNL/ygm#128)
created ygm::container::detail::bag_impl::for_all_pairs() to be used with pair bags.
created make_similar() functions for map and array containers that return empty containers with the same comm and default value (and size for arrays)
Added missing cereal include for std::pair
moved template metaprogramming boilerplate into its own header.
Added special functionality to bag::for_all() so that it can accept split (first, second) signatures.
removed vestigial header
Added compiler guard to bag::for_all() and added more helpful compiler error messages.
removed the make_similar function in favor of something more disciplined in the future.
removed make_similar from multimap
remove legacy pair signature support for ygm::container::map (remove legacy pair signature support for ygm::container::map LLNL/ygm#132)
removed support for pair arguments in local and remote map lambdas.
Added more informative compiler error messages for map lambda signature checks
Removed outdate comment text [skip ci]
minor cleanup of compiler error message [skip ci]
Added support for local lambdas with (value_type&) signatures for arr… (Added support for local lambdas with (value_type&) signatures for arr… LLNL/ygm#133)
Added support for local lambdas with (value_type&) signatures for arrays with constexpr compiler guards.
Opens ygm::container::array::owner() call to public API and adds ygm::container::array::is_mine() operation to match other containers (Array Owner Operation LLNL/ygm#136)
Reducing Adapter Reduction Tree (Reducing Adapter Reduction Tree LLNL/ygm#139)
Adds basic caching layer to reducing_adapter without multi-hop caching
Adds reduction tree to reducing_adapter
Adds sanity check that reducing_adapter cache slot is empty after flushing
Fixes bug where reducing_adapter cache was declared non-empty when a value was placed in the underlying container
Adds missing check of pthis in reducing_adapter
Moves reducing_adapter class to ygm::container::detail
Added compile-time guards to all local and remote container lambdas (Added compile-time guards to all local and remote container lambdas LLNL/ygm#138)
placed compiler guards on remote array lambdas
Added compiler guards to local set lambdas
changed local lambda signatures of disjoint_set to separated [](const value_type &, const value_type &) format to match map. Also added compiler guards to disjoint_set local and remote lambdas.
Feature reduce by key (Feature reduce by key LLNL/ygm#144)
Added reduce_by_key and started new traits features.
Adds communicator collectives. (Adds communicator collectives. LLNL/ygm#143)
Adds communicator collectives.
fixed MPI_Comm.
bcast & is_same
Added rank-aware RNG wrapper. It can be modified with different rank/… (Added rank-aware RNG wrapper. It can be modified with different rank/… LLNL/ygm#140)
Added rank-aware RNG wrapper. It can be modified with different rank/seed strategies.
updated random namespace to mimic STL
fixed namespace device -> engine
removed shared_random_device. Will reintroduce when we add a post barrier callback concept
moved most random machinery into ygm::detail and made std::mt19937 the default random engine
Hotfix/recursive double receive (Hotfix/recursive double receive LLNL/ygm#147)
Adds .git to Git repos fetched in CMakeLists.txt (Adds .git to Git repos fetched in CMakeLists.txt LLNL/ygm#150)
Added new set methods. (Added new set methods. LLNL/ygm#151)
Update comm_impl.hpp (Update comm_impl.hpp LLNL/ygm#152)
Removed old debug asserts.
static_assert no longer triggers when using a lambda with disjoint_set::async_union_and_execute that requires a pointer to the disjoint set (Bugfix/disjoint set optional pointer LLNL/ygm#158)
Feature/bag shuffle (Feature/bag shuffle LLNL/ygm#130)
Added local and global shuffles to bag container
Added test case for shuffles
Cleaning up a little code
Updated global bag shuffle to accept RNG as well
Updated bag shuffles to utilize ygm:default_random_engine
Removed some comments
Adding template for RandomEngine and functions where no rng argument is passed
Finished templating shuffle functions properly
Bugfix/disjoint set logic (Bugfix/disjoint set logic LLNL/ygm#159)
Overcomplicated disjoint set with path splitting storing parents and their ranks
Simplifies disjoint_set
Adds missing const in async_union_and_execute
Avoid running up tree when other_item is my_parent
Avoid running up tree when other_item is my_parent
Finishes initial path-splitting union-by-rank disjoint set implementation
Adds missing barrier at beginning of all_compress()
Bugfix/disjoint set optional pointer (Bugfix/disjoint set optional pointer LLNL/ygm#160)
Stops static_assert from triggering when disjoint_set::async_union_and_execute lambda requires a pointer to the disjoint set
Feature/bag balancing (Feature/bag balancing LLNL/ygm#146)
Rough draft for rebalencing bag function
WIP: Second iteration of bag rebalance code
Added second iteration of rebalance method.
Appended rebalance tests to test_bag.cpp
Updated rebalancing algorithm to reduce space complexity. Added local_pop(int n) function to pop multiple values at once.
Updated test to reflect bag rebalancing sizes being congruent with ygm arrays
static_assert no longer triggers when using a lambda with disjoint_set::async_union_and_execute that requires a pointer to the disjoint set (Bugfix/disjoint set optional pointer LLNL/ygm#158)
Feature/bag shuffle (Feature/bag shuffle LLNL/ygm#130)
Added local and global shuffles to bag container
Added test case for shuffles
Cleaning up a little code
Updated global bag shuffle to accept RNG as well
Updated bag shuffles to utilize ygm:default_random_engine
Removed some comments
Adding template for RandomEngine and functions where no rng argument is passed
Finished templating shuffle functions properly
Bugfix/disjoint set logic (Bugfix/disjoint set logic LLNL/ygm#159)
Overcomplicated disjoint set with path splitting storing parents and their ranks
Simplifies disjoint_set
Adds missing const in async_union_and_execute
Avoid running up tree when other_item is my_parent
Avoid running up tree when other_item is my_parent
Finishes initial path-splitting union-by-rank disjoint set implementation
Adds missing barrier at beginning of all_compress()
Bugfix/disjoint set optional pointer (Bugfix/disjoint set optional pointer LLNL/ygm#160)
Stops static_assert from triggering when disjoint_set::async_union_and_execute lambda requires a pointer to the disjoint set
Rough draft for rebalencing bag function
Merged with updated develop branch to prep for pull request.
Added second iteration of rebalance method.
Appended rebalance tests to test_bag.cpp
Updated rebalancing algorithm to reduce space complexity. Added local_pop(int n) function to pop multiple values at once.
Updated test to reflect bag rebalancing sizes being congruent with ygm arrays
Several small fixes to make rebalancing run with better space efficiency
Fixed async_insert vector pass by value error. Made small fix for accessing map elements to save memory.
Fixed O(p) time complexity issue when sending values for rebalance
Adding support for Arrow 12.0. (Adding support for Arrow 12.0. LLNL/ygm#162)
arrow parquet file reader
arrow parquet file reader
arrow parquet file reader
arrow parquet file reader
arrow parquet file reader
arrow parquet file reader
arrow parquet file reader
arrow parquet file reader
arrow parquet file reader
arrow parquet file reader test
Update arrow_parquet_stream_reader.cpp
Fixes typo in schema_to_string name
Fixes typo in schema_to_string name
arrow parquet file reader - updated how files in a directory are read
merge develop into develop-arrow
adding support for Arrow 12.0
merge develop into develop-arrow
Added missing for_all test for counting set (Added missing for_all test for counting set LLNL/ygm#165)
added tagged bag container (added tagged bag container LLNL/ygm#163)
Added tagged_bag container
Added ygm_container_type tags to containers. Additionally added compile time functions to check container type.
Move Constructor for Set (Move Constructor for Set LLNL/ygm#166)
Adds test of std::vector of YGM sets
Fixes size check in test of vector of YGM sets
Adds move constructor to YGM set
Removes comm_impl and bag_impl (Removes comm_impl and bag_impl LLNL/ygm#167)
Removes comm::impl
Properly passes lambdas to pack_lambda operations and removes mention of comm::impl from interrupt_mask
Removes comm copy constructor
Changes comm in arrow_parquet_parser to a reference
Removes comm_impl.hpp includes from comments
Removes bag_impl
Added missing limits include (Added missing limits include LLNL/ygm#168)
Feature/comm progress (Feature/comm progress LLNL/ygm#171)
Added comm::progress() and comm::wait_until(Fn).
Added comm::process_incoming()
Adding consume_all to set. (Adding consume_all to set. LLNL/ygm#169)
Adding consume_all to set.
Made adapter more clear.
Hotfix/multi output filesystem (Hotfix/multi output filesystem LLNL/ygm#176)
Adds missing filesystem include to multi_output.hpp
Replaces csv_parser.hpp include in ndjson_parser.hpp with line_parser.hpp
Removes unnecessary ygm::io::detail namespace
Renamed 3 local comm member functions. (Renamed 3 local comm member functions. LLNL/ygm#173)
fix type of the get_ygm_ptr call in set.hpp (fix type of the get_ygm_ptr call in set.hpp LLNL/ygm#178)
fix type of set get_ygm_ptr
add a test for the ygm set pointer
also fix the set pointer for multiset
Hotfix/arrow_ci (Hotfix/arrow_ci LLNL/ygm#180)
Adding recent version of Arrow to CMakeLists.txt
make -j
Adds CMake flag to require a version of Arrow to be found, specifically for use in CI to make it obvious when Apache Arrow is not being found
Adds disjoint_set::clear() function (Adds disjoint_set::clear() function LLNL/ygm#183)
Feature/context aware progress (Feature/context aware progress LLNL/ygm#182)
Check calling context in comm.local_progress()
Fix spelling error and increase test trip count
Add Parquet -> JSON Converter (Add Parquet -> JSON Converter LLNL/ygm#181)
Change parquet parser wrt schema
Specifically, schema() returns physical type rather than logical type.
Add paquet -> json converter
Update Parquet JSON reader example
Arrow 14.0 (Arrow 14.0 LLNL/ygm#185)
Change the cmake file to support Apache Arrow v14.0
Fixes array partitioning and bag rebalancing to be more balanced. (Bugfix/array partitioning LLNL/ygm#187)
Copies all private variables in ygm::container::detail::array_impl copy constructor (Hotfix/array copy constructor LLNL/ygm#189)
Removes impl from ygm::container::array (Removes impl from ygm::container::array LLNL/ygm#190)
Parallel Parquet File Reading (v2) (Parallel Parquet File Reading (v2) LLNL/ygm#188)
Parallel parquet reader v2
Add comments
Add tests for parquet reader (Add tests for parquet reader LLNL/ygm#191)
Add tests for parquet reader
Fix wrong indent
Brush up on parque reader test
Updates Regarding Arrow Parquet (Updates Regarding Arrow Parquet LLNL/ygm#193)
Read values using std::optional in the ParquetToJson converter
Support Arrow v15
Bugfix/disjoint set compress (Bugfix/disjoint set compress LLNL/ygm#195)
Adds print statements to see if obvious bugs exist in disjoint_set::all_compress()
Removes race condition in disjoint_set::all_compress() by preparing all parent queries before sending any
Removing print statements from disjoint_set
Adds check of parent ranks before compressing disjoint sets
Adds check of parent ranks before compressing disjoint sets
Adds check of parent ranks before compressing disjoint sets
Avoid querying parent during disjoint_set::all_compress() when item is a root
Fixes missing reference for local_rep_query inside of update_rep_functor of disjoint_set::all_compress()
More detailed debugging output in all_compress()
Fixing logic for when a parent request comes for an item that had to query its own parent in current all_compress() round
Fixing logic for when a parent request comes for an item that had to query its own parent in current all_compress() round
Cleaning up all_compress() queries as they return
Cleaning up all_compress() queries as they return
Adding additional debug printing to disjoint_set::all_compress()
Avoids potential race condition where all_compress() is using whether parent of item being queried has returned in place of directly tracking whether item has found its root
Cleaning up disjoint_set::all_compress() code
Removing unused code
Add parquet2variant converter (Add parquet2variant convertor LLNL/ygm#194)
Bugfix/disjoint set compress (Bugfix/disjoint set compress LLNL/ygm#196)
Adds missing async_visit function to disjoint_set
Fixes stopping criteria on while loop in disjoint_set::all_compress()
Fixes tracking of ongoing compression updates (Fixes tracking of ongoing compression updates LLNL/ygm#197)
Updating version number to 0.6 (Updating version number to 0.6 LLNL/ygm#198)
Removes performance directory (Removes performance directory LLNL/ygm#199)
Fixing Github actions to run on pull requests to master branch