You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardexpand all lines: content/45.user_friendliness.md
+1-1
Original file line number
Diff line number
Diff line change
@@ -75,7 +75,7 @@ When loading a dataset, yt will attempt to determine what the format of the data
75
75
76
76
While these may seem like simple, obvious changes to make, they can hide difficult technical challenges, and more importantly, have dramatically improved the user experience for people using yt.
Project Jupyter is an overarching term for a collection of related projects that provide an extensive, end-to-end suite for the user experience of developing code and narrative, as described in depth in (among other papers) @doi:10.1109/MCSE.2021.3059263 and @soton403913.
81
81
While many in the yt community utilize yt through python scripts executed on the command line or through submission queues on high-performance computing resources, a large fraction utilize Jupyter Notebooks for their data exploration.
Copy file name to clipboardexpand all lines: content/50.halo_finding_and_catalogs.md
+1-1
Original file line number
Diff line number
Diff line change
@@ -6,7 +6,7 @@ Being able to identify halos, as well as their associated baryonic content, is n
6
6
Furthermore, convergence studies and cross-simulation comparisons requires a consistent method for identifying dark matter halos, as well as the ability to track their growth over time.
7
7
8
8
In past versions of `yt`, several specific halo finders were bundled and made available to work on any class of data `yt` was able to read.
9
-
These included the HOP halo finder, the classic Friends-of-Friends (FOF) halo finder [@doi:10.1086/191003], a scalable and Parallel HOP [@doi:10.1086/305535], and a wrapping of the ORIGAMI code [@doi:10.1142/9789814623995_0378] for filament identification.
9
+
These included the HOP halo finder, the classic Friends-of-Friends (FOF) halo finder [@doi:10.1086/191003], a scalable and Parallel HOP [@doi:10.1086/305535], and a wrapping of the ORIGAMI code [@doi:10.1142/9789814623995_0378] for filament identification.
10
10
To do so, `yt` would utilize direct in-memory connectors with these implementations; whereas typically data connectors are written for each individual dataset format for individual halo finding methods, this enabled a single connector to be written from `yt` to the halo finder.
11
11
In addition to these bundled halo finders, a direct in-memory interface with Rockstar [@doi:10.1088/0004-637X/762/2/109] was developed that sidestepped Rockstar's built in load-balancing to minimize data duplication and transfer.
Copy file name to clipboardexpand all lines: content/55.scaling_parallelism.md
+8-12
Original file line number
Diff line number
Diff line change
@@ -1,4 +1,4 @@
1
-
## Scaling and Parallelism
1
+
## Scaling and Parallelism
2
2
3
3
To support cases where data volume results in long processing time or large memory requirements, yt operations have been parallelized using the Message Passing Interface (MPI; @mpi40).
4
4
When designing the parallel interface for yt, as discussed in [@doi:10.1088/0067-0049/192/1/9], the design goals included ensuring that scripts required little to no adjustments to be run in parallel.
@@ -8,19 +8,18 @@ In the intervening time, the parallel operation infrastructure has been rewritte
8
8
9
9
Almost all of the operations in yt that are conducted in parallel follow a straightforward method of decomposing work and consolidating results:
10
10
11
-
1. Identify which chunking method (see @sec:chunking) is most appropriate for the operation.
12
-
2. Consolidate chunks according to IO minimization and assign to individual MPI tasks
13
-
3. Join (potentially applying reduction operations) final results to provide solution to *all tasks* in the group
11
+
1. Identify which chunking method (see @sec:chunking) is most appropriate for the operation.
12
+
2. Consolidate chunks according to IO minimization and assign to individual MPI tasks
13
+
3.Join (potentially applying reduction operations) final results to provide solution to _all tasks_ in the group
14
14
15
15
The final step, of joining across tasks, results in the final set of values being accessible to all tasks; this is not a universal "final step" in parallel operations, and in some cases results in substantial duplication of memory.
16
16
This compromise was accepted as a result of the design goals of ensuring that scripts can run unmodified.
17
17
18
-
The parallelism in yt heavily leans upon the "index" for a dataset either being available *already* at initiation time on all tasks, or that index being *accessible* through IO operations or fast generation.
18
+
The parallelism in yt heavily leans upon the "index" for a dataset either being available _already_ at initiation time on all tasks, or that index being _accessible_ through IO operations or fast generation.
19
19
This provides a degree of load-balancing that can be conducted, as estimates of memory and processing requirements are available on all tasks (and thus the load-balancing calculations are deterministic across all tasks).
20
20
In essence, this means that for grid-based datasets, the entire grid hierarchy is available on all processors; for octrees or particle datasets, it means that at least a rough estimate of the distribution of values must be available (and identical) on all processors.
21
21
This doesn't prevent opaquely distributed datasets from being decomposed, but it does allow datasets whose distribution is well-described to be decomposed with greater precision.
22
22
23
-
24
23
### Multi-Level Parallelism
25
24
26
25
In its original implementation of parallelism, yt utilized a single, global MPI communicator (`MPI_COMM_WORLD`).
@@ -31,7 +30,7 @@ For example, when conducting halo finding and analysis (see @sec:halo_finding) y
31
30
This takes place by specifying a task size at the top level (or allowing yt's internal heuristics to determine it) and then distributing work to sub-communicators, each of which is then used for decomposition inside that top-level task.
32
31
33
32
In addition to multi-level communicators, yt utilizes OpenMP constructs exposed in Cython in several places.
34
-
This includes in the software volume rendering (see @sec:software-volume-rendering), in the pixelization operations for SPH data (see @sec:sph-analysis), calculation of gravitational binding energy (see @sec:analysis-modules) and for computing the bounding volume hierarchy for rendering finite element meshes (see @sec:unstructured-mesh).
33
+
This includes in the software volume rendering (see @sec:software-volume-rendering), in the pixelization operations for SPH data (see @sec:sph-analysis), calculation of gravitational binding energy (see @sec:analysis-modules) and for computing the bounding volume hierarchy for rendering finite element meshes (see @sec:unstructured_mesh).
35
34
In some instances, the Cython interface to OpenMP has had unpredictable performance implications; owing to this, the usage of OpenMP within yt has been somewhat conservative.
36
35
37
36
### Parallelism Interfaces
@@ -42,7 +41,7 @@ This parallelism is instrumented through the use of the yt "chunking" interface,
42
41
The high-level interface to the `DerivedQuantity` subclasses computes the data chunks in the source data object and then assigns these to individual MPI tasks in the current top-level communicator.
43
42
Each initializes storage space for the intermediate values, iterates over its assigned chunks and constructs intermediate reductions, and then the finalization step involves broadcasting the values to all other tasks and completing the final set of operations.
44
43
For projections, the procedure is very similar; those datasets with an index duplicated across MPI tasks (such as patch-based grid datasets) are collapsed along a dimension and each MPI task fills in the values, which are then reduced through a broadcast operation.
45
-
Utilizing these operations requires *no* modifications to user-facing code other than a call to `yt.enable_parallelism()` at the start of the script.
44
+
Utilizing these operations requires _no_ modifications to user-facing code other than a call to `yt.enable_parallelism()` at the start of the script.
46
45
47
46
The user-facing parallel constructs allow for somewhat greater flexibility in defining parallel task decomposition.
48
47
Many objects in yt, particularly those such as the `DatasetSeries` object, have constituent data objects on which analysis can be conducted.
@@ -72,8 +71,6 @@ For many types of data analysis, particularly those operations conducted across
72
71
73
72
### Performance of Operations
74
73
75
-
76
-
77
74
### Inline Analysis
78
75
79
76
It is possible to instrument a simulation code to call Python routines inline during its execution.
@@ -84,9 +81,8 @@ In these cases, `yt` did not pass around datasets between MPI tasks, but rather
84
81
Within Enzo, all of the communication between Python and C++ was managed through Enzo's usage of the C API.
85
82
This required some knowledge of how Python conducts garbage collection, and required ensuring that reference counting was managed correctly to avoid memory leaks.
86
83
87
-
This non-standardized approach to conducting *in situ* visualization led to the creation and development of the library `libyt` which serves as an intermediary layer between simulation codes and `yt` (and Python in general.)
84
+
This non-standardized approach to conducting _in situ_ visualization led to the creation and development of the library `libyt` which serves as an intermediary layer between simulation codes and `yt` (and Python in general.)
88
85
This library encapsulates all Python API calls, manages references, and provides a systematic method for providing data pointers to Python.
89
86
`libyt` provides a stable C-based API, and is accessible from numerous different languages.
90
87
It also provides a custom-built `yt` frontend for accepting data.
91
88
A more complete description is outside the scope of this paper, and we refer the reader to (**MJT: cite in prep manuscript**).
Copy file name to clipboardexpand all lines: content/70.sustainability.md
+1-1
Original file line number
Diff line number
Diff line change
@@ -11,7 +11,7 @@ However, at the risk of belaboring a point that has been well-explored elsewhere
11
11
A tension exists, however, between support of an existing project and the support of new projects in an ecosystem.
12
12
By supporting an existing project, resources can tend to become concentrated; conversely, if a project supports a broader research agenda, that resource concentration can result in greater effort-multipliers for individuals who utilize the project.
13
13
We're aware of this tension in yt; in fact, while yt has been grant-supported, most of the grant development has gone to a very small number of groups.
14
-
This grant funding has been provided through the National Science Foundation, the Gordon and Betty Moore Foundation, the Department of Energy, the Chan Zuckerberg Initiative and other sources. [@doi:10.6084/m9.figshare.2061465.v1, [@doi:10.6084/m9.figshare.909413.v1], [@doi:10.5281/zenodo.4158589].
14
+
This grant funding has been provided through the National Science Foundation, the Gordon and Betty Moore Foundation, the Department of Energy, the Chan Zuckerberg Initiative and other sources. [@doi:10.6084/m9.figshare.2061465.v1], [@doi:10.6084/m9.figshare.909413.v1], [@doi:10.5281/zenodo.4158589].
15
15
Grants have supported the development of new features, including specific functionality for analysis routines and support for non-astronomical domains.
16
16
17
17
Into each of these grants has been explicit support for community building, constituted by the development of documentation, videos, and tutorials, as well as mentoring of new contributors and shepherding the growth of the project through code review and issue management.
0 commit comments