forked from mfem/mfem
-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathCHANGELOG
1979 lines (1531 loc) · 95.7 KB
/
CHANGELOG
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Finite Element Discretization Library
__
_ __ ___ / _| ___ _ __ ___
| '_ ` _ \ | |_ / _ \| '_ ` _ \
| | | | | || _|| __/| | | | | |
|_| |_| |_||_| \___||_| |_| |_|
https://mfem.org
Version 4.2.1 (development)
===========================
- Added initial support for GPU-accelerated versions of PETSc that works with
MFEM_USE_CUDA if PETSc has been configured with CUDA support. Examples 1 and 9
in the examples/petsc directory have been modified to work with --device cuda.
Examples with GAMG (ex1p) and SLEPc (ex11p) are also provided.
- Memory management:
* Added method Device::SetMemoryTypes that can be used to change the default
host and device MemoryTypes before Device setup.
* In class MemoryManager, added methods GetDualMemoryType and
SetDualMemoryType; dual MemoryTypes are used to determine the second
MemoryType (host or device) when only one MemoryType is specified in methods
of class Memory.
* Added Memory constructor for setting both the host and device MemoryTypes.
* Switched the default behavior of device memory allocations so that they
are deferred until the device pointer is needed.
* Added a second Umpire device MemoryType, DEVICE_UMPIRE_2, with
corresponding allocator that can be set with the method
MemoryManager::SetUmpireDevice2AllocatorName.
* Added HOST_PINNED MemoryType and a pinned host allocator for CUDA and HIP.
- Added support for Caliper: a library to integrate performance profiling
capabilities into applications. See examples/caliper for more details.
- Added support for explicit vectorization in the high-performance templated
code for Fujitsu's A64FX ARM microprocessor architecture.
- Added AlgebraicCeedSolver that does matrix-free algebraic p-multigrid for
diffusion problems with the Ceed backend.
- Introduced new options for the mesh-explorer miniapp to visualize the actual
element attributes in parallel meshes while retaining the visualization of
the domain decomposition.
- Introduced solver interface for linear problems with constraints, a few
concrete solvers that implement the interface, and a demonstration of their
use in Example 28(p), which solves an elasticity problem with zero normal
displacement (but allowed tangential displacement) on two boundaries.
- Added high-order matrix-free auxiliary Maxwell solver for H(curl) problems,
as described in Barker and Kolev 2020 (https://doi.org/10.1002/nla.2348). See
Example 3p and linalg/auxiliary.?pp.
- Added a new miniapp block-solvers that compares the performance of various
solvers for mixed finite element discretization of the second order scalar
elliptic equations. Currently available solvers in the miniapp include a
block-diagonal preconditioner that is based on approximate Schur complement
(implemented in ex5p), and a newly implemented solver DivFreeSolver, which
exploits a multilevel decomposition of the Raviart-Thomas space and its
divergence-free subspace. See the miniapps/solvers directory for more details.
- Added a new miniapp for computing (signed) distance functions to a point
source or zero level set. See miniapps/shifted/distance.cpp.
- Added matrix-free GPU-enabled implementations of GradientInterpolator and
IdentityInterpolator.
- Added interface to MUMPS direct solver. Its usage is demonstrated in ex25p.
See http://mumps.enseeiht.fr/ for more details. Supported versions >= 5.1.1.
- Added three ESDIRK time integrators: implicit trapezoid rule, L-stable
ESDIRK-32, and A-stable ESDIRK-33.
- Introduced a new non-conforming mesh format that fixes known inconsistencies
of legacy "MFEM mesh v1.1" NC format and works consistently in both serial and
parallel. ParMesh::ParPrint can now print non-conforming AMR meshes that can
be used to restart a parallel AMR computation. Example 6p has been extended to
demonstrate restarting from a previously saved checkpoint. Note that parallel
NC data files are compatible with serial code, e.g., can be viewed with serial
GLVis. Loading of legacy NC mesh files is still supported.
- Added support for 1D non-conforming meshes (which can be useful for parallel
load balancing and derefinement).
- Added a "scaled Jacobian" visualization option in the Mesh Explorer miniapp to
help identify elements with poor mesh quality.
- Added support for the "BR2" discontinuous Galerkin discretization for
diffusion via DGDiffusionBR2Integrator (see Example 14/14p).
- Generalized the Multigrid class to support non-geometric multigrid. The
previous functionality, based on FiniteElementSpaceHierarchy, is now available
in the derived class GeometricMultigrid.
- Upgraded the Catch unit test framework from version 2.13.0 to version 2.13.2.
- The TMOP mesh optimization algorithms were extended to GPU:
- QualityMetric #1, #2, #7 and #77 are available in 2D, #302, #303, #315
and #321 in 3D
- Both AnalyticAdaptTC and DiscreteAdaptTC TargetConstructor are available
- Kernels for normalization and limiting have been added
- The AdvectorCG now also supports AssemblyLevel::PARTIAL
- Added a new command line boolean option (`--all`) to the unit tests to launch
*all* non-regression tests.
- Added support for different modes of QuadratureInterpolator on GPU.
The layout (QVectorLayout::byNODES|byVDIM) and the tensor products modes can
be enabled before calling the Mult, Values, Derivatives, PhysDerivatives and
Determinants methods.
- Implemented a filter method for the Navier miniapp to stabilize highly
turbulent flows in direct numerical simulation.
- Added HIP support to the CMake build system.
- Added support for reading high-order Lagrange meshes in VTK format. Arbitrary-
orders and all element types are supported. See the VTK blog for more info:
https://blog.kitware.com/wp-content/uploads/2018/09/Source_Issue_43.pdf.
- Added support for reading VTK meshes in XML format.
- Added partial assembly and device support to Example 25/25p, with diagonal
preconditioning.
- Implemented a variable step-size IMEX (VSSIMEX) method for the Navier miniapp.
- Added new mesh quality metrics and improved the untangling capabilities of the
TMOP-based mesh optimization algorithms.
- Added convective and skew-symmetric integrators for the nonlinear term in the
Navier-Stokes equations.
- Added new miniapp directory mtop/ with optimization-oriented block parametric
non-linear form and abstract integrators. Two new miniapps, ParHeat and
SeqHeat, demonstrate parallel and sequential implementation of gradients
evaluation for linear diffusion with discrete density.
- Changed the interface for the error estimator.
- Implemented the parallel Kelly error indicator for scalar-valued problems.
- Added new classes DenseSymmetricMatrix and SymmetricMatrixCoefficient for
efficient evaluation of symmetric matrix coefficients. This replaces the now
deprecated EvalSymmetric in MatrixCoefficient. Added DiagonalMatrixCoefficient
for clarity, which is a typedef of VectorCoefficient.
- Added support for AMG preconditioners for non-symmetric systems (e.g.
advection-dominated problems) using hypre's approximate ideal restriction
(AIR) AMG. Requires hypre version 2.14.0 or newer. Usage is illustrated in
example 9/9p.
- Implemented an adaptive linear solver tolerance option for NewtonSolver based
on the algorithm of Eisenstat and Walker.
- Added support for nonscalar coefficient with VectorDiffusionIntegrator.
- Extending support for L2 basis functions using MapTypes VALUE and INTEGRAL in
linear interpolators and GridFunction "GetValue" methods.
- Variable order spaces, p- and hp-refinement. This is the initial (serial)
support for variable-order FiniteElementCollection and FiniteElementSpace.
The new method FiniteElementSpace::SetElementOrder can be called to set an
arbitrary order for each mesh element. The conforming interpolation matrix
will now automatically constrain p- and hp- interfaces, enabling general
hp-refinement in both 2D and 3D, on uniform or mixed NC meshes. Support for
parallel variable-order spaces will follow shortly.
- Added support for creating refined meshes for all element types (e.g. by
splitting high-order elements into low-order refined elements), including
mixed meshes. The LOR Transfer miniapp (miniapps/tools/lor-transfer.cpp) now
supports meshes with any element geometry.
- Gitlab CI: use Spack (and Uberenv) to automate the build of TPLs.
- Added new miniapps demonstrating: 1) the use of GSLIB for overlapping grids,
see gslib/schwarz_ex1, and 2) coupling different physics in different domains,
see navier/cht. Note that gslib v1.0.7 is require (see INSTALL for details).
- Added a new, very simple example (ex0 and parallel version ex0p). This
example solves a simple Poisson problem using H1 elements (the same problem as
ex1), but is intended to be extremely simple and approachable for new users.
- Meshes consisting of any type of elements (including mixed meshes) can be
converted to all-simplex meshes using Mesh::MakeSimplicial.
- Several of the mesh constructors (creating Cartesian meshes, refined (LOR)
meshes, simplex meshes, etc.) are now available as "named constructors", e.g.
Mesh::MakeCartesian2D or Mesh::MakeRefined. The legacy constructors are marked
as deprecated.
- Added support for creating periodic meshes with Mesh::MakePeriodic. The
requisite periodic vertex mappings can be created with
Mesh::CreatePeriodicVertexMapping.
- Added support for transferring dual fields between high-order and low-order
refined finite element spaces using the transposed versions of the
L2ProjectionGridTransfer operators. This functionality is illustrated in the
lor-transfer miniapp.
- Improved interface for using the Ginkgo library, including: support for matrix-
free operators in Ginkgo solvers, new wrappers for Ginkgo preconditioners, HIP
support, and reduction of unnecessary data copies.
- Added initial support for hypre's mixed integer (mixedint) capability, which
uses different data types for local and global indices in order to save memory
in large problems. This capability requires that hypre was configured with the
--enable-mixedint option. Note that this option is currently tested only in
ex1p and may not work in more general settings.
- Added new functionality for constructing low-order refined discretizations and
solvers, see the LORDiscretization and LORSolver classes. A new basis type for
H(curl) and H(div) spaces is introduced to give spectral equivalence. This
functionality is illustrated in the LOR solvers miniapp in miniapps/solvers.
- Added sample meshes in the `data` subdirectory showing the reference elements
of the six currently supported element types; ref-segment.mesh,
ref-triangle.mesh, ref-square.mesh, ref-tetrahedron.mesh, ref-cube.mesh, and
ref-prism.mesh.
- Added a high-order extension of the shifted boundary method to solve PDEs on
non body-fitted meshes. This is illustrated in the new Shifted Diffusion
miniapp, see miniapps/shifted/diffusion.cpp.
libCEED integration improvements
--------------------------------
- Refactor the libCEED integration
- Add support for VectorCoefficient with libCEED backends.
- Add support for ConvectionIntegrator, and VectorConvectionNLFIntegrator with
libCEED backends.
Version 4.2, released on October 30, 2020
=========================================
High-Performance Computing
--------------------------
- Added support for explicit vectorization in the high-performance templated
code, which can now take advantage of specific classes on the following
architectures:
* x86 (SSE/AVX/AVX2/AVX512),
* Power8 & Power9 (VSX),
* BG/Q (QPX).
These are disabled by default, but can be enabled with MFEM_USE_SIMD=YES.
See the new file linalg/simd.hpp and the new directory linalg/simd.
- Added an Element Assembly mode compatible with GPU device execution for H1 and
L2 spaces in the mass, convection, diffusion, transpose, and the face DG trace
integrators. See option '-ea' in Example 9. When enabled, this assembly level
stores independent dense matrices for the elements, and independent dense
matrices for the faces in the DG case.
- Added a Full Assembly mode compatible with GPU device execution. This assembly
level builds on top of the Element Assembly kernels to compute a global sparse
matrix. All integrators supported by element assembly are also supported by
full assembly. See the '-fa' option in Example 9.
- Optimized the AMD/HIP kernel support and enabled HIP support in the libCEED
integration. This is now available via the "ceed-hip" device backend.
- Improved the libCEED integration to support:
* AssemblyLevel::NONE for Mass, Diffusion, VectorMass, and VectorDiffusion
Integrators. This level computes the full operator evaluation "on the fly".
* VectorMassIntegrator and VectorDiffusionIntegrator.
* All types of (scalar) Coefficients.
- Added partial assembly / device support for:
* H(div) bilinear forms and VectorFEDivergenceIntegrator.
* BlockOperator, see the updated Example 5.
* Complex operators, see the updated Example 22.
* Chebyshev accelerated polynomial smoother.
* Convergent diagonal preconditioning on non-conforming adaptively refined
meshes, see Example 6/6p.
- Added CUDA support for:
* Sparse matrix-vector multiplication with cuSPARSE,
* SUNDIALS ODE integrators, see updated SUNDIALS modification of Example 9/9p.
Linear and nonlinear solvers
----------------------------
- Added a new solver class for simple integration with NVIDIA's multigrid
library, AmgX. The AmgX class is designed to work as a standalone solver or
preconditioner for existing MFEM solvers. It uses MFEM's sparse matrix format
for serial runs and the HypreParMatrix format for parallel runs. The new
solver may be configured to run with one GPU per MPI rank or with more MPI
ranks than GPUs. In the latter case, matrices and vectors are consolidated to
ranks communicating with the GPUs and the solution is then broadcasted.
Although CUDA is required to build, the AmgX support is compatible with the
MFEM CPU device configuration. The examples/amgx folder illustrates how to
integrate AmgX in existing MFEM applications. The AmgX solver class is
partially based on: "AmgXWrapper: An interface between PETSc and the NVIDIA
AmgX library", by Pi-Yueh Chuang and Lorena A. Barba, doi:10.21105/joss.00280.
- Added initial support for geometric h- and p-multigrid preconditioners for
matrix-based and matrix-free discretizations with basic GPU capability, see
Example 26/26p.
- Added support for the CVODES package in SUNDIALS which provides ODE solvers
with sensitivity analysis capabilities. See the CVODESSolver class and the new
adjoint miniapps in the miniapps/adjoint directory.
- Added an interface to the MKL CPardiso solver, an MPI-parallel sparse direct
solver developed by Intel. See Example 11p for an illustration of its usage.
- Added support for the SLEPc eigensolver package, https://slepc.upv.es.
- Upgraded SuperLU interface to use SuperLU_DIST 6.3.1. Added a simple SuperLU
example in the new directory examples/superlu.
- Extended the KINSOL (SUNDIALS) nonlinear solver interface to support the
Jacobian-free Newton-Krylov method. A usage example is shown in Example 10p.
- Block arrays of parallel matrices can now be merged into a single parallel
matrix with the function HypreParMatrixFromBlocks. This could be useful for
solving block systems with parallel direct solvers such as STRUMPACK.
- Added CUDA support for SUNDIALS ODE integrators. See the updated SUNDIALS
modification of Example 9/9p.
- Added wrappers for hypre's flexible GMRES solver and the new parallel ILU
preconditioner. The latter requires hypre version 2.19.0 or later.
Discretization improvements
---------------------------
- Extended GSLIB-FindPoints integration to support simplices and interpolation
of functions from L2, H(div) and H(curl) spaces.
- Added support for computing asymptotic error estimates and convergence rates
for the whole de Rham sequence based on the new class ConvergenceStudy and new
member methods in GridFunction and ParGridFunction. See the rates.cpp file in
the tests/convergence directory for sample usage.
- Extended the GetValue and GetVectorValue methods of GridFunction to support
evaluation on boundary elements and, in the continuous field case, arbitrary
mesh edges and faces. This requires passing an ElementTransformation argument.
- Added support for matrix-free interpolation and restriction operators between
continuous H1 finite element spaces of different order on the same mesh or
with the same order on uniformly refined meshes.
- The Coefficient classes based on a C-function pointer (FunctionCoefficient,
VectorFunctionCoefficient and MatrixFunctionCoefficient) now use the more
general std::function class template. This allows the classes to be backward
compatible (i.e. they can still work with C-functions) and, in addition,
support any "callable", e.g. lambda functions.
- Non-conforming meshes are now supported with block nonlinear forms. See the
updated Example 19/19p.
Meshing improvements
--------------------
- The graph linear ordering library Gecko, previously an external dependency, is
now included directly in MFEM. As a result, Mesh::GetGeckoElementOrdering is
always available. The interface has also been improved, see for example the
Mesh Explorer miniapp.
- Added support for finite difference-based gradient and Hessian approximation
in the TMOP mesh optimization algorithms. This improves the accuracy of the
Hessian for r-adaptivity using discrete fields, and allows use of skewness
and orientation based metrics.
- Improved Gmsh reader (version 2.2), which now supports both high-order and
periodic meshes. Segments, triangles, quadrilaterals, and tetrahedra are
supported up to order 10. Wedges and hexahedra are supported up to order 9.
For sample periodic meshes, see the periodic*.msh files in the data directory.
- Added support for construction of (serial) non-conforming meshes. Hanging
nodes can be marked with Mesh::AddVertexParents when building the mesh with
the "init" constructor. The usage is demonstrated in a new meshing miniapp,
Polar NC, which generates meshes that are non-conforming from the start.
- Added support for r-adaptivity with more than one discrete field. This allows
the user to specify different discrete functions for controlling the size,
aspect-ratio, orientation, and skew of elements in the mesh.
- Additional TMOP improvements:
* Capability for approximate tangential mesh relaxation.
* Support and examples for using TMOP on mixed meshes.
* Complete integrator action accounting for spatial derivatives of discrete
and analytic targets.
New and updated examples and miniapps
-------------------------------------
- Added a new miniapp, Navier, that solves the time-dependent Navier-Stokes
equations of incompressible fluid dynamics. See the miniapps/navier directory
for more details.
- Added 10 new example codes:
* Example 25/25p demonstrates the use of a Perfectly Matched Layer (PML) for
electromagnetic wave propagation (indefinite Maxwell).
* Example 26/26p shows how to construct matrix-free geometric and p-multigrid
preconditioner for the Laplace problem.
* Example 27/27p demonstrates the enforcement of Dirichlet, Neumann, Robin,
and periodic boundary conditions with either H1 or DG Laplace problems.
* Versions of Example 1/1p in examples/amgx demonstrating the use of AmgX,
to solve the Laplace problem with AMG preconditioning on GPUs.
* A version of Example 11p in examples/petsc demonstrating the use of SLEPc,
to solve the Laplace eigenproblem with shift-and-invert transformation.
* A version of Example 1 in examples/superlu demonstrating the use of SuperLU
to solve the Laplace problem.
- Added a new Field Interpolation miniapp in miniapps/gslib that demonstrates
transfer of grid functions between different meshes using GSLIB-FindPoints.
- Added 2 miniapps in the new miniapps/adjoint directory demonstrating how to
solve adjoint problems in MFEM using the CVODES package in SUNDIALS. Both of
these miniapps require the MFEM_USE_SUNDIALS configuration option.
* The cvsRoberts_ASAi_dns miniapp solves a backward adjoint problem for a
system of ODEs, evaluating both forward and adjoint quadratures in serial.
* The adjoint_advection_diffusion miniapp solves a backward adjoint problem
for an advection diffusion PDE, evaluating adjoint quadratures in parallel.
- Added 4 additional meshing miniapps:
* The Minimal Surface miniapp solves Plateau's problem: the Dirichlet problem
for the minimal surface equation.
* The Twist miniapp demonstrates how to stitch together opposite surfaces of a
mesh to create a topologically periodic mesh.
* The Trimmer miniapp trims away parts of a mesh based on element attributes.
* Polar NC shows the construction of polar non-conforming meshes.
- Several examples and miniapps were updated to include:
* Full and element assembly support in Example 9/9p.
* Partial assembly with diagonal preconditioning in Examples 4/4p/5/5p/22/22p.
* Diagonal preconditioner in Example 6/6p for partial assembly with AMR.
* The option to plot a function in Mesh Explorer.
* A new test problem showing a mixed bilinear form for H1, H(curl), H(div) and
L2, with partial assembly support in Example 24/24p.
* Weak Dirichlet boundary conditions (Nitsche) to the NURBS miniapp.
Data management and Visualization
---------------------------------
- Added support for ADIOS2 for parallel I/O with ParaView visualization. See
Examples 5, 9, 12, 16. The classes adios2stream and ADIOS2DataCollection
provide the interface to generate ADIOS2 Binary Pack (BP4) directory datasets.
- Added VTU output of boundary elements and attributes and parallel VTU (PVTU)
output of parallel meshes for visualization using ParaView.
Improved testing
----------------
- Added a GitLab pipeline that automates PR testing on supercomputing systems
and Linux clusters at Lawrence Livermore National Lab (LLNL). This can be
triggered only by LLNL developers, see .gitlab-ci.yml, the .gitlab directory
and the updated CONTRIBUTING.md file.
- Added additional testing for convergence, the parallel mesh I/O, and for the
libCEED integration in MFEM in the tests/ directory.
- Upgraded the Catch unit test framework from version 1.6.1 to version 2.13.0.
Miscellaneous
-------------
- Renamed "Backend::DEBUG" to "Backend::DEBUG_DEVICE" to avoid conflicts,
as DEBUG is sometimes used as a macro.
- Added a new IterativeSolverMonitor class that allows to monitor the residual
and solution with an IterativeSolver after every iteration.
- Added power method to iteratively estimate the largest eigenvalue and the
corresponding eigenvector of an operator.
- Added support for face integrals on the boundaries of NURBS meshes.
- In SLISolver, changed the residual inner product from (Br,r) to (Br,Br) so the
solver can work with non-SPD preconditioner B.
- Added new coefficient and vector coefficient classes for QuadratureFunctions,
with new LinearForm integrators which use them.
- The integration order used in the ComputeLpError and ComputeElementLpError
methods of class GridFunction has been increased.
- Change the IntegrationRule inside VectorDiffusionIntegrator to use the same
quadrature as DiffusionIntegrator.
- The README.html files previously included in several source directories have
been removed. Use the corresponding pages at mfem.org instead.
- Various other simplifications, extensions, and bugfixes in the code.
Version 4.1, released on March 10, 2020
=======================================
Starting with this version, the MFEM open source license is changed to BSD-3.
Improved GPU capabilities
-------------------------
- Added initial support for AMD GPUs based on HIP: a C++ runtime API and kernel
language that can run on both AMD and NVIDIA hardware.
- Added support for Umpire, a resource management library that allows the
discovery, provision, and management of memory on machines with multiple
memory devices like NUMA and GPUs, see https://github.com/LLNL/Umpire.
- GPU acceleration is now available in 3 additional examples: 3, 9 and 24.
- Improved RAJA backend and multi-GPU MPI communications.
- Added a "debug" device designed specifically to aid in debugging GPU code by
following the "device" code path (using separate host/device memory spaces and
host <-> device transfers) without any GPU hardware.
- Added support for matrix-free diagonal smoothers on GPUs.
- The current list of available device backends is: "ceed-cuda", "occa-cuda",
"raja-cuda", "cuda", "hip", "debug", "occa-omp", "raja-omp", "omp",
"ceed-cpu", "occa-cpu", "raja-cpu", and "cpu".
- The MFEM memory manager now supports different memory types, associated with
the following memory backends:
* Default host memory, using standard C++ new and delete,
* CUDA pointers, using cudaMalloc and HIP pointers, using hipMalloc,
* Managed CUDA/HIP memory (UVM), using cudaMallocManaged/hipMallocManaged,
* Umpire-managed memory, including memory pools,
* 32- or 64-byte aligned memory, using posix_memalign (WIN32 also supported),
* Debug memory with mmap/mprotect protection used by the new "debug" device.
libCEED support
---------------
- Added support for libCEED, the portable library for high-order operator
evaluation developed by the Center for Efficient Exascale Discretizations in
the Exascale Computing Project, https://github.com/CEED/libCEED.
- This initial integration includes Mass and Diffusion integrators. libCEED GPU
backends can be used without specific MFEM configuration, however it is highly
recommended to use the "cuda" build option to minimize memory transfers.
- Both CPU and GPU modes are available as MFEM device backends (ceed-cpu and
ceed-cuda), using some of the best performing CPU and GPU backends from
libCEED, see the sample runs in examples 1 and 6.
- NOTE: The current default libCEED GPU backend (ceed-cuda) uses atomics and
therefore is non-deterministic.
Partial assembly and matrix-free discretizations
------------------------------------------------
- The support for matrix-free methods on both CPU and GPU devices based on a
partially assembled operator decomposition was extended to include:
* DG integrators, (for now only in the Gauss-Lobatto basis), see Example 9,
* H(curl) bilinear forms, see Example 3,
* vector mass and vector diffusion bilinear integrators,
* convection integrator with improved performance,
* gradient and vector divergence integrators for Stokes problems,
* initial partial assembly mode for NonlinearForms.
- Diagonals of partially assembled operators can now be computed efficiently.
See the new methods AssembleDiagonal in BilinearForm, AssembleDiagonalPA in
BilinearFormIntegrator and the implementations in fem/bilininteg_*.cpp.
- In many examples, the partial assembly algorithms provide significantly
improved performance, particularly in high-order 3D runs on GPUs.
Meshing improvements
--------------------
- The algorithms for mesh element numbering were changed to have significantly
better caching and parallel partitioning properties. Both initial (see e.g.
Mesh::GetHilbertElementOrdering) and ordering after uniform refinement were
improved. NOTE: new ordering can have a round-off effect on solver results.
- Added support for non-conforming AMR on both prisms and tetrahedra, including
coarsening and parallel load balancing. Anisotropic prism refinement is only
available in the serial version at the moment.
- The TMOP mesh optimization algorithms were extended to support r-adaptivity.
Target matrices can now be constructed either via a given analytical function
(e.g. spatial dependence of size, aspect ratio, etc., for each element) or via
a (Par)GridFunction specified on the original mesh.
- The TMOP algorithms were also improved to support non-conforming AMR meshes.
- Added support for creating refined versions of periodic meshes, making use of
the new L2ElementRestriction class. This class also allows for computing
geometric factors on periodic meshes using partial assembly.
Discretization improvements
---------------------------
- Added support for GSLIB-FindPoints, a general high-order interpolation utility
that can robustly evaluate a GridFunction in an arbitrary collection of points
in physical space. See INSTALL for details on building MFEM with GSLIB, and
miniapps/gslib for examples of how to use this feature.
- Added support for complex-valued finite element operators and fields using a
2x2 block structured linear system to mimic complex arithmetic. New classes
include: ComplexGridFunction, SesquilinearForm, ComplexLinearForm, and their
parallel counterparts.
- Added second order derivatives of NURBS shape functions.
- Added support for serendipity elements of arbitrary order on affinely-mapped
square elements. Basis functions for these elements can be visualized using
an option in the display-basis miniapp.
- Two integrators related to Stokes problems, (Q grad u, v) and (Q div v, u),
where u and the components of v are in H1, were added/modified to support full
and partial assembly modes. See the new GradientIntegrator and the updated
VectorDivergenceIntegrator classes in fem/bilininteg.hpp, as well as the PA
kernels in fem/bilininteg_gradient.cpp and fem/bilininteg_divergence.cpp.
- Added a nonlinear vector valued convection integrator (Q u \cdot grad u, v)
where u_i and v_i are in H1. This form occurs e.g. in the Navier-Stokes
equations. The integrator supports the partial assembly mode for its
action. In full assembly mode we also provide the GetGradient method that
computes the linearized version of the integrator.
- Added a new method, MixedBilinearForm::FormRectangularLinearSystem, that can
be used to impose boundary conditions on the non-square off-diagonal blocks of
a block operator (similar to FormLinearSystem in the square case).
Linear and nonlinear solvers
----------------------------
- Added support for Ginkgo, a high-performance linear algebra library for GPU
and manycore nodes, with a focus on sparse solution of linear systems. For
more details see linalg/ginkgo.hpp and the example code in examples/gingko.
- Added support for HiOp, a lightweight HPC solver for nonlinear optimization
problems, see class HiOpNLPOptimizer and the example codes in examples/hiop.
- Added a general interface for specifying and solving nonlinear constrained
optimization problems through the new classes OptimizationProblem and
OptimizationSolver, see linalg/solver.hpp.
- Added a block ILU(0) preconditioner for DG-type discretizations. Example 9
(DG advection) now takes advantage of this for implicit time integration.
- New time integrators: Adams-Bashforth, Adams-Moulton and several integrators
for 2nd order ODEs, see the new Example 23.
- Added a LinearSolve(A,X) convenience method to solve dense linear systems. In
the trivial cases, i.e., square matrices of size 1 or 2, the system is solved
directly, otherwise, LU factorization is employed.
New and updated examples and miniapps
-------------------------------------
- Added a collection of 7 playful miniapps in miniapps/toys that illustrate the
meshing and visualization features of the library in more relaxed settings.
The toys include simulations of cellular automata, Rubik's cube, Mandelbrot
set, a tool to convert any image to mfem mesh, and more.
- Added 8 new example codes:
* Example 22/22p demonstrates the use of the new complex-valued finite element
operators by defining and solving a family of time-harmonic PDEs related to
damped harmonic oscillators.
* Example 23 solves a simple 2D/3D wave equation with the new second order
time integrators.
* Example 24/24p demonstrates usage of mixed finite element spaces in bilinear
forms. Partial assembly is supported in this example.
* A version of Example 1 in examples/ginkgo demonstrating the use of the
Gingko interface to solve a linear system.
* A version of Example 9/9p in examples/hiop demonstrating the nonlinear
constrained optimization interface and use of the SLBQP and HiOp solvers.
- Added two new miniapps: Find Points and Field Diff in miniapps/gslib that show
how GSLIB-FindPoints can be used to interpolate a (Par) GridFunction in an
arbitrary number of physical space points in 2D and 3D. The GridFunction must
be in H1 and in the same space as the mesh that is used to find the points.
- Added a simple miniapp, Get Values, that extracts field values at a set of
points, from previously saved data via DataCollection classes.
- Several examples and miniapps were updated:
* Added device support in Example 3/3p and Example 9/9p.
* Example 1/1p and Example 3/3p now use diagonal preconditioning in partial
assembly mode.
* Example 9/9p now supports implicit time integration, using the new block
ILU(0) solvers as preconditioners for the linear system.
* The mesh-optimizer and pmesh-optimizer miniapps now include the new
r-adaptivity capabilities of TMOP. They were also updated to support mesh
optimization on non-conforming AMR meshes.
* New options to reorder and partition the mesh and boundary attribute
visualization (key 'b') are now available in the mesh-explorer miniapp.
- Collected object files from the miniapps/common directory into a new library,
libmfem-common for the convenience of application developers. The new library
is now used in several miniapps in the electromagnetic and tools directories.
Improved testing
----------------
- Added a large number of unit tests in the tests/unit directory, including
several parallel unit tests.
- Added a new directory, tests/scripts, with several shell scripts that perform
simple checks on the code including: code styling, documentation formatting,
proper use of .gitignore, and preventing the accidental commit of large files.
- It is recommended that developers run the above tests scripts (via the runtest
script) before pushing to GitHub. See the README file in tests/scripts.
- The Travis CI settings have been updated to include an initial Checks stage
which currently runs the code-style, documentation and gitignore test scripts,
as well as a final stage for optional checks/tests which currently runs the
branch-history script.
Miscellaneous
-------------
- Added support for output in the ParaView XML format. Both low-order and
high-order Lagrange elements are supported. Output can be in ASCII or binary
format. The binary output can be compressed if MFEM is compiled with zlib
support (MFEM_USE_ZLIB). See the new ParaViewDataCollection class and the
updated Examples 5/5p and 9/9p.
- Upgraded the SUNDIALS interface to utilize SUNDIALS 5.0. This necessitated a
complete rework of the interface and requires changes at the application
level. Example usage of the new interface can be found in examples/sundials.
- Switched from gzstream to zstr for the implementation of zlib-compressed C++
output stream. The build system definition now uses MFEM_USE_ZLIB instead of
MFEM_USE_GZSTREAM, but the code interface (e.g. ofgzstream) remains the same.
- Various other simplifications, extensions, and bugfixes in the code.
API changes
-----------
- In the enum classes MemoryType and MemoryClass, "CUDA" was renamed to "DEVICE"
which now denotes either "CUDA" or "HIP" depending on the build configuration.
In the same enum classes, "CUDA_UVM" was renamed to "MANAGED".
Version 4.0, released on May 24, 2019
=====================================
Unlike previous MFEM releases, this version requires a C++11 compiler.
GPU support
-----------
- Added initial support for hardware devices, such as GPUs, and programming
models, such as CUDA, OCCA, RAJA and OpenMP.
- The GPU/device support is based on MFEM's new backends and kernels working
seamlessly with a new lightweight device/host memory manager. The kernels can
be implemented either in OCCA, or as a simple wrapper around for-loops, which
can then be dispatched to RAJA and native backends. See the files forall.hpp
and mem_manager.hpp in the general/ directory for more details.
- Several of the MFEM example codes (ex1, ex1p, ex6, and ex6p) can now take
advantage of GPU acceleration with the backend selectable at runtime. Many of
the linear algebra and finite element operations (e.g. partially assembled
bilinear forms) have been extended to take advantage of kernel acceleration by
simply replacing loops with the MFEM_FORALL() macro.
- In addition to native CUDA kernels, the library currently supports OCCA, RAJA
and OpenMP kernels, which could be mixed and matched in different parts of the
same application. We plan on adding support for more programming models and
devices in the future, without the need for significant modifications in user
code. The list of current backends is: "occa-cuda", "raja-cuda", "cuda",
"occa-omp", "raja-omp", "omp", "occa-cpu", "raja-cpu", and "cpu".
- GPU-related limitations:
* Hypre preconditioners are not yet available in GPU mode, and in particular
hypre must be built in CPU mode.
* Only constant coefficients are currently supported on GPUs.
* Optimized element assembly, and matrix-free bilinear forms are not
implemented yet. Element batching is currently ignored.
* In device mode, full assembly is performed on the host (but the matvec
action is performed on the device).
* Partial assembly kernels are not implemented yet for simplices.
Discretization improvements
---------------------------
- Partial assembled finite element operators are now available in the core
library, based on the new classes PABilinearFormExtension, ElementRestriction,
DofToQuad and GeometricFactors (associated with the classes BilinearForm,
FiniteElementSpace, FiniteElement and Mesh, respectively). The kernels for
partial assembled Setup/Assembly and Action/Mult are implemented in the
BilinearFormIntegrator methods AssemblePA and AddMultPA.
- Added support for a general "low-order refined"-to-"high-order" transfer of
GridFunction data from a "low-order refined" (LOR) space defined on a refined
mesh to a "high-order" (HO) finite element space defined on a coarse mesh. See
the new classes InterpolationGridTransfer and L2ProjectionGridTransfer and the
new LOR Transfer miniapp: miniapps/tools/lor-transfer.cpp.
- Added element flux, and flux energy computation in class ElasticityIntegrator,
allowing for the use of Zienkiewicz-Zhu type error estimators with the
integrator. For an illustration of this addition, see the new Example 21.
- Added support for derefinement of vector (RT + ND) spaces.
- Added a variety of coefficients which are sums or products of existing
coefficients as well as grid function coefficients which return the
divergence, gradient, or curl of their GridFunctions.
Support for wedge elements and meshes with mixed element types
--------------------------------------------------------------
- Added support for wedge-shaped mesh elements of arbitrary order (with Geometry
type PRISM) which have two triangular faces and three quadrilateral faces.
Several examples of such meshes can be found in the data/ directory.
- Added support for mixed meshes containing triangles and quadrilaterals in 2D
or tetrahedra, wedges, and hexahedra in 3D. This includes support for uniform
refinement of such meshes. Several examples of such meshes can be found in the
data/ directory.
- Added H1 and L2 finite elements of arbitrary order for Wedge elements.
- Added support for reading and writing linear and quadratic meshes containing
wedge elements in VTK mesh format. Several examples of such meshes can be
found in the data/ directory.
Other meshing improvements
--------------------------
- Improved the uniform refinement of tetrahedral meshes (also part of the
uniform refinement of mixed 3D meshes). The previous refinement algorithm is
still available as an option in Mesh::UniformRefinement. Both can be used in
the updated Mesh Explorer miniapp.
- The local tetrahedral mesh refinement algorithm in serial and in parallel now
follows precisely the paper:
D. Arnold, A. Mukherjee, and L. Pouly, "Locally Adapted Tetrahedral Meshes
Using Bisection", SIAM J. Sci. Comput. 22 (2000), 431-448.
This guarantees that the shape regularity of the elements will be preserved
under refinement.
- The TMOP mesh optimization algorithms were extended to support user-defined
space-dependent limiting terms. Improved the TMOP objective functions by more
accurate normalization of the different terms.
- Added support for parallel communication groups on non-conforming meshes.
- Improved parallel partitioning of non-conforming meshes. If the coarse mesh
elements are ordered as a sequence of face-neighbors, the parallel partitions
are now guaranteed to be continuous. To that end, inline quadrilateral and
hexahedral meshes are now by default ordered along a space-filling curve.
- A boundary in a NURBS mesh can now be connected with another boundary. Such a
periodic NURBS mesh is a simple way to impose periodic boundary conditions.
- Added support for reading linear and quadratic 2D quadrilateral and triangular
Cubit meshes.
New and updated examples and miniapps
-------------------------------------
- Added a new meshing miniapp, Toroid, which can produce a variety of torus
shaped meshes by twisting a stack of wedges or hexahedra.
- Added a new meshing miniapp, Extruder, that demonstrates the capability to
produce 3D meshes by extruding 2D meshes.
- Added a simple miniapp, LOR Transfer, for visualizing the actions of the
transfer operators between a high-order and a low-order refined spaces.
- Added a new example, Example 20/20p, that solves a system of 1D ODEs derived
from a Hamiltonian. The example demonstrates the use of the variable order,
symplectic integration algorithm implemented in class SIAVSolver.
- Added a new example, Example 21/21p, that illustrates the use of AMR to solve
a linear elasticity problem. This is an extension of Example 2/2p.
New and improved solvers and preconditioners
--------------------------------------------
- Added support for parallel ILU preconditioning via hypre's Euclid solver.
- Added support for STRUMPACK v3 with a small API change in the class
STRUMPACKSolver, see "API changes" below.
Miscellaneous
-------------
- Added unit tests based on the Catch++ library in the test/ directory.
- Renamed the option MFEM_USE_OPENMP to MFEM_USE_LEGACY_OPENMP. This legacy
option is deprecated and planned for removal in a future release. The original
option name, MFEM_USE_OPENMP, is now used to enable the new OpenMP backends in
the new kernels.
- In SparseMatrix added the option to perform MultTranspose() by matvec with
computed and stored transpose matrix. This is required for deterministic
results when using devices such as CUDA and OpenMP.
- Altered the way FGMRES counts its iterations so that it matches GMRES.
- Various other simplifications, extensions, and bugfixes in the code.
- Construct abstract parallel rectangular truedof-to-truedof operators via
Operator::FormDiscreteOperator().
API changes
-----------
- In multiple places, use Geometry::Type instead of int, where appropriate.
- In multiple places, use Element::Type instead of int, where appropriate.
- The Mesh methods GetElementBaseGeometry and GetBdrElementBaseGeometry no
longer have a default value for their parameter, they only work with an
explicitly given index.
- In class Mesh, added methods useful for queries regarding the types of
elements present in the mesh: HasGeometry, GetNumGeometries, GetGeometries,
and class Mesh::GeometryList.
- The struct CoarseFineTransformations (returned by the Mesh method
GetRefinementTransforms) now stores the embedding matrices separately for each
Geometry::Type.
- In class ParMesh, replaced the method GroupNFaces with two new methods:
GroupNTriangles and GroupNQuadrilaterals. Also, replaced GroupFace with two
methods: GroupTriangle and GroupQuadrilateral.
- In class ParMesh, made the two RefineGroups methods protected.
- Removed the virtual method Element::GetRefinementFlag, it is only used by the
derived class Tetrahedron.
- Added new methods: Array::CopyTo, Tetrahedron::Init.
- In class STRUMPACKSolver, the method SetMC64Job() was replaced by the new
methods: DisableMatching(), EnableMatching(), and EnableParallelMatching().
Version 3.4, released on May 29, 2018
=====================================
More general and efficient mesh adaptivity
------------------------------------------
- Added support for PUMI, the Parallel Unstructured Mesh Infrastructure from
https://scorec.rpi.edu/pumi. PUMI is an unstructured, distributed mesh data
management system that is capable of handling general non-manifold models and
effectively supports automated adaptive analysis. PUMI enables for the first
time support for parallel unstructured modifications of MFEM meshes.
- Significantly reduced MPI communication in the construction of the parallel
prolongation matrix in ParFiniteElementSpace, for much improved parallel
scaling of non-conforming AMR on hundreds of thousands of MPI tasks. The
memory footprint of the ParNCMesh class has also been reduced.
- In FiniteElementSpace, the fully assembled refinement matrix is now replaced
by default by a specialized refinement operator. The operator option is both
faster and more memory efficient than using the fully assembled matrix. The
old approach is still available and can be enabled, if needed, using the new
method FiniteElementSpace::SetUpdateOperatorType().
Discretization improvements
---------------------------
- Added support for a general "high-order"-to-"low-order refined" transfer of
GridFunction and true-dof data from a "high-order" finite element space
defined on a coarse mesh, to a "low-order refined" space defined on a refined
mesh. The new methods, GetTransferOperator and GetTrueTransferOperator in the
FiniteElementSpace classes, work in both serial and parallel and support
matrix-based as well as matrix-free transfer operator representations. They
use a new method, GetTransferMatrix, in the FiniteElement class similar to
GetLocalInterpolation, that allows the coarse FiniteElement to be different
from the fine FiniteElement.
- Added class ComplexOperator, that implements the action of a complex operator
through the equivalent 2x2 real formulation. Both symmetric and antisymmetric
block structures are supported.
- Added classes for general block nonlinear finite element operators (deriving
from BlockNonlinearForm and ParBlockNonlinearForm) enabling solution of
nonlinear systems with multiple unknowns in different function spaces. Such
operators have assemble-based action and also support assembly of the gradient
operator to enable inversion with Newton iteration.
- Added variable order NURBS: for each space each knot vector in the mesh can
have a different order. The order information is now part of the finite
element space header in the NURBS mesh output, so NURBS meshes in the old
format need to be updated.
- In the classes NonlinearForm and ParNonlinearForm, added support for
non-conforming AMR meshes; see also the "API changes" section.
- New specialized time integrators: symplectic integrators of orders 1-4 for
systems of first order ODEs derived from a Hamiltonian and generalized-alpha
ODE solver for the filtered Navier-Stokes equations with stabilization. See
classes SIASolver and GeneralizedAlphaSolver in linalg/ode.hpp.
- Inherit finite element classes from the new base class TensorBasisElement,
whenever the basis can be represented by a tensor product of 1D bases.
- Added support for elimination of boundary conditions in block matrices.
New and updated examples and miniapps
-------------------------------------
- Added a new serial and parallel example (ex19) that solves the quasi-static
incompressible hyperelastic equations. The example demonstrates the use of
block nonlinear forms as well as custom block preconditioners.
- Added a new serial example (ex23) to demonstrate the use of second order
time integration to solve the wave equation.
- Added a new electromagnetics miniapp, Maxwell, for simulating time-domain
electromagnetics phenomena as a coupled first order system of equations.
- A simple local refinement option has been added to the mesh-explorer miniapp
(menu option 'r', sub-option 'l') that selects elements for refinement based
on their spatial location - see the function 'region()' in the source file.
- Added a set of miniapps specifically focused on Isogeometric Analysis (IGA) on
NURBS meshes in the miniapps/nurbs directory. Currently the directory contains
variable order NURBS versions of examples 1, 1p and 11p.
- Added PUMI versions of examples ex1, ex1p, ex2 and ex6p in a new examples/pumi
directory. The new examples demonstrate the PUMI APIs for parallel and serial
mesh loading (ex1 and ex1p), applying BCs using classification (ex2), and
performing parallel mesh adaptation (ex6p).
- Added two new miniapps related to DataCollection I/O in miniapps/tools:
load-dc.cpp can be used to visualize fields saved via DataCollection classes;
convert-dc.cpp demonstrates how to convert between MFEM's different concrete
DataCollection options.
- Example 10p with its SUNDIALS and PETSc versions have been updated to reflect
the change in the behavior of the method ParNonlinearForm::GetLocalGradient()
(see the "API changes" section) and now works correctly on non-conforming AMR