ENH: Create DataModifiedAction that marks DataObjects as being modifi…

…ed by a filter (BlueQuartzSoftware#735)

* Update filters with information about which DataArrays are being modified/deleted
* Fix broken pipelines
* Clean up more filter arguments and pipeline files
* Store DataObject::Type in the DataObjectModification struct.

---------

Signed-off-by: Michael Jackson <[email protected]>

pipelines/Combo-EBSD-osc_r0c0.d3dpipeline

@@ -66,7 +66,7 @@
     },
     {
       "args": {
-        "cell_euler_angles_array_path": "DataContainer/CellData/EulerAngles",
+        "euler_angles_array_path": "DataContainer/CellData/EulerAngles",
         "rotation_axis": [
           0.0,
           0.0,

src/Plugins/ComplexCore/docs/RemoveFlaggedFeaturesFilter.md

@@ -1,12 +1,16 @@
-# Remove Flagged Features
+# Remove/Extract Flagged Features
 
 ## Group (Subgroup)
 
 Processing (Cleanup)
 
 ## Description
 
-This **Filter** will remove **Features** that have been flagged by another **Filter** from the structure.  The **Filter** requires that the user point to a boolean array at the **Feature** level that tells the **Filter** whether the **Feature** should remain in the structure.  If the boolean array is *false* for a **Feature**, then all **Cells** that belong to that **Feature** are temporarily *unassigned* and after all *undesired* **Features** are removed, the remaining **Features** are isotropically coarsened to fill in the gaps left by the removed **Features**.
+This **Filter** will remove **Features** that have been flagged by another **Filter** from the structure.  The **Filter**
+requires that the user point to a boolean array at the **Feature** level that tells the **Filter** whether the **Feature**
+should remain in the structure.  If the boolean array is *false* for a **Feature**, then all **Cells** that belong to that
+**Feature** are temporarily *unassigned*. Optionally, after all *undesired* **Features** are removed, the remaining **Features** are
+isotropically coarsened to fill in the gaps left by the removed **Features**.
 
 % Auto generated parameter table will be inserted here
 

src/Plugins/ComplexCore/docs/RemoveFlaggedVertices.md

@@ -6,14 +6,14 @@ DREAM3D Review (Geometry)
 
 ## Description
 
-This **Filter** removes **Vertices** from the supplied **Vertex Geometry** that are flagged by a boolean mask array.
-Specifically, **Vertices** flagged as *true* are removed from the **Geometry**. A new reduced **Vertex Geometry** is
+This **Filter** removes **Vertices** from the supplied **Vertex Geometry** that are flagged as **TRUE** by a boolean mask array.
+A new reduced **Vertex Geometry** is
 created that contains all the remaining **Vertices**. It is unknown until run time how many **Vertices** will be removed
 from the **Geometry**. Therefore, this **Filter** requires that a new **Data Container** be created to contain the
-reduced **Vertex Geometry**. This new **Data Container** will contain copies of any **Feature** or **Ensemble** *
-*Attribute Matrices** from the original **Data Container**. Additionally, all **Vertex** data will be copied, with
-tuples *removed* for any **Vertices** removed by the **Filter**. The user must supply a name for the reduced **Data
-Container**, but all other copied objects (**Attribute Matrices** and **Attribute Arrays**) will retain the same names
+reduced **Vertex Geometry**. This new **Vertex Geometry** will contain copies of any **Feature** or **Ensemble**
+**Attribute Matrices** from the original **Data Container**. Additionally, all **Vertex** data will be copied, with
+tuples *removed* for any **Vertices** removed by the **Filter**. The user must supply a name for the reduced **Vertex Geometry**,
+but all other copied objects (**Attribute Matrices** and **Attribute Arrays**) will retain the same names
 as the original source.
 
 *Note:* Since it cannot be known before run time how many **Vertices** will be removed, the new **Vertex Geometry** and

src/Plugins/ComplexCore/docs/ReplaceElementAttributesWithNeighborValuesFilter.md

@@ -12,8 +12,8 @@ maximum or minimum value. The attributes of the neighbor with the maximum/minimu
 reference **Cell**.
 
 *Note:* By default, the **Filter** will run only one iteration of the cleanup. If the user selects the *Loop Until Gone*
-option, then the **Filter** will run iteratively until no **Cells** exist that meet the users criteria. So, if a **Cell
-** meets the threshold and so are all of its neighbors, then that **Cell** will not be changed during that iteration and
+option, then the **Filter** will run iteratively until no **Cells** exist that meet the users criteria. So, if a **Cell**
+meets the threshold and so are all of its neighbors, then that **Cell** will not be changed during that iteration and
 will remain unchanged until one of its neighbors gets changed by a **Cell** further away.
 
 ## Examples

src/Plugins/ComplexCore/pipelines/ReplaceElementAttributesWithNeighbor.d3dpipeline

@@ -104,7 +104,7 @@
     },
     {
       "args": {
-        "cell_euler_angles_array_path": "DataContainer/Cell Data/EulerAngles",
+        "euler_angles_array_path": "DataContainer/Cell Data/EulerAngles",
         "rotation_axis": [
           0.0,
           0.0,
@@ -192,7 +192,7 @@
     },
     {
       "args": {
-        "confidence_index_array_path": "DataContainer/Cell Data/Confidence Index",
+        "comparison_data_path": "DataContainer/Cell Data/Confidence Index",
         "loop": true,
         "min_confidence": 0.10000000149011612,
         "selected_comparison": 0,

src/Plugins/ComplexCore/pipelines/VtkRectilinearGridWriter.d3dpipeline

@@ -192,7 +192,7 @@
             {
               "args": {
                 "cell_phases_array_path": "DataContainer/CellData/Phases",
-                "confidence_index_array_path": "DataContainer/CellData/Confidence Index",
+                "correlation_array_path": "DataContainer/CellData/Confidence Index",
                 "crystal_structures_array_path": "DataContainer/CellEnsembleData/CrystalStructures",
                 "ignored_data_array_paths": [],
                 "image_geometry_path": "DataContainer",

src/Plugins/ComplexCore/src/ComplexCore/Filters/AlignSectionsFeatureCentroidFilter.cpp

@@ -9,6 +9,7 @@
 #include "complex/Parameters/FileSystemPathParameter.hpp"
 #include "complex/Parameters/GeometrySelectionParameter.hpp"
 #include "complex/Parameters/NumberParameter.hpp"
+#include "complex/Utilities/FilterUtilities.hpp"
 
 #include <filesystem>
 
@@ -132,6 +133,10 @@ IFilter::PreflightResult AlignSectionsFeatureCentroidFilter::preflightImpl(const
     return {MakeErrorResult<OutputActions>(-3425, "Write Alignment Shifts is TRUE but the output file path is empty. Please ensure the file path is set for the alignment file.")};
   }
 
+  // Inform users that the following arrays are going to be modified in place
+  // Cell Data is going to be modified
+  complex::AppendDataObjectModifications(dataStructure, resultOutputActions.value().modifiedActions, cellDataGroupPath, {});
+
   // Return both the resultOutputActions and the preflightUpdatedValues via std::move()
   return {std::move(resultOutputActions), std::move(preflightUpdatedValues)};
 }

src/Plugins/ComplexCore/src/ComplexCore/Filters/AlignSectionsListFilter.cpp

@@ -8,6 +8,7 @@
 #include "complex/Parameters/BoolParameter.hpp"
 #include "complex/Parameters/FileSystemPathParameter.hpp"
 #include "complex/Parameters/GeometrySelectionParameter.hpp"
+#include "complex/Utilities/FilterUtilities.hpp"
 
 #include <filesystem>
 
@@ -92,6 +93,10 @@ IFilter::PreflightResult AlignSectionsListFilter::preflightImpl(const DataStruct
                                                               imageGeom.getNumYCells(), imageGeom.getNumZCells()))};
   }
 
+  // Inform users that the following arrays are going to be modified in place
+  // Cell Data is going to be modified
+  complex::AppendDataObjectModifications(dataStructure, resultOutputActions.value().modifiedActions, imageGeom.getCellDataPath(), {});
+
   return {std::move(resultOutputActions), std::move(preflightUpdatedValues)};
 }
 

src/Plugins/ComplexCore/src/ComplexCore/Filters/ErodeDilateBadDataFilter.cpp

@@ -10,6 +10,8 @@
 #include "complex/Parameters/GeometrySelectionParameter.hpp"
 #include "complex/Parameters/MultiArraySelectionParameter.hpp"
 #include "complex/Parameters/NumberParameter.hpp"
+#include "complex/Utilities/DataGroupUtilities.hpp"
+#include "complex/Utilities/FilterUtilities.hpp"
 
 using namespace complex;
 
@@ -102,6 +104,10 @@ IFilter::PreflightResult ErodeDilateBadDataFilter::preflightImpl(const DataStruc
     MakeErrorResult(-16700, fmt::format("Operation Selection must be 0 (Dilate) or 1 (Erode). {} was passed into the filter. ", pOperationValue));
   }
 
+  // Inform users that the following arrays are going to be modified in place
+  // Cell Data is going to be modified
+  complex::AppendDataObjectModifications(dataStructure, resultOutputActions.value().modifiedActions, pFeatureIdsArrayPathValue.getParent(), pIgnoredDataArrayPathsValue);
+
   // Return both the resultOutputActions and the preflightUpdatedValues via std::move()
   return {std::move(resultOutputActions), std::move(preflightUpdatedValues)};
 }

src/Plugins/ComplexCore/src/ComplexCore/Filters/MinNeighbors.cpp

@@ -9,6 +9,7 @@
 #include "complex/Parameters/MultiArraySelectionParameter.hpp"
 #include "complex/Parameters/NumberParameter.hpp"
 #include "complex/Utilities/DataGroupUtilities.hpp"
+#include "complex/Utilities/FilterUtilities.hpp"
 
 namespace complex
 {
@@ -304,7 +305,7 @@ std::string MinNeighbors::humanName() const
 //------------------------------------------------------------------------------
 std::vector<std::string> MinNeighbors::defaultTags() const
 {
-  return {className(), "Minimum", "Neighbors", "Memory Management", "Cleanup"};
+  return {className(), "Minimum", "Neighbors", "Memory Management", "Cleanup", "Remove Features"};
 }
 
 //------------------------------------------------------------------------------
@@ -359,6 +360,10 @@ IFilter::PreflightResult MinNeighbors::preflightImpl(const DataStructure& dataSt
   auto numNeighborsPath = args.value<DataPath>(k_NumNeighbors_Key);
   auto minNumNeighbors = args.value<uint64>(k_MinNumNeighbors_Key);
 
+  complex::Result<OutputActions> resultOutputActions;
+
+  std::vector<PreflightValue> preflightUpdatedValues;
+
   std::vector<DataPath> dataArrayPaths;
 
   std::vector<usize> cDims = {1};
@@ -386,8 +391,14 @@ IFilter::PreflightResult MinNeighbors::preflightImpl(const DataStructure& dataSt
                                            fmt::format("The following DataArrays all must have equal number of tuples but this was not satisfied.\n{}", tupleValidityCheck.error()))};
   }
 
-  OutputActions actions;
-  return {std::move(actions)};
+  // Inform users that the following arrays are going to be modified in place
+  // Cell Data is going to be modified
+  complex::AppendDataObjectModifications(dataStructure, resultOutputActions.value().modifiedActions, featureIdsPath.getParent(), {});
+  // Feature Data is going to be modified
+  complex::AppendDataObjectModifications(dataStructure, resultOutputActions.value().modifiedActions, numNeighborsPath.getParent(), {});
+
+  // Return both the resultOutputActions and the preflightUpdatedValues via std::move()
+  return {std::move(resultOutputActions), std::move(preflightUpdatedValues)};
 }
 
 //------------------------------------------------------------------------------
@@ -400,7 +411,7 @@ Result<> MinNeighbors::executeImpl(DataStructure& data, const Arguments& args, c
   // If running on a single phase, validate that the user has not entered a phase number
   // that is not in the system ; the filter would not crash otherwise, but the user should
   // be notified of unanticipated behavior ; this cannot be done in the dataCheck since
-  // we don't have acces to the data yet
+  // we don't have access to the data yet
   if(applyToSinglePhase)
   {
     auto& featurePhasesArray = data.getDataRefAs<Int32Array>(featurePhasesPath);
@@ -453,7 +464,7 @@ Result<> MinNeighbors::executeImpl(DataStructure& data, const Arguments& args, c
   int32 count = 0;
   for(const auto& value : activeObjects)
   {
-    value == true ? count++ : count = count;
+    value ? count++ : count = count;
   }
   std::string message = fmt::format("Feature Count Changed: Previous: {} New: {}", currentFeatureCount, count);
   messageHandler(complex::IFilter::Message{complex::IFilter::Message::Type::Info, message});

src/Plugins/ComplexCore/src/ComplexCore/Filters/RemoveFlaggedFeaturesFilter.cpp

@@ -14,6 +14,7 @@
 #include "complex/Parameters/GeometrySelectionParameter.hpp"
 #include "complex/Parameters/MultiArraySelectionParameter.hpp"
 #include "complex/Parameters/StringParameter.hpp"
+#include "complex/Utilities/FilterUtilities.hpp"
 
 using namespace complex;
 
@@ -45,13 +46,13 @@ Uuid RemoveFlaggedFeaturesFilter::uuid() const
 //------------------------------------------------------------------------------
 std::string RemoveFlaggedFeaturesFilter::humanName() const
 {
-  return "Extract/Remove Flagged Features";
+  return "Remove/Extract Flagged Features";
 }
 
 //------------------------------------------------------------------------------
 std::vector<std::string> RemoveFlaggedFeaturesFilter::defaultTags() const
 {
-  return {className(), "Processing", "Cleanup"};
+  return {className(), "Processing", "Cleanup", "Remove Features"};
 }
 
 //------------------------------------------------------------------------------
@@ -60,9 +61,9 @@ Parameters RemoveFlaggedFeaturesFilter::parameters() const
   Parameters params;
 
   params.insertSeparator(Parameters::Separator{"Input Parameters"});
-  params.insertLinkableParameter(std::make_unique<ChoicesParameter>(k_Functionality_Key, "Selected Operation", "Whether to extract features into new geometry or remove or extract then remove",
-                                                                    to_underlying(Functionality::Remove),
-                                                                    ChoicesParameter::Choices{"Remove", "Extract", "Extract then Remove"})); // sequence dependent DO NOT REORDER
+  params.insertLinkableParameter(
+      std::make_unique<ChoicesParameter>(k_Functionality_Key, "Selected Operation", "Whether to [0] remove features, [1] extract features into new geometry or [2] extract and then remove",
+                                         to_underlying(Functionality::Remove), ChoicesParameter::Choices{"Remove", "Extract", "Extract then Remove"})); // sequence dependent DO NOT REORDER
   params.insert(std::make_unique<BoolParameter>(k_FillRemovedFeatures_Key, "Fill-in Removed Features", "Whether or not to fill in the gaps left by the removed Features", false));
 
   params.insertSeparator(Parameters::Separator{"Geometry"});
@@ -76,7 +77,7 @@ Parameters RemoveFlaggedFeaturesFilter::parameters() const
                                                   "The prefix name for each of new cropped (extracted) geometry \n\nNOTE: a '-' will automatically be added between the prefix and number",
                                                   "Extracted_Feature"));
 
-  params.insertSeparator(Parameters::Separator{"Required Input Cell Feature Data"});
+  params.insertSeparator(Parameters::Separator{"Required Input Feature Data"});
   params.insert(std::make_unique<ArraySelectionParameter>(k_FlaggedFeaturesArrayPath_Key, "Flagged Features", "Specifies whether the Feature will remain in the structure or not", DataPath{},
                                                           ArraySelectionParameter::AllowedTypes{DataType::boolean, DataType::uint8}, ArraySelectionParameter::AllowedComponentShapes{{1}}));
   params.insert(std::make_unique<MultiArraySelectionParameter>(k_IgnoredDataArrayPaths_Key, "Attribute Arrays to Ignore", "The list of arrays to ignore when removing flagged features",
@@ -108,13 +109,15 @@ IFilter::PreflightResult RemoveFlaggedFeaturesFilter::preflightImpl(const DataSt
 {
   auto pFeatureIdsArrayPathValue = filterArgs.value<DataPath>(k_CellFeatureIdsArrayPath_Key);
   auto pFlaggedFeaturesArrayPathValue = filterArgs.value<DataPath>(k_FlaggedFeaturesArrayPath_Key);
+  auto operationType = filterArgs.value<ChoicesParameter::ValueType>(k_Functionality_Key);
+  auto fillGaps = filterArgs.value<BoolParameter::ValueType>(k_FillRemovedFeatures_Key);
 
   PreflightResult preflightResult;
   complex::Result<OutputActions> resultOutputActions;
   std::vector<PreflightValue> preflightUpdatedValues;
 
-  auto* featureIds = dataStructure.getDataAs<Int32Array>(pFeatureIdsArrayPathValue);
-  if(featureIds == nullptr)
+  auto* featureIdsPtr = dataStructure.getDataAs<Int32Array>(pFeatureIdsArrayPathValue);
+  if(featureIdsPtr == nullptr)
   {
     return {nonstd::make_unexpected(std::vector<Error>{Error{-9890, fmt::format("Could not find selected Feature Ids Data Array at path '{}'", pFeatureIdsArrayPathValue.toString())}})};
   }
@@ -124,20 +127,20 @@ IFilter::PreflightResult RemoveFlaggedFeaturesFilter::preflightImpl(const DataSt
     return {nonstd::make_unexpected(std::vector<Error>{Error{-9891, fmt::format("Could not find selected Flagged Features Data Array at path '{}'", pFlaggedFeaturesArrayPathValue.toString())}})};
   }
 
-  DataPath cellFeatureAttributeMatrixPath = pFlaggedFeaturesArrayPathValue.getParent();
-  const auto* cellFeatureAM = dataStructure.getDataAs<AttributeMatrix>(cellFeatureAttributeMatrixPath);
-  if(cellFeatureAM == nullptr)
+  DataPath const cellFeatureAttributeMatrixPath = pFlaggedFeaturesArrayPathValue.getParent();
+  const auto* cellFeatureAmPtr = dataStructure.getDataAs<AttributeMatrix>(cellFeatureAttributeMatrixPath);
+  if(cellFeatureAmPtr == nullptr)
   {
     return {nonstd::make_unexpected(std::vector<Error>{
         Error{-9892, fmt::format("Could not find the parent Attribute Matrix for the selected Flagged Features Data Array at path '{}'", pFlaggedFeaturesArrayPathValue.toString())}})};
   }
 
-  std::string warningMsg = "";
-  for(const auto& [identifier, object] : *cellFeatureAM)
+  std::string warningMsg;
+  for(const auto& [identifier, object] : *cellFeatureAmPtr)
   {
-    if(const auto* srcNeighborListArray = dynamic_cast<const INeighborList*>(object.get()); srcNeighborListArray != nullptr)
+    if(const auto* srcNeighborListArrayPtr = dynamic_cast<const INeighborList*>(object.get()); srcNeighborListArrayPtr != nullptr)
     {
-      warningMsg += "\n" + cellFeatureAttributeMatrixPath.toString() + "/" + srcNeighborListArray->getName();
+      warningMsg += "\n" + cellFeatureAttributeMatrixPath.toString() + "/" + srcNeighborListArrayPtr->getName();
     }
   }
   if(!warningMsg.empty())
@@ -149,13 +152,24 @@ IFilter::PreflightResult RemoveFlaggedFeaturesFilter::preflightImpl(const DataSt
   auto pFunctionality = filterArgs.value<ChoicesParameter::ValueType>(k_Functionality_Key);
   if(pFunctionality != to_underlying(Functionality::Remove))
   {
-    DataPath tempPath = pFlaggedFeaturesArrayPathValue.getParent().createChildPath(k_boundsName);
-    auto action = std::make_unique<CreateArrayAction>(DataType::uint32, std::vector<usize>{featureIds->getNumberOfTuples()}, std::vector<usize>{featureIds->getNumberOfComponents() * 6}, tempPath);
+    DataPath const tempPath = pFlaggedFeaturesArrayPathValue.getParent().createChildPath(k_boundsName);
+    auto action =
+        std::make_unique<CreateArrayAction>(DataType::uint32, std::vector<usize>{featureIdsPtr->getNumberOfTuples()}, std::vector<usize>{featureIdsPtr->getNumberOfComponents() * 6}, tempPath);
 
     // After the execute function has been done, delete the temp array
     resultOutputActions.value().appendDeferredAction(std::make_unique<DeleteDataAction>(tempPath));
   }
 
+  // If we are in any way removing features, inform the user
+  if(operationType == 0 || operationType == 2)
+  {
+    // Inform users that the following arrays are going to be modified in place
+    // Cell Data is going to be modified
+    complex::AppendDataObjectModifications(dataStructure, resultOutputActions.value().modifiedActions, pFeatureIdsArrayPathValue.getParent(), {});
+    // Feature Data is going to be modified
+    complex::AppendDataObjectModifications(dataStructure, resultOutputActions.value().modifiedActions, pFlaggedFeaturesArrayPathValue.getParent(), {});
+  }
+
   return {std::move(resultOutputActions), std::move(preflightUpdatedValues)};
 }