Verification & Validation
Every verified filter is proven correct against an independent oracle — not against the output of an older version.
Why this exists
DREAM.3D 6.5.171 has been in the field for a number of years, but it was never subjected to formal Verification & Validation (V&V). That means “DREAM3D-NX matches DREAM.3D 6.5.171” is not a correctness check. It is a difference explanation check.
So we invert the usual order. Each filter must first be shown correct independently of the legacy application, against an oracle chosen before any legacy comparison is run. Only then do we compare against DREAM.3D 6.5.171, and that comparison becomes the migration guidance rather than a verdict.
Where the two disagree, the oracle decides which one is wrong. Often it is DREAM.3D 6.5.171. Sometimes that is DREAM3D-NX.
How a filter is verified
Every verified filter is checked against at least one of five oracle classes. The first four are preferred; the fifth requires documented justification for why nothing better was feasible.
| Class | What it is |
|---|---|
| Analytical | Closed-form expected output on hand-built input — thresholds, crops, rotations, color conversions, array arithmetic. |
| Reference implementation | A trusted external library produces the expected output — NumPy, SciPy, MTEX, EbsdLib, Eigen. |
| Paper-based | The filter reproduces a published figure, table, or equation from a named reference. |
| Invariant | Properties the output must satisfy by derivation — FeatureIds contiguous from 1, phase fractions summing to 1, mass conservation. |
| Expert-visual | A domain expert signs off on canonical cases. Last resort. |
One rule governs the order: pick the oracle before running any DREAM.3D 6.5.171 comparison. A legacy-derived expectation cannot verify the code it was derived from.
Each verified filter carries three artifacts in the source tree, pinned to the commit and exemplar-archive checksum they were verified against: a one-page V&V report, a deviations document, and exemplar provenance.
What the campaign found
As of DREAM3D-NX 7.4.2, 41 filters carry V&V reports and 115 behavioral deviations from DREAM.3D 6.5.171 are documented. Thirty-four of those 41 filters turned up at least one deviation.
| Category | Count |
|---|---|
| Deliberate algorithmic, design, or UX choices | 42 |
| Defects in DREAM.3D 6.5.171 | 17 |
| Defects in DREAM3D-NX | 8 |
| Defects present in both | 7 |
| Defects at the library or unattributed boundary | 14 |
| Precision differences | 13 |
| Library or API differences | 11 |
| Other | 3 |
| Total documented deviations | 115 |
Roughly 40 percent are outright defects. The remainder are differences we chose deliberately, or consequences of newer libraries and higher-precision math — all documented rather than left for a user to discover.
Fixed in DREAM3D-NX 7.4.2
Twenty-five deviations across nine filters are resolved in this release. If you used these filters in DREAM3D-NX 7.4.1 or earlier, results should be regenerated.
| Filter | Fixed |
|---|---|
| Read H5OINA Data | Thirteen defects, including a hexagonal phi2 alignment applying 30 degrees where radians were required, Euler values written to the wrong tuple slab, lattice angles in the wrong unit, and ten reader error paths that never reached the user. |
| Segment Features (C-Axis Misalignment) | Segmentation could begin from an unvalidated seed, adding an empty feature, shifting every FeatureId, or growing a feature from a masked-out voxel. Also affected the EBSD and Scalar segmentation filters, which share the driver. Datasets with unindexed cells are now processed rather than rejected, and RectGrid input no longer crashes. |
| Multi-Threshold Objects | Nested comparison sets could produce an all-false or incorrect mask. Legacy pipeline conversion discarded the nesting and reset unions to AND, so A AND (B OR C) was evaluated as A AND B AND C. |
| Erode/Dilate Bad Data | The X, Y, and Z Direction parameters had no effect on the output. |
| Compute Schmids | An uninitialized orientation slot, corrected in EbsdLib 3.1.1. |
| Read Oxford Instr. EBSD Data (.ctf) | Preflight hard-coded a z-extent, mis-sizing single-slice imports. |
| Compute Feature Neighbor C-Axis Misalignments | Orientation-math defect shared with the legacy implementation, corrected on the DREAM3D-NX side. |
| Compute Feature Neighbor Misorientations | As above. |
| Rotate Euler Reference Frame | A zero-length rotation axis is now rejected in preflight instead of silently producing NaN. |
Defects found in DREAM.3D 6.5.171
Verifying against independent oracles also surfaced seventeen defects across eleven filters in the legacy application that DREAM3D-NX does not share. Several have been present since 2013. DREAM.3D 6.5.171 is no longer maintained, so these are documented as migration guidance: if you rely on results it produced, these are worth reviewing.
| Filter | Defect in DREAM.3D 6.5.171 |
|---|---|
| Neighbor Orientation Correlation | Runs only half the intended cleanup passes, so multi-pass fills stop early. Copies from the last scanned neighbor rather than the most similar one. Can overwrite a cell with data from a different phase. |
| Require Minimum Number of Neighbors | Three defects in the merge and marking passes, including rejected features that are not revisited. |
| Compute Feature Neighbors | Shared surface areas are wrong for any dataset with anisotropic spacing. Every feature is marked a surface feature when the X or Y dimension is 1. |
| Neighbor Orientation Comparison (Bad Data) | Fails to flip a bad voxel whose good-neighbor count is exactly equal to the specified threshold. A different-phase neighbor’s misorientation can be counted as in-tolerance. |
| Read EDAX EBSD Data (.ang) | A file whose phase sections do not start at index 1 causes a segmentation fault. |
| Compute Feature Average Orientations | Writes garbage for a feature with no contributing voxels, where DREAM3D-NX writes a clean identity quaternion. |
| Compute Schmids | Returns sentinel values for skipped features rather than flagging them. |
| Compute IPF Colors (Face) | Incorrect result in the two-phase branch. |
| Compute Kernel Average Misorientations | Legacy-only defect; DREAM3D-NX has been correct since the port. |
| Compute Feature Reference C-Axis Misorientations | Legacy-only defect. |
| Create Element Array from Feature Array | Negative Feature Ids complete without error and write undefined values. |
Reading a deviation report
Deviations are addressed by a stable ID — FilterName-D3 — that survives
renames, with the filter UUID as the permanent anchor. Each entry records the
symptom, the root cause and which side it belongs to, who is affected, and a
recommendation on which implementation to trust.
The reports live alongside the source they describe:
src/Plugins/<Plugin>/vv/<FilterName>.md # the V&V report
src/Plugins/<Plugin>/vv/deviations/<FilterName>.md # deviations from 6.5.171
src/Plugins/<Plugin>/vv/provenance/<archive>.md # exemplar provenance
This work is ongoing. Filters are verified in priority order, and each release adds to the set.
Source code and full V&V reports can be found at https://github.com/BlueQuartzSoftware/simplnx/tree/develop/src