Huge numeric discrepancies on mac_arm and other architectures
In some persistence tests, we had to relax numeric limits to prevent failure under certain architectures.
There are two possible causes:
- Either our current measure of discrepancy is unreasonably blowing up irrelevant differences in channels with very low intensity (see below, ex #953 (closed)).
- Or some of our computations are numerically unstable, and ought to be improved.
The problem appeared on Windows after an OS upgrade (#129 (closed)), then on Mac_arm after a security update (#597 (closed)). For the concerned tests (#597 (closed), #817 (closed)), limits were relaxed globally.
Then many more discrepancies were found on other architectures supported by the Debian science project (#952 (closed)). For these, an extra CMake option BA_3ARCH
was introduced.
Hint:
- Persistence tests are controlled by Tests/Functional/Persist/CMakeLists.txt
- Individual test cases are implemented in Tests/Suite/Common/TestSuite.h
Discrepancies are mean relative differences as computed by DiffUtil::checkRelativeDifference
Instead, we should look at the worst data bin. For two reasons:
- Fundamentally, we want each single computation to be correct.
- We then could make error reports more specific, which in turn might elp in debugging.
But there is also a good reason for the current solution, namely that we should not be overly concerned about relative differences in bins that have very low absolute intensity (relative to high-intensity bins of course). Therefore we first need to invent a reasonable measure for worst-case deviations.