#1461 GFX3D Y-Sorting assert() bug

[rogerman]

This bug is in reference to [r5089] and [r5130]. Prior to r5089, Y-sort could crash on certain combinations of compilers/optimizations and operating system. But discussion with zeromus suggests I only covered up the bug in r5089, but not actually fixed it. r5130 was introduced to check the sorting predicate function, which is the piece of code currently in question.

To trigger the assert, first you need to revert std::stable_sort to std::sort in gfx3d_doFlush(), as this assert will not get triggered with std::stable_sort. Next, I've provide two test cases to trigger the assert:

• Super Mario 64 DS: Start in Adventure Mode and choose File A -- assert the moment the world starts.
• Nanostray 2: Start in Adventure Mode and Continue, then on the planets screen, choose Naizoh Habitat. Activate the green arrow to start -- assert the moment the level starts.

Note that these are the exact same moments that would cause a crash prior to r5089, but now cause an assert in r5130.

[rogerman]

At the time of assert, there are comparisons being made between NaNs and real numbers. This would explain a lot of what's happening!

The predicate functions do one comparison each on the POLY.miny and POLY.maxy values. Both of these values are set in gfx3d_doFlush() at gfx3d.cpp:2192.

• POLY.miny and POLY.maxy are both set to 'verty'.
• 'verty' is dependent on 'vertw' by division.
• If 'vertw' == 0.0f, then 'verty' becomes a NaN via divide-by-zero.

So if we use this hack...:
if (vertw == 0.0f) vertw = 0.000001f;

Now the assert is satisfied and everything works. When reverting back to using std::sort with gfx3d_ysort_compare_kalven() as the predicate, no crashes occur as long as the NaN is avoided.

So two new questions arise from this:

1. Are we reading the vertex coordinates correctly?
2. Is 0 a valid value for w?

[rogerman]

Actually implement the hack workaround in [r5132].
I'll keep this bug open until we can answer both the questions I posted above with some level of certainty, or if there is a test case that actually reveals something.

[rogerman]

After years of testing without any reported issues, I think it should be safe to close out this one. See commit git#35e834f for more details.