BPMemory: Handle fog configuration where both A and C are infinity/NaN

The console appears to behave against standard IEEE754 specification
here, in particular around how NaNs are handled. NaNs appear to have no
effect on the result, and are treated the same as positive or negative
infinity, based on the sign bit.

However, when the result would be NaN (inf - inf, or (-inf) - (-inf)),
this results in a completely fogged color, or unfogged color
respectively. We handle this by returning a constant zero for the A
varaible, and positive or negative infinity for C depending on the sign
bits of the A and C registers. This ensures that no NaN value is passed
to the GPU in the first place, and that the result of the fog
calculation cannot be NaN.

1 files changed, 3 insertions, 3 deletions

diff --git a/Source/Core/VideoBackends/Software/Tev.cpp b/Source/Core/VideoBackends/Software/Tev.cpp
index 836299e715..572da58f9c 100644
--- a/Source/Core/VideoBackends/Software/Tev.cpp
+++ b/Source/Core/VideoBackends/Software/Tev.cpp
@@ -751,14 +751,14 @@ void Tev::Draw()
       // ze = A/(B - (Zs >> B_SHF))
       const s32 denom = bpmem.fog.b_magnitude - (Position[2] >> bpmem.fog.b_shift);
       // in addition downscale magnitude and zs to 0.24 bits
-      ze = (bpmem.fog.a.GetA() * 16777215.0f) / (float)denom;
+      ze = (bpmem.fog.GetA() * 16777215.0f) / static_cast<float>(denom);
     }
     else
     {
       // orthographic
       // ze = a*Zs
       // in addition downscale zs to 0.24 bits
-      ze = bpmem.fog.a.GetA() * ((float)Position[2] / 16777215.0f);
+      ze = bpmem.fog.GetA() * (static_cast<float>(Position[2]) / 16777215.0f);
     }
 
     if (bpmem.fogRange.Base.Enabled)
@@ -796,7 +796,7 @@ void Tev::Draw()
                        // GXInitFogAdjTable): 1/cos = c/b = sqrt(a^2+b^2)/b
     }
 
-    ze -= bpmem.fog.c_proj_fsel.GetC();
+    ze -= bpmem.fog.GetC();
 
     // clamp 0 to 1
     float fog = (ze < 0.0f) ? 0.0f : ((ze > 1.0f) ? 1.0f : ze);