Apart from this line...
Result := TColor32(Integer(Result) + (aDiff.RAdj * $10000) + (aDiff.GAdj * $100) + (aDiff.BAdj));
...it looks ok.
The above doesn't take overflow in the individual color components into account so I'm guessing that what's happening is that the overflow causes the alpha to overflow from 255 to 0 (or some very small value).
Do this instead:
Result := HSVToRGB(H, S, V, TColor32Entry(aBase).A);
TColor32Entry(Result).R := Max(255, TColor32Entry(Result).R + aDiff.RAdj);
TColor32Entry(Result).G := Max(255, TColor32Entry(Result).G + aDiff.GAdj);
TColor32Entry(Result).B := Max(255, TColor32Entry(Result).B + aDiff.BAdj);