On R 3.4.0: > phyper(10, 14, 0, 11, log=FALSE) [1] 0 > phyper(11, 15, 0, 12, log=FALSE) [1] 0 > phyper(10, 14, 0, 11, log=TRUE) [1] -Inf > phyper(11, 15, 0, 12, log=TRUE) [1] NaN Warning message: In phyper(11, 15, 0, 12, log = TRUE) : NaNs produced The last command should also return -Inf.

My guess is that in `phyper()` the value of pd = pdhyper(...) is some small negative number, which is why log() returns NaN.

My previous comment was incorrect. Actually, in pdhyper(10, 14, 0, 11): sum = -1.0 and for phyper(11, 15, 0, 12): sum = -1.1, which makes log1p() generate NaN.

The trivial fix would be to check d = dhyper(). If it's -Inf (for lop_p==TRUE case), then the phyper() result is -Inf and no pd = pdhyper() calculation is necessary. Same for d == 0 && lop_p == FALSE. But it's not clear to me whether pdhyper() can give incorrect results for the other input parameters combinations.