During the day it’s white, but it’s also overhead and blindingly bright so we don’t spend much time looking at it. As it gets closer to the horizon Rayleigh scattering begins filtering out the bluer light and the sun becomes yellow, then orange, then red. It also gets closer to our eyeline and becomes mildly safer to look at so we look at it a lot more. This in turn leads us to believe it’s always yellow