Agreed. First of all that would make running backups more complex, and would require either manual interaction, or very careful automation of some kind.
And any public facing service (like blog and some stuff) would still need to be accessible somehow, so…
Connection closed means somebody is listening to the port and failing/not willing to reply.
Unless some network middlemen is closing your connection (ssh should be on port > 1024 to be safe from ISP throttling), your ssh server is severely strained (oom, disk full…) or your F2B is kicking in.