Here we also have 2 cats and have now given birth to a healthy daughter.
Mine aren't allowed further than the backyard (is closed off) due to a purebred Maine Coon and a dog that had an encounter with an animal control officer, luckily now completely healthy again. I just had my husband do the litter box out of an abundance of caution. Otherwise you can just cuddle with your cats. The biggest risk you run is with young cats that carry toxoplasma more often than older cats. Wear gloves in the garden when you're digging in the soil because it can be in the soil and wash raw vegetables well because it can also be on them. It's spread by mice and they run over and around the vegetables and if the cat catches one it gets it that way. Once you've caught it once you're immune to it and the risks are minimal. Only if you haven't had it yet can it be very harmful to your unborn child because it can pass through the placenta. So just let someone else do the litter box, it's not a fun chore anyway so it's not that bad