Oh YuCK A-jock!
I am a swimmer and the water environment does loosen everything up so you are not totally stuffed. You can do more kick sets if you are so clogged that you feel it is getting in the water too much. Having something to wipe your nose at the end is good also.
I have not had a cold for about 2 years(knock on wood), but I know I swam through it. As long as I was not coughing a lot, which interferes a lot, then I did not worry about the stuffiness.
I have a friend who was fighting a sinus infection last week. She joked it made her butterfly better as her tendency is to slam her head too much when entering the water, which is bad form. With the sinus infection, the impact would cause great pain, so it reminded her that her form was wrong!
For me, I usually go by the rule, above the neck, OK to exercise, below, not OK, and I also check my fatigue level. If feeling very fatigued, and you try to push to hard, you may set yourself up for complications such as sinus infections, bronchitis, and more colds. So listen to your body, and get back in gradually.