My guess is that it's a router problem since you have the same effect with both computers. Is this a modem you own or one that you rent from your service provider? Mine is from the service provider so I can replace it as often as I like!

If you own it then I don't suggest you be so cavalier in trying to replace it. The only thing I can suggest is to try the hard reset button or contact Linksys customer service to see what they suggest. Linksys is usually a pretty good name so there shouldn't be too many problems.
You shouldn't have to worry about going through the setup process again. The 26 digit number (mine is 26, yours may be different) that you have to enter to access the 'net through the modem shouldn't change if you reset it, so I'd say it's worth a try.
But then I have two out of three computers in pieces at the moment, so maybe I'm not the best guy to ask!
Def