what's your upload bandwidth -
www.speedtest.net
Perform a traceroute to the server when it happens and post the results:
start > run
cmd {hit enter}
tracert serverhostname(found in server setup)
Try powering off the modem-router-pc (in that order) for at least 60 seconds each (this will clear all cached data)
Try re-installing or updating the router firmware. Check the manufacturers website for your make/model
just to test, try connecting directly to the router using ethernet cables.