In your opening post you said it was taking 2 - 3mins but after changing ports, that is down to one minute so perhaps it lies with the router.
Is the delay now on start up as well as after a restart ?
Switch off your PC and router and remove all power cables as well as the Ethernet cable.
Leave those off for about 30mins then switch just the router on.
When all of the lights are on, Ethernet connect and switch the PC on.
You may want to try a different router port.
When everything is back up and running, reboot to see if there is any improvement.
Those steps power cycle the router but leaving it off for 30mins should give you a new external IP address.
I can make a note of your current IP address and compare it when you post next after the recycle and reboot.