it does sound like an internet problem, mine can sometimes load a little slow and by the time I can move the match countdown is on 2 : P
But yeh I think it will have something to do with your download speed as everytime you enter the match your computer has to reload everything again. If your download speed is slow then the map loading will be slow, while your enemy's download speed has already loaded it all up. Seems unfair, but thats how the system works (or at least I think that's how).
I would suggest lowering some of the game graphics settings like render distance (you only need it short for PVP duels anyway). It could help cut down loading times. If this doesn't help, I'm not too sure what else to suggest apart from improving your internet connection and/or download speed. You'd have to Google that bit though, I'm not the best techy!
Hope I helped!