I've never figured out what causes it, but I've noticed that going either to the skyblock hub or the main hub and then back to the island alleviates the lag.
Doesn't always work, but it usually goes away after a bit. The lag appears to be more server-side than client.