Hey! From experience of having a massive automatic cactus farm the cactus will grow whether your close or not, as in the timer will continue to tick down.
But I'm not sure whether the blocks will actively update and grow if your far away, I know spawners and the like don't.
It is probably worth clearing out the chests, going to the opposite end of the island until te cactus have grown then somehow checking.
Sorry I can't be of more help!