I don't think an anti-teaming plugin would ever work on CubeCraft. There would be way to many false kicks. You said if two players are within 5 blocks of each other for a decent amount of time and neither of them are taking damage from each other, it would kick them. There are many cases where players could be within 2 blocks of each other and not battling but also not teaming. Here is an example: You are out geared on your island, a player approaches, you dig a hole two blocks beneath the surface and bury yourself, shifting so you aren't detected. The player arrives on your island and stands right on the block above you, sorting his or her inventory but would kill you in a heart beat if they found you. With a anti-teaming plugin, both players would be kicked for teaming even though neither of them are teaming and would fight each other if the one hiding was detected.
So an anti-teaming plugin could probably never work (just going by your example). The best thing we can do to stop players from teaming is report those who are and get the word out that it's illegal to team in solo game modes. Maybe in the future there will be a more full proof way to prevent it but as of now, there isn't. Humans will always do whatever they can to increase their chances to win, teaming is just one way, although illegal, to improve their chances at winning.