Hello CubeCrafters!
Welcome to the second edition of "Behind the Cube"! We hope these development blogs can give you guys some insight into what we are doing behind the scenes. In this edition, we will be discussing the development of Sentinel since we've started in May.
Why is cheating such a problem?
Minecraft allows for a lot of creativity and freedom within the game. In fact, this is the very reason we can create awesome experiences and work on the detection of cheats on the server side with great precision. Minecraft would not be the same without the modding community. However, the darker side of the modding community is when modifications are created to promote unfair advantages. This is where we come in.
The main thing that separates Minecraft from most other online games is that movement is client authoritative, so your game session calculates your motion from your inputs. When you move, your position is sent to the server for it to be updated. On the other hand, server authoritative movement means all motion is calculated on the server's side. When you press the W key to move forward, that action is sent to the server and the server will attempt to calculate where your next position will be. Then, your position sent back to you to tell you where you should be. It’s a bit more complicated than that but that is the general idea (Here’s an article about it if you’re interested in how other online games handle movement). The Minecraft protocol gives a lot of authority to the client that allows them to create destructive cheats. However, Minecraft is designed in a way where we can easily confirm almost everything sent from the client to check for cheats.
The people that are cheating on the network can be very smart. They can be quick to understand why they are being banned and adapt to avoid being banned. No longer are we fighting against a few people with a few clients, but a wide range of clients and large groups of people targeting the network being the first to try and find a bypass. The abundance of stolen/compromised accounts makes it easy for people to continue to come back after they’ve been banned. Once they’ve been banned, they can adjust their settings the next time they join in an attempt to bypass. This can be a difficult situation for us. On one hand, we want to remove the cheater as quickly as possible to minimize their impact on the game. On the other hand, quickly banning a player makes it easier for them to understand why they've been banned. We have to find a compromise between making sure the cheaters impact on the game is minimal, while also preventing them from learning why they've been banned so quickly. Connection blocks help a lot with preventing users from continuing to cheat once they've been banned, but we see the majority of cheaters are on home ISP's using dynamic IP's. Once there's a connection block applied to their IP, they can simply restart their router to get a new IP.
The dilemma of creating a new check
The first thing to understand is that AntiCheat development on a network size of CubeCraft's is a very, very long process. We shouldn’t have the expectations that we can eradicate the problems of cheaters in a short period of time. It can be a constant cat and mouse game between us and the client developers. We release a check, and they try to figure out a way around it.
We take multiple different approaches to detecting cheaters. We may sometimes look into popular hacked clients and figure out flaws that we can detect. We may look at a players behavior over a period of time to determine if what they are doing is common with a player that is cheating. Ideally, we look at how the Minecraft Vanilla client does something and apply that logic on the server. We can then use that on the server to detect if the player is sending data that does not come from a Vanilla client (in action this can get much more complicated).
A cool packet debugging tool we've developed to
help make debugging packets easier
These different approaches come with their pros and cons and vary in development time. Releasing a detection that detects a certain implementation of a cheat could catch a lot of cheaters quickly, but it servers very little long term protection. Once it has been bypassed, we are stuck in the same position as we were before. At the same time, working on a more concrete check that aims to give long term protection can mean we have to bare with the situation we have now until we can finalize this check. This can be sometimes difficult to deal with, especially with a growing pressure from the community to deal with cheaters.
The biggest problem with a detection is the false positives that it may bring. We spend most of our time not on creating new checks, while fixing issues with our current ones. False positives are our biggest priority as we cannot allow a check to start banning if there are instances where a legit player can be flagged. A detection can have a great detection rate, but the false positives prevent it from being able to ban. This can be tricky to tackle as they may not always mean a simple fix. It can be also difficult as we are mostly relying on the violation data associated with an alert to try and fix the issue without having a step by step on how to recreate the false positive. Having a lot of false positives with a check usually means there is a fundamental flaw with how the detection works. At this point, we need to decide if we should continue to try and fix all of these issues or completely scrap our work and start on something new.
When a player is detected for cheating by a new detection,
an alert is sent for staff to verify if the player was cheating or not
Now with Minecraft Bedrock edition, creating an anti-cheat is extremely difficult. The game is still incredibly easy to create mods for, but we have no access to view its source code and therefore create checks based on its internal algorithms. So we're flying blind, making checks created solely from trial and error. This is why development for Sentinel Bedrock is extremely slow. And because we don't have source code or documentation, any false positives that occur are impossible to diagnose with certainty. This is especially a problem when our Bedrock network reaches player counts of over 20,000 players daily, while being harder to develop for than our Java network. Creating an anti-cheat for a closed source video-game is incredibly difficult, which is why our Java Sentinel is much more mature.
We're asking our loyal Bedrock players to please be patient, because we're trying our best despite the circumstances.
Our first introduction to Sentinel
When we were first introduced to Sentinel at the end of May, there was a lot of things we knew we had to cover. There were a lot of issues with the current detections and various clients that were able to bypass most of our detections. Having detections that had a lot of false positives and little detection rate of cheaters was a problem.
Since then, we have introduced many different new checks that are meant to replace the old ones, as well as checks to detect cheaters that were not already detected. We've also done a lot of work to the internals of Sentinel to make creating new checks easier and to improve certain aspects of Sentinel to help improve performance and accuracy. Most of the detections that were on the network when we first started have been removed or improved. Dawson (@funkemunky) has worked a lot in making our code framework a lot more stable and provide tools for stable detection making, as well as creating many new additional checks movement and combat wise. @Elevated has worked on patching numerous client exploits and has done an incredible job. @Hulk_ has been working on Bedrock Sentinel detections, exploits from clients in Minecraft Java, and many quality of life improvements for Moderator-Team/Sentinel interactions.
Our progress to Sentinel since we've started!
Apart from working directly on Sentinel, there's also a lot of moving parts outside of Sentinel. We work to maintain all things related to Sentinel. This includes appeals, bug fixing, and systems we use for staff involvement. Appeals take up most of our time besides programming. Appeals are very important to us as we want to make sure that if a player was falsely banned, they are unbanned as quickly as possible and we can work to fix the issue that caused the player to be banned. You may recognize Dawson and Hulk_ from the responses to your appeals, and @Elevated if you're lucky. Appeals give us a good guide as to how a detection is doing and allow us to spot any issues that we did not notice while the detection was in the testing stage. But, appeals can be time-consuming. After enabling a new detection for banning on Bedrock, we were seeing up to a hundred appeals a day which caused delays in how long an appeal took to be looked at. Since then, we've made adjustments to how appeals are handled to make sure that we are answering appeals in a timely manner (thanks @Elenahh for organizing this and to the entire moderation team for the help).
If you're lucky enough, you may be able to witness Sentinel in the player form
We've been working a lot to redesign Sentinel to make things easier for staff and for future development. We work closely with the staff team as they provide most of the feedback about the detections in testing. There will never be a time where Sentinel will be able to detect all cheaters, so we want to make sure that we are able to provide the tools for staff to be able to quickly deal with the undetected cheaters. Staff work to fill in the areas that Sentinel lacks in, while Sentinel works to help make things manageable for staff.
The perfect AntiCheat - What's to come next?
The perfect AntiCheat will probably never exist. Cheaters will be constantly changing their techniques as we play the catch up game. But, given enough time we can work to create something that minimizes the cheater's impact on a game. We still have a lot of plans for both Java and Bedrock to improve all aspects of Sentinel.
A more detailed look into our Sentinel ban statistics
On Java, we plan to continue to improve our existing checks and put more focus on preventing movement related cheats. There are some major changes coming but we are continuing to test to ensure there are no issues. Currently, we are also working to improve our existing detections before introducing anything new. Many new detections have been implemented on Java that have been tweaked and improved, and we will continue to do so into the future. We have more plans in the future to add more detections for minor cheats that we did not focus on at the beginning. In fact, we already detect a lot of "ghost" cheats that players use to provide an unfair advantage without being blatant and plan to implement much more. We definitely enjoy the feedback from our community regarding this issue and are working towards a completely fair competitive community.
On Bedrock, we have a lot of plans for future detections to help improve the situation. Yes, we have heard your cries! We are doing our absolute best but its been quite a challenge. We do not have the knowledge or tools that we do with Java Edition so creating new detections can take much longer. It can also be a challenge managing both Bedrock and Java at the same time since these games have similar but fundamentally different mechanics.
Our current plans are to focus on movement and combat related cheats. We have already released a couple of detections already automatically removing cheaters but we are hoping to work on something that serves better long-term protection and a lower false-positive rate. As we did with Java, we want to work more heavily on the internals of Sentinel on Bedrock to help make creating new checks in the future easier.
Looking back at when we first started, Sentinel has come a long way. We are continuing to learn on what works best and the best approach to dealing with cheaters. All of us have learned a lot and are still constantly improving our methodology. We know it can be hard to understand what is going on behind the scenes with Sentinel as most information is kept private. Every day we work on Sentinel and it can be frustrating that the community is unable to see the progress we make. We hope that this blog will give better insight into what we've been doing and the challenges we need to overcome.
Written by:
Welcome to the second edition of "Behind the Cube"! We hope these development blogs can give you guys some insight into what we are doing behind the scenes. In this edition, we will be discussing the development of Sentinel since we've started in May.
Why is cheating such a problem?Minecraft allows for a lot of creativity and freedom within the game. In fact, this is the very reason we can create awesome experiences and work on the detection of cheats on the server side with great precision. Minecraft would not be the same without the modding community. However, the darker side of the modding community is when modifications are created to promote unfair advantages. This is where we come in.
The main thing that separates Minecraft from most other online games is that movement is client authoritative, so your game session calculates your motion from your inputs. When you move, your position is sent to the server for it to be updated. On the other hand, server authoritative movement means all motion is calculated on the server's side. When you press the W key to move forward, that action is sent to the server and the server will attempt to calculate where your next position will be. Then, your position sent back to you to tell you where you should be. It’s a bit more complicated than that but that is the general idea (Here’s an article about it if you’re interested in how other online games handle movement). The Minecraft protocol gives a lot of authority to the client that allows them to create destructive cheats. However, Minecraft is designed in a way where we can easily confirm almost everything sent from the client to check for cheats.
The people that are cheating on the network can be very smart. They can be quick to understand why they are being banned and adapt to avoid being banned. No longer are we fighting against a few people with a few clients, but a wide range of clients and large groups of people targeting the network being the first to try and find a bypass. The abundance of stolen/compromised accounts makes it easy for people to continue to come back after they’ve been banned. Once they’ve been banned, they can adjust their settings the next time they join in an attempt to bypass. This can be a difficult situation for us. On one hand, we want to remove the cheater as quickly as possible to minimize their impact on the game. On the other hand, quickly banning a player makes it easier for them to understand why they've been banned. We have to find a compromise between making sure the cheaters impact on the game is minimal, while also preventing them from learning why they've been banned so quickly. Connection blocks help a lot with preventing users from continuing to cheat once they've been banned, but we see the majority of cheaters are on home ISP's using dynamic IP's. Once there's a connection block applied to their IP, they can simply restart their router to get a new IP.
The dilemma of creating a new checkThe first thing to understand is that AntiCheat development on a network size of CubeCraft's is a very, very long process. We shouldn’t have the expectations that we can eradicate the problems of cheaters in a short period of time. It can be a constant cat and mouse game between us and the client developers. We release a check, and they try to figure out a way around it.
We take multiple different approaches to detecting cheaters. We may sometimes look into popular hacked clients and figure out flaws that we can detect. We may look at a players behavior over a period of time to determine if what they are doing is common with a player that is cheating. Ideally, we look at how the Minecraft Vanilla client does something and apply that logic on the server. We can then use that on the server to detect if the player is sending data that does not come from a Vanilla client (in action this can get much more complicated).
A cool packet debugging tool we've developed to
help make debugging packets easier
These different approaches come with their pros and cons and vary in development time. Releasing a detection that detects a certain implementation of a cheat could catch a lot of cheaters quickly, but it servers very little long term protection. Once it has been bypassed, we are stuck in the same position as we were before. At the same time, working on a more concrete check that aims to give long term protection can mean we have to bare with the situation we have now until we can finalize this check. This can be sometimes difficult to deal with, especially with a growing pressure from the community to deal with cheaters.
The biggest problem with a detection is the false positives that it may bring. We spend most of our time not on creating new checks, while fixing issues with our current ones. False positives are our biggest priority as we cannot allow a check to start banning if there are instances where a legit player can be flagged. A detection can have a great detection rate, but the false positives prevent it from being able to ban. This can be tricky to tackle as they may not always mean a simple fix. It can be also difficult as we are mostly relying on the violation data associated with an alert to try and fix the issue without having a step by step on how to recreate the false positive. Having a lot of false positives with a check usually means there is a fundamental flaw with how the detection works. At this point, we need to decide if we should continue to try and fix all of these issues or completely scrap our work and start on something new.
When a player is detected for cheating by a new detection,
an alert is sent for staff to verify if the player was cheating or not
Now with Minecraft Bedrock edition, creating an anti-cheat is extremely difficult. The game is still incredibly easy to create mods for, but we have no access to view its source code and therefore create checks based on its internal algorithms. So we're flying blind, making checks created solely from trial and error. This is why development for Sentinel Bedrock is extremely slow. And because we don't have source code or documentation, any false positives that occur are impossible to diagnose with certainty. This is especially a problem when our Bedrock network reaches player counts of over 20,000 players daily, while being harder to develop for than our Java network. Creating an anti-cheat for a closed source video-game is incredibly difficult, which is why our Java Sentinel is much more mature.
We're asking our loyal Bedrock players to please be patient, because we're trying our best despite the circumstances.
Our first introduction to SentinelWhen we were first introduced to Sentinel at the end of May, there was a lot of things we knew we had to cover. There were a lot of issues with the current detections and various clients that were able to bypass most of our detections. Having detections that had a lot of false positives and little detection rate of cheaters was a problem.
Since then, we have introduced many different new checks that are meant to replace the old ones, as well as checks to detect cheaters that were not already detected. We've also done a lot of work to the internals of Sentinel to make creating new checks easier and to improve certain aspects of Sentinel to help improve performance and accuracy. Most of the detections that were on the network when we first started have been removed or improved. Dawson (@funkemunky) has worked a lot in making our code framework a lot more stable and provide tools for stable detection making, as well as creating many new additional checks movement and combat wise. @Elevated has worked on patching numerous client exploits and has done an incredible job. @Hulk_ has been working on Bedrock Sentinel detections, exploits from clients in Minecraft Java, and many quality of life improvements for Moderator-Team/Sentinel interactions.
Our progress to Sentinel since we've started!
Apart from working directly on Sentinel, there's also a lot of moving parts outside of Sentinel. We work to maintain all things related to Sentinel. This includes appeals, bug fixing, and systems we use for staff involvement. Appeals take up most of our time besides programming. Appeals are very important to us as we want to make sure that if a player was falsely banned, they are unbanned as quickly as possible and we can work to fix the issue that caused the player to be banned. You may recognize Dawson and Hulk_ from the responses to your appeals, and @Elevated if you're lucky. Appeals give us a good guide as to how a detection is doing and allow us to spot any issues that we did not notice while the detection was in the testing stage. But, appeals can be time-consuming. After enabling a new detection for banning on Bedrock, we were seeing up to a hundred appeals a day which caused delays in how long an appeal took to be looked at. Since then, we've made adjustments to how appeals are handled to make sure that we are answering appeals in a timely manner (thanks @Elenahh for organizing this and to the entire moderation team for the help).
If you're lucky enough, you may be able to witness Sentinel in the player form
We've been working a lot to redesign Sentinel to make things easier for staff and for future development. We work closely with the staff team as they provide most of the feedback about the detections in testing. There will never be a time where Sentinel will be able to detect all cheaters, so we want to make sure that we are able to provide the tools for staff to be able to quickly deal with the undetected cheaters. Staff work to fill in the areas that Sentinel lacks in, while Sentinel works to help make things manageable for staff.
The perfect AntiCheat - What's to come next?The perfect AntiCheat will probably never exist. Cheaters will be constantly changing their techniques as we play the catch up game. But, given enough time we can work to create something that minimizes the cheater's impact on a game. We still have a lot of plans for both Java and Bedrock to improve all aspects of Sentinel.
A more detailed look into our Sentinel ban statistics
On Java, we plan to continue to improve our existing checks and put more focus on preventing movement related cheats. There are some major changes coming but we are continuing to test to ensure there are no issues. Currently, we are also working to improve our existing detections before introducing anything new. Many new detections have been implemented on Java that have been tweaked and improved, and we will continue to do so into the future. We have more plans in the future to add more detections for minor cheats that we did not focus on at the beginning. In fact, we already detect a lot of "ghost" cheats that players use to provide an unfair advantage without being blatant and plan to implement much more. We definitely enjoy the feedback from our community regarding this issue and are working towards a completely fair competitive community.
On Bedrock, we have a lot of plans for future detections to help improve the situation. Yes, we have heard your cries! We are doing our absolute best but its been quite a challenge. We do not have the knowledge or tools that we do with Java Edition so creating new detections can take much longer. It can also be a challenge managing both Bedrock and Java at the same time since these games have similar but fundamentally different mechanics.
Our current plans are to focus on movement and combat related cheats. We have already released a couple of detections already automatically removing cheaters but we are hoping to work on something that serves better long-term protection and a lower false-positive rate. As we did with Java, we want to work more heavily on the internals of Sentinel on Bedrock to help make creating new checks in the future easier.
Looking back at when we first started, Sentinel has come a long way. We are continuing to learn on what works best and the best approach to dealing with cheaters. All of us have learned a lot and are still constantly improving our methodology. We know it can be hard to understand what is going on behind the scenes with Sentinel as most information is kept private. Every day we work on Sentinel and it can be frustrating that the community is unable to see the progress we make. We hope that this blog will give better insight into what we've been doing and the challenges we need to overcome.
Written by:
- @Hulk_ - AntiCheat Developer
- @funkemunky - AntiCheat Developer
