How I failed my first Minecraft event launch, and then made it right the second time.
# Chapter 1
Impending Doom
Me and my friends have been working on a PvP event in Minecraft for 60 people. It was a simple event with 4 kingdoms, 15 players residing in each one.
Not everything was going our way. We were running out of time, and so we had to make a number of compromises, as well as making some mistakes along the way, one of which bit us in the ass hard.
I was taking care of the technical aspect of the server, as well as making scripts for the event (using Skript), while my friends were taking care of location-specific command sequences using command blocks for some of our actors. Besides the fact that we were rushing and extremely nervous, it was fine. We were doing things on-time, and so the day for the event came.
Before we started the event, we realized that we forgot to make world-wide barriers to restrict players from entering each others' kingdoms. This realization came minutes before the event started. All four of us collectively started making barriers, all while players were having fun in the lobby. A whole 30 seconds before the event start, we finished. Phew, that was a doozy, everything's gonna be ok now… right?
# Chapter 2
Hell Unleashed
All of us actors get into position, anticipating the start of an epic event, with epic dialogue and epic TPS, but, alas, that is not meant to be. The moment we started teleporting players to their kingdoms, TPS took a large dip, while MSPT skyrocketed into the fucking stratosphere! What is this? I double- no, I triple-checked that performance and the hardware should be optimal! Not even the fact that we were using a VPS should've been an issue, at least not of that magnitude. While my thoughts were racing thinking on the next step, I watched as TPS went below 1, and people's ping decided to join MSPT in its journey to cosmos.
Eventually, people were being kicked due to timeout, and the server was crumbling to dust. While that was happening, I attempted to open the Spark report, maybe there I'll find out what's going wrong? That I thought, foolishly. Of course, it showed nothing of value. The mob count was relatively high, yes, but this is not what's causing issues. I look at the function causing the slowdown while the UI is lagging like crazy, and I see… nothing. Of course I see nothing! All it says is that tick processing takes a shit-ton of time, but it says nothing on where it takes such a long time!
The server crashed… It could not handle load. But why? I know the hardware was good, it was working perfectly, even when players joined and were waiting in the lobby that was up in the sky. The problems started after we teleported them and loaded more regions, but what could even be causing the issue? I didn't have much time to think before my friend named MommyWithBalls (not her real name) brought 2 extremely smart and experienced sys-admins.
# Chapter 3
Reflection
I was in a state of despair. Deeply saddened by what happened that noon, I was listening as Mike and Guide (not their real names) were discussing what might've caused the issue. The only conclusion they could come up with after consulting with me on the optimizations and how I've set up the server, is that VPS was at fault.
"…and so you need a VDS", Mike said, "otherwise, you'll be experiencing the same issues! I have no doubt that this is the cause of the issue, since everything else seems to be ok". Guide was saying the same thing, although, he was the one to recommend us the VPS we were using. "That's still really strange…", Guide said, "I was certain that VPS could handle a mere 60 players. There might've been slight dips in TPS from time to time, but I'm surprised that it was that bad. I've never seen a VPS lag that much with such a small amount of players before".
I must note that the server was well optimized for a Purpur server. Both rendering and simulation distance were set at 6, and everything was configured to squeeze as much performance as possible, to an excessive degree, even.
I was now convinced that it was a VPS issue. We needed a VDS provider, and so Mike had one he could recommend. He said that HostBrr has really cheap and very good VDS hosting, and he helped us find a good deal! A 9950X with 24GB RAM, sign me up! But before we even thought about renting it, I needed a plan on how to set it up, with which both helped me do as well.
Both Mike and Guide went their own way after we were done making the plan. We thanked MommyWithBalls for helping us to calm down, and for calling up the sys-squad to consult me. I was feeling like a child, I felt like I knew nothing after talking to them, and it was thrilling! I felt like I was learning programming again! But it was no time to be happy to know there's a new ceiling to reach, it was time to revise and check the plan. Let's see…
# Chapter 4
Big Plans
You may skip this chapter if you don't want to hear me yap about very smart and a little complicated compooter thingies.
Besides doing the mundane stuff, like updating the system, setting up SSH, disabling root, etc. There was the more interesting stuff I've yet to touch upon.
The only thing I knew from the more interesting stuff was ZRAM, which is basically like downloading free RAM. It works by making a certain portion of system RAM be SWAP space, but it is no ordinary SWAP space, because we explicitly encourage the Linux kernel to shove stuff in there, even if it's accessed more often. You might be thinking that it is going to tank the CPU, but it won't, because we're going to use the holy grail of compression — ZSTD! It's an extremely efficient algorithm that is only slightly worse than XZ in its compression ratios, but unlike XZ, it's BLAZINGLY FAST (not written in Rust, though). Now, by setting it to 50%, we, effectively, nearly doubled our RAM!
I also had to setup XanMod Kernel and Google BBR networking stack. This would make network throughput much better. You'd think that making it work requires significant fuckery on my end, but no, it's basically a one-line install, for both of these.
After this will be done, I'll just need to install JDK Temurin and that's it. Good thing I had been using it on my local instance of the server I've been testing with before I'd get my hands on the server… Oh, shoot, we need to get the server.
# Chapter 5
Acquiring Hardware
After some time of being tolerant towards banks being incredibly annoying, we were able to acquire the server. Eureka! It is now in our grasp!
I've successfully installed Debian 13 on it, fucked up SSH, reinstalled the system because I forgot the password (twice), and finally stopped being a dumbass and got it to work.
I've updated the system, checked for known vulnerabilities, disabled SSH-ing as root, installed Fail2Ban, Firewall, ZRAM XanMod, etc. And now, I finally had it all setup.
Hmm? Mike wrote a message telling me to check the hardware. I do as he says, and it's… it's the wrong hardware… No problem, the hardware is close to what I ordered, but I would still like my ordered hardware. I make a support ticket, and in the meantime chat with Mike.
He suggested to also do some IP checks on how epic the internet on that beast is. It's really, really good, and, what's even better, the subnet we're on is not blacklisted by RKN! This is huge! The event was targeted at a Russian-speaking audience, with most players (obviously) being from Russia. This meant we don't need any VPN or Reverse Proxy shenanigans.
Meanwhile, we were waiting for an answer…
# Chapter 6
Playerbase Loss
Many players, dissatisfied, said that they won't be on the next launch, and some players said that they couldn't be on the next launch due to being occupied on the day of the next launch (which, to clarify, was the same day of the next week).
This is bad, the event relies on players being balanced between kingdoms. If some players on one team left while others were full, it will not be fair. We managed to get a couple of players from the outside to join our event, but it was not nearly enough to compensate for the loss of players on one of the teams. The worst part? We couldn't just fling players left and right to different kingdoms, because otherwise they would just become traitors, dissatisfied with the fact they were taken away from the community of their kingdom because of some other players leaving. We had to leave it as is, hoping for the best.
# Chapter 7
Locationality And Hardware Complications
An email! Support answered, saying they can begin right away, I just have to tell them when I'm ready. I stop the server and send the message. At the end of the day, I get a message that the migration to new hardware was complete, and the hardware was correct this time! I go and do the checks again and… This subnet is throttled by RKN. This is horrible! We don't have a reverse proxy, nor do we have the resources to rent a machine for one. And even if we do, the reverse proxy itself has to be in just the right place so that ping is good.
I frantically write another ticket.
"After migration to the new hardware, I've noticed a drastic degradation in speed to and from Russian regions. Here's the speeds [Both Reports]. Could you, please, send us to the old subnet, or one that is not heavily throttled by RKN, so the speed can be restored?"
I get another message, confirming that they can do that. I stop the server and ask them to begin. After it is done, I check again, and there it is, speed is restored! However, I go and check the hardware and it's not the one I rented. I ask about it in the ticket and they say that it's the same hardware, but due to a bug, they can't enable CPU Passthrough, "but it's fine", Mike says, "it's only 15-20% slower than if it were through CPU Passthrough". The hardware is still powerful enough to handle 60 people easily, so I thank them and close the ticket. I can't risk losing that subnet.
# Chapter 8
Mistakes That Keep On Giving
I finished up some scripts on the local Minecraft server instance, and then I finally sent it over to the server. And we were ballin'. Me and my comrades were actively working on the event, making sure it's all going to be perfect this time, but there was a problem, and it was not a technical one…
After much time in preparation, we finished doing all that needed doing. TPS was marvelous, MSPT was lookin' absolutely sexy, and ping was a chef's kiss. 50-ish players prancing around in the lobby, minutes before event starts. We're ready this time.
Me and other actors get into our designated zones and nervously wait. No, it was just me, I was the one nervous. Finally, after all of this suffering, people will finally be able to play! And so the event starts a… Wait, huh? What's with TPS? No, no, no, NO, NO, NO, NO! The issue is back again. But why? We migrated, this can't be a hardware issue anymore, there is no way. Spark report shows the exact same thing as the last time this catastrophe happened.
I call in Guide to help out debug the issue. We cut mob spawning, tweak some settings, but nothing works. We investigate further, and one of my friends who was doing the map, out of nowhere, decided to break one command block, and then it clicks. The issues were caused by that one singular command block. We asked him what it did and it was basically along the lines of this:
/team join MOD @e[type=!player]This singular command was assigning every single entity to the same team, every tick. And just imagine, there were people around the map, each with their own mob cap. There were thousands of mobs, and remember, this is not some fancy Folia where you can multithread that process, no, it was all being ran on a single core. Let that sink in. And the worst part? I was the one who told him that that command was fine when I was asked about it. I let that happen, I made my own nightmares.
# Chapter 9
God, Please, Let This Be Over
Did things go smooth after that? Not quite. Remember the missing players?
The team with least players also had the least amount of seasoned fighters, most left because our first failure. It was the most unfun experience for them, they were rolled like a cigar by literally one seasoned fighter from another kingdom. They were not happy, to say the least.
As a side note on actors, it was the only thing that went smoothly.
Later on, nearing the end of the event, there was a cheater we caught too late, and after that, one of the less experienced players in combat gave his account to the one guy who stomped the small kingdom earlier. Neither of these were a pleasant experience to anyone involved, but after that was done, it went more or less normally.
# Epilogue
What's the moral of the story? Minecraft is a cursed game, go play something else LMAO XDDDDDD.
We later on realized that this problem would've been much easier to track if we were on Folia, since ticking is multi-threaded. We'd directly saw which region is causing the lag.
In general, just don't use command blocks. Ever. Learn writing plugins, don't repeat my mistakes.
# Bonus Chapter
Conflicting Plugins
I had certain issues trying to get TAB, Essentials, and FlectonePulse to work together without conflicting. You see, all these plugins do more than could be implied from their name alone. TAB doesn't just manage the player list, it also manages scoreboards, nametags, belowname objectives and it dynamically creates and manages /teams in order to do its name formatting. Oh, yeah, it also does name formatting, which all of them do. See the issue?
I had to selectively disable each feature that was conflicting, and disable name formatting in all of these plugins, because we needed the /team system for our event. For some reason, it's the only way in Minecraft to color a name, give it a prefix and a suffix and do some other cool things, like nametag hiding.