Developer log
One Toxic Fog cast kicked a player off EU-1. Oops.
A Toxic Fog cast disconnected a player on EU-1. Here is why it happened, the four-step fix, and what the map server does now. Charts included.
By Xirune Team · November 6, 2026 · 11 min read

On EU-1, a player cast Toxic Fog on a pack of monsters and got disconnected. The session simply closed. This is the story of why, how we fixed it in four steps, and what we found when we looked at the map server afterwards.
If you only want the result: in our dense-fight test, every player used to be dropped within seconds. Now 1,800 out of 1,800 sessions finish the full minute.
These changes ship with tonight’s World server update.
Get the launcher · Join the waitlist · Join us on Discord
What happened on EU-1
The server told us nothing useful. The player’s session ended with “closed by the server”, no reason attached, a few moments after a Toxic Fog cast on a monster pack in the Arena. The process kept running and the player could log straight back in.
We added logging and reproduced it on a local server. The cause was a queue. Every player has a small outbound queue of 128 slots on the server, and the server used to put one slot’s worth of news in it for every single thing that happened: one hit, one poison tick, one monster state. When the queue was full, the server closed the session.
One spell pulse, tens of thousands of messages, a queue with 128 slots.
The arithmetic. In our test fight, 360 monsters stand in a pack and 100 players watch them. One Toxic Fog pulse poisons every monster, and each poisoned monster sends one message to each viewer: 360 × 100 = 36,000 queue entries in a single server turn. Add more casts and the number multiplies. The queue holds 128. Making it bigger would not have helped, because the work grows with casts × monsters × viewers.
There was a second kill path too. In our bigger test fights players were disconnected with the text “persistence unavailable”. It sounds like a database problem, and it was not. It was the map’s command queue (1,024 slots) filling up because one server turn took seconds, and the server answered by dropping the player. No database was involved.
Before: everyone kicked
We built a benchmark that is deliberately nasty: 360 monsters, equipped players, Toxic Fog and Hercules cast by every player, for 60 seconds, at 100, 300 and 500 players. Before the fix, every player at every size was disconnected within seconds.

Our first idea was to pack many events into one queue slot. It helped some bursts and still kicked 51 of 100 players, so we dropped it. A smaller bucket is still a bucket that fills.
The fix, in four steps
1. Never kick on a full queue. Resync instead. When a player’s queue is under pressure, the server now holds back, and then sends the player a fresh picture of what they can see. If a command cannot be served in time it gets an Overloaded answer, and the session stays open. The player might see a hitch, not a disconnect.
Under the hood: ordinary overflow of the outbound queue, the command queue or a player’s own command pipeline no longer closes a session. A player who is not draining at all for 20 seconds is still ended with a clear reason. Right after this step a handful of sessions still closed (3, 2 and 1 at 100, 300 and 500 players). That was an exception we had allowed for this step, and the next step removed the need for it.
2. One frame per player per tick. The server now builds at most one frame per player every 50 ms, instead of one message per event. A kill batch shows the idea: one Hercules cast that killed 145 monsters used to queue a message per kill. Now it is one batch. We checked in the real game client that all 145 deaths, kill credits and souls arrived.
Under the hood: a poison pulse on 360 monsters used to queue 361 messages per viewer. It is now one frame, and the queue between the map and the network went from 128 slots to 2. After this step, 1,800 of 1,800 benchmark sessions finished with zero kicks.
3. Delta rows, encoded once. A health change used to re-send a whole character. Now the server sends only what changed, and it encodes each row once and shares the same bytes with every viewer instead of rebuilding it for each one.
4. Byte budgets, distance tiers and a slow-client policy. Every player gets a byte budget per tick. Players far away (more than 12 tiles) are updated twice a second, nearby ones twenty times a second. Nobody is hidden. A slow connection gets fewer, newer frames and an occasional resync, and is not kicked.
We tested the slow case on purpose: 499 normal players and one on a link shaped to 256 kbit/s with 5% packet loss. All 500 finished the minute with no disconnects. The shaped player received about 23 KiB/s and needed one fresh resync.

Bandwidth per player at 100, 300 and 500 players went from 38.6, 84.1 and 56.6 KiB/s after step 1 to 12.9, 37.2 and 20.9 KiB/s after step 4. Step 1 is the first run where players stayed connected long enough to measure; before it, nobody lived through the minute, so there is no fair “before” number. Step 3 helped a lot at 100 players and left the 500 number about flat (28.6 to 29.3). We kept that bar in the chart.
The map server was doing too much work
With nobody being kicked, we could finally profile a full minute of 500 players in one spot. The map server (one worker per map) was busy for 58.8 of 60 seconds. Three loops used about 80% of that time, and all three did the same kind of work: walk every player or every monster, even when almost nothing had changed.
- A. A per-player timer sweep, about 29% of the time. Each player remembered when each visible thing was last sent, and every 50 ms the server swept that memory for every player, roughly 14 million lookups a second at 500 players.
- B. Item expiry, about 18%. Every player’s turn checked every player’s items for expiry, about 250,000 checks per tick for something that rarely changes.
- C. Monster rescans, about 30%. Every monster update rebuilt every nearby player’s complete list of visible monsters, about 540,000 lookups per update, whether one monster changed or all 360.
The fix for loop A is in: the server now forgets a timer when something leaves a player’s view, and does not sweep.

At 100 players the server uses 22.5% of a core, down to 14.1% when everyone casts at the same instant, and 22.7% down to 12.8% when casts are spread over a second. At 300 players CPU drops by 14 to 28%. At 500 players it still uses about 112% of a core, so one dense crowd of 500 is still heavy and the profile shows where the next work is.

What players feel is the answer rate. With casts spread over a second, 300 players saw 1,308 “Overloaded” refusals before and 0 after. With everyone casting in the same instant at 300 players, the refusals stayed about the same (2,600 to 2,610). At 500 players the server accepted more casts: 780 to 1,422 when everyone casts at once, and 1,433 to 1,651 when spread out.
Fixes B and C: about half the CPU, a quarter of the wait
Fixes B and C are in, and they are the ones that move the needle.
B. Remember item expiry until the inventory changes. The server used to check every item of every player for expiry on every tick, for something that almost never changes. Now it remembers the result and only checks again when that player’s inventory changes. Items still expire on time.
C. Keep track of who sees which monster as they move. The server used to rebuild every nearby player’s full list of visible monsters on every monster update. Now it keeps that list up to date as players and monsters move, and touches only what changed. We ran the new bookkeeping next to the old full rescan in a debug check (1,000 ticks, 2,653 comparisons) and saw 0 differences.

CPU is the headline. When everyone casts at once, the server went from 112% of a core to 106% (B) to 63% (C) at 500 players, from 75% to 57% to 29% at 300, and from 15% to 14% to 7% at 100. With casts spread over a second, 500 players went from 114% to 102% to 64%. A single dense crowd of 500 now fits in about two thirds of one core in this test.
The “after A” bars are a fresh control run taken right before B, so they differ a little from the fix A chart above (at 300 players, 75% here and 79% there). Each fix is compared with its own same-day control.
Slow turns shrank even more. The slowest 1% of map turns (p99, rounded up to a bucket) went from 10 to 2 ms at 100 players, 20 to 5 ms at 300 and 50 to 10 ms at 500, between B and C.
What players feel is fewer “Overloaded” refusals:
| Overloaded refusals | before B | after B | after C |
|---|---|---|---|
| 300 players, all at once | 2,525 | 1,021 | 0 |
| 500 players, all at once | 13,027 | 12,150 | 566 |
| 500 players, spread over a second | 12,874 | 10,445 | 23 |
That is 95 to 99.8% fewer refusals at 500 players between B and C. The server also did more real work: at 500 players casting at once it accepted 1,675 casts after B and 2,994 after C. And the safety result held: 1,800 of 1,800 sessions finished again, with 0 kicks, on both B and C.
C is measured against B as one cumulative step. It also includes a small fix to the order in which the server flushes view changes, so we do not claim how much of the gain belongs to each line of the change.
What this does not tell you
- It is a benchmark, not a live fight. The test players are scripted clients on one machine, casting Toxic Fog every 20 seconds and Hercules shortly after, on 360 monsters. Real fights look different, but the EU-1 disconnect shows the heavy case does happen.
- One run per point. We ran each data point once on a shared machine, so treat small differences as noise. The big ones, like 100% kicked to 0%, are not.
- Steps 1 to 4 were measured on a development-optimized build, and fixes A, B and C on a release build. Nothing here claims how many players EU-1 can hold on the internet.
- The bandwidth figures are raw protocol bytes before any network overhead.
- 500 players in one spot is much lighter now, not solved. At 500 players the server uses about 63% of one core, and “Overloaded” refusals fell by 95 to 99.8%, but 566 refusals in one run is not zero. Fixes B and C are still one pair of runs per point on a shared machine.
Help us test
Update through the launcher and enter EU-1. If you ever get disconnected in a big fight, press F9 and tell us what you cast and how many monsters and players were around. That helps us find the next one.
