Introduction

This blogpost will give you a glimpse into the the MechWarrior: Living Legends development process that our team undertakes when bugs are identified. In this example, you will see that even issues perceived as relatively low in severity can tie up several resources to fully resolve and ensure all existing features work as intended after a fix is implemented. Please understand that we are not able to supply you with code snippets or screenshots of actual work so this update is rather verbose. Even so, we will attempt to give you a better idea of how much work goes into this project, even for small things that may be perceived to be “easy fixes” in the eye of a player.

Example Case

For the purposes of illustration I am using an bug that was recently address with the ‘Mech ejection system. The bug was fixed as of BETA 0.2.0, but it still provides an excellent example. The bug was as follows:

1. Player “active” ejects from a ‘Mech using “F” control that is currently affiliated to their team.
2. The “head” armour component is flagged as destroyed.
3. Player re-enters the vehicle.
4. Player can “active” eject again.

The intended functionality of the ”active” ejection feature should be:

1. Player “active” ejects from a ‘Mech using “F” control that is currently affiliated to their team.
2. The “head” armour component is flagged as destroyed.
3. Player re-enters the vehicle.
4. Player cannot “active” eject again (using “F”), but passive ejection (using ”F”) is provided as an alternate escape mechanism.

Players should not be able to issue an “active” eject request again when the head component has been destroyed.  The design was intended to simulate the destruction of a one-time-use ejection mechanism of a ‘Mech cockpit.  Further complicating this issue:

1. Player “active” ejects from a ‘Mech that is currently affiliated to their team.
2. The “head” armour component is flagged as destroyed.
3. An enemy player captures this ‘Mech by boarding it.
4. The enemy player is not able to eject at all unless they power down the vehicle first (using “P” and then “F”).

This created a disparity in behaviour between the ejection sequence between “friendly” and “captured” ‘Mechs.  The incorrect functionality was reported by our Alpha Tester team and then evaluated against the intended design.  Next we will discuss how the analysis of the error and resulting remedy were undertaken.

Step 1 – Locate the Issue

Logically, the issue was determined to reside within the customized eject code of our game. Locating the specific offending code block  involved took quite some time since the eject system spans across 3 different files in our source code.To fully isolate the specific function that was causing the issue, our programmers created code breaks to step through the ejection feature.  This setup the next step in the process; unit testing through the repro steps provided and analyzing the chain of events that are invoked by the code.

Step 2 – Chain of Events

When the ejection system is triggered the system goes through the following chain of events to give the player the end result of being ejects out of a vehicle:

1. Player requests the Eject
2. System issues a “Eject Request” to the player input
3. The player input request is captured by the vehicle controls which triggers the “Initiate Eject” event
4. Display the Eject timer on the HUD
5. Lock other functions of the vehicle
6. Checks if vehicle is powered up, and chooses ejection type as a result
7. The vehicle sends back a “Initiate Eject” to the player input
8. Player input triggers the physical ejection after the timer runs out which adds the eject impulse to the player character
9. The player exits the vehicle and the eject impulse is applied

The chain may seem overly complicated, but working with the mod SDK can be difficult. Without access to the CryEngine vehicle code, we have to create work arounds, and in many cases to simulate behavior that would otherwise be easy to add directly to the engine vehicle code.

As you might notice the above chain of events does not have a function to disallow players from being able to eject again once an initial ejection has taken place. So at a minimum we had to extend the code with the required feature.

Step 3 – Add the New System to the Current Code

Since it was determined that there was no system in place capable of offering what the design dictated, we had to plug it in to our already existing code that handles the ejection sequence. Or first idea was to simply remove the ejection seat from the vehicle. Each vehicle is initialized with equipment. The ejection seat itself is one of several pieces of equipment that are added to the vehicle configuration in code when the vehicle is initialised (during power up). One simple solution would be to remove the ejection equipment once an ejection sequence was requested. The implementation was easily done by adding a parameter which told the engine: You now do not have an eject seat anymore.

Unfortunately, it was not that simple. There were unforeseen complications from adding the feature in this manner:

1. A player ejects.
2. The player re-enters the vehicle.
3. The player is not able to exit the vehicle anymore at all until powered down.
4. After power down, the player can exit by hopping out.
5. After re-entering the vehicle the player is able to eject again (because the vehicle power-on sequence is called, reinitializing the vehicle).

So what went wrong? Just removing the equipment from the ‘Mech did not have the desired result. Back to combing through the code!

Step 4 – Debug the New Implementation

As easy as it was to implement the removal of the ejection seat it did not reach all of the desired outcomes. The issue lies within CryEngine2: Once a vehicle is abandoned the ownership goes from the client to the server. The client set the flag that the ejection seat has been used locally (by the player) but as soon as the server takes over the vehicle the seat is reinitialized through the configuration and ownership change, and hence the vehicle now has a new ejection seat. After re-entering the vehicle as a client we take this data from the server (as we claim ownership of that vehicle) which also includes the reinstalled ejection seat. So what now?

Step 5 – Adjust the Implementation

After figuring out that CryEngine2 was actually the source of this issue we had to find a work around. How do we tell the system that an ejection seat has been used and is now not available anymore? We do have a variable storing the “used” state of the ejection seat but as described it gets reset as soon as the server takes control. Here are the two approaches that our coders used to broadcast the “used” state to everyone:

1. Serialize the value of our variable across the network
2. Invoke a RMI (Remote Method Invocation) call across the network on all clients and the server

Method 1 – Serialization

After adding the code required to tell the engine to update our value on all systems we tested again. But as you can guess this did not fix the issue. This time, the source of the issue was found to be with how CryEngine2 handles value serialization. The issue, as described in Step 4, was also preventing our value from being serialized in a way that we needed it to be. The explanation is simple: even when serializing a value from the client to the server, once the server takes ownership it re-initializes the vehicle (loading the equipment information) and serializes our reset “eject used” value across the network. A nice attempt, but we were no further along in remedying this “little” issue.

Method 2 – RMI (Remote Method Invocation)

First, a quick explanation on how RMI is used in a computer game. When an RMI call is issued, the client or server  asks all connected nodes to run a certain code function. Crysis and MWLL use this approach for many “one shot” network events. For our ejection seat, this actually did what we needed: Tell everyone connected to the game session that the ejection seat has been used and make it unavailable for every player trying to use it in this vehicle. So our code function called by the RMI would set a variable on all clients for that specific vehicle that the ejection seat has been destroyed and is non-functional for “active” ejection.

Step 6 – Verification

The last step in the implementation process is to verify everything works as intended and that no other features have been impacted. While working on it we had already established a repro steps to trigger the bug. These repro steps are now repeated by our tester team using the corrected code. Assuming that the issue has been resolved and that no additional bugs have been introduced via the fix, the feature is marked as FIXED and the code is added to the next build.

Summary

We hope that this has provided you with a little insight into our development process and the complexities therein. The team are working long hours every week to make things like this happen as well as bring you new features. We know that the majority of players don’t have an understanding of what is really involved behind the scenes when hitting the “F” key on your keyboard and ejecting from your vehicle in the heat of battle. After reading this post, we hope that you have an increased appreciation of all the features we add to make Mechwarrior: Living Legends possible for everyone to enjoy.

Thank you to Toth and KingLeerUK for technical and authoring assistance.

Bookmark It