Game engine web developers, 64 (2 A) doing and not doing protocol and API

By Antonio Harris,2015-11-12 00:23
77 views 0
Game engine web developers, 64 (2 A) doing and not doing protocol and API

    Game engine web developers, 64 (2 A) : doing and not

    doing protocol and API

    before this series of articles"Game engine 64 web developers to do and not do (a) : the client side, "Sergey when adding support network game engine are introduced in the customer while the attention. This article, Sergey will be combined with the actual combat, about the agreement and pay attention to the point of the API.

    This blog will continue to tell about to implement network to support the game engine, of course, there will also be analyzed all types except the browser based game and platform.

    The first article in the series, will focus on here does not involve protocol client application network development.This series include:

    ; Protocols and APIs

    ; Protocols and APIs (continued)

    ; Server-Side (Store-Process-and-Forward Architecture)

    ; Server-Side (deployment, optimizations, and testing)

    ; Great TCP-vs-UDP Debate

    ; UDP

    ; TCP

    ; Security (TLS/SSL)

    ; ……

    8 a. custom Marshalling: please use "simple streaming API

    DIY marshalling can be realized through a variety of ways.A simple and effective approach is to provide "simple streaming" compose/parse function, such as OutputMessage & compose_uint16 (OutputMessage &, uint16_t)/uint16_t

    parse_uint16 (Parser &) - for all need on the network transmission of data types.In this case, the OutputMessage is a class/structure, encapsulate the concept of a message, after add other attributes will increase, while the Parser is created by an input message object, it has a pointer to the input message and one for the moment occurred in the offset.

    Compose and parse the asymmetry between (Compose is aimed directly at the news, and parse need to create separate Parser object) is not fully enforced, but in practice it is a very good thing (in particular, which allows the contents in the message stored parsing, allowing repeated parsing, unchanged in analytical form of the message, etc.).Generally speaking, this simple method is also suitable for large-scale environment, but the games have more efforts are needed to keep the information consistency between the composer and the parser.

    A composing may be as follows:

    uint16_t abc, def;//initialized with some meaningful values

    OutputMessage msg;


    The corresponding parsing example is this:

    InputMessage& msg;//initialized with a valid incoming message

    Parser parser(msg);

    uint16_t abc = parser.parse_uint16();

    uint16_t def = parser.parse_uint16();

    This "simple streaming" compose/parse the API (and based on it, such as the following IDL, and different from compose/parse the API based on the size of the clear to handle function) of an advantage is to use what format is not important - fixed size or variable size (i.e., coding such as VLQ and null terminated string encoding is completely feasible)., on the other hand, it's incomparable performance (even if the caller to determine the size of the message in advance, it is also conducive to add similar void reserve (OutputMessage &, size_t max_sz); such a function).

    8 b. Custom Marshalling: provide some with IDL - to - code the IDL compiler

    To compose/parse a simple ascension is some sort of statement is used to describe the message (a - the IDL interface definition language) and compile it into

    compose_uint16 ()/parse_uint16 () sequence.Case, the statements look like an XML declaration.


     name=“def“ type=“uint16“ />



    Then you will need to provide a compiler, it reads the above statement and produce something similar to the following:

    struct idl_struct_XYZ {

     uint16_t abc;

     uint16_t def;

     void compose(OutputMessage& msg) {




     void parse(Parser& parser) {

     abc = parser.parse_uint16();

     def = parser.parse_uint16();



    struct idl_message_ZZZ {

     uint16_t abc;

     idl_struct_XYZ zzz;

     void compose(OutputMessage& msg) {




     void parse(Parser& parser) {

     abc = parser.parse_uint16();




    Implementing such a compiler is a very simple (most developers have a certain experience just a few days can be completed; by the way, using the Python such language is easier - took me half a day).

    It is important to note that the interface definition language does not require must be XML -- for example, for programmers familiar with YACC, parsing the same example, in C style to rewrite the IDL is not very difficult (again, the compiler does not need to take a few days - that is, if you have already used the Lex and YACC/Bison/Flex). struct XYZ {

     uint16 abc;

     uint16 def;


    message struct ZZZ {

     uint16 abc;

     struct XYZ;


    Another way of realization of marshalling is through RPC calls;In this case, the RPC function prototype is an IDL.It should be noted, however, is blocking type of RPC calls are not suitable for Internet applications (this will in Part IIb is discussed in detail in the # 12);Although item # 13, on the other hand, do not use the Unity 3 d style of no return to the starting point of a non-blocking RPC is good, I still like to structure mapping into news, because it can more clearly explain what is happening.

    8 c. Third party Marshalling: use the format of the platform and language independent

    For the C programming language, marshalling problem lies not in whether "marshal", but "what to marshalling".In theory, any serialization mechanism can do, but the fact is platform and language independent serialization or marshalling mechanism (such as JSON) than the specified platform and language (Python pickles, for example).

    8 d. for frequent internal interactive games that use binary format

    For data format, there is a strong but not a recent trend is to use text-based format (such as XML) is better than using binary format (for example VLQ or the ASN. 1 BER).This argument is necessary, for the game.Although text format can simplify the debugging and provide better interactivity, but they naturally large (the same is generally true even after the compression), and requires more processing time, it will be in the fire game up and give you a heavy blow (in both traffic and server CPU time).The author's experience is: for the high requirement of interactive processing, using binary format is usually more suitable for (although abnormal may depend on specific such as the change of volume, frequency, etc.).

    For binary format, in order to simplify the debugging and improve interactivity, can according to the analysis of IDL with a message and to implement in text format to print the independent procedure is very convenient.Even better way is to use a aimed at was debugging/logging library to do it.

    8 e. for frequent external interaction using a text format

    Different from internal interactive games, external interactions such as pay is usually based on text (XML), usually run well.For frequent external interaction, all parameters become less obvious for text format (due to the rare), but the debug/interoperability becomes more important.

    8 f. Please consider before abandoning the ASN. 1

    Asn.1 is a need to focus on the binary format (that is, strictly speaking, the ASN. 1 can also generate and parse the XML through XER).It allows the general marshalling, has its own IDL, applied to communication (ASN. 1 is the most commonly used on the Internet as an x. 509 certificate format).Binary marshalling and at first glance, it is needed.A look again, you may fall in love with it, may be because of the complex correlation and hate it, but you don't try, you never know.

    The author thinks that, the ASN. 1 is not obsession, it is very heavy, and similar streaming API born in performance have greatly increased, at the very least, can unless the ASN. 1) compiled into code, but it is not at all in the game.As a result, developers should look at the ASN. 1 and the available libraries (especially in an open source of

    ASN. 1 compiler [the ASN 1 c]), then according to specific project, whether it is appropriate.

    Use asn1c compilers, performance good ASN. 1 is closer to the above description of streaming, although the author of the ASN. 1 can match the simple streaming question (most for executing the ASN. 1 analysis need to significantly increase more configuration).However, if someone did benchmark testing, can reply, because after using asn1c difference is not obvious.In addition, if the performance difference substantially smaller (even in marshalling, 2 times the performance of the difference in the overall performance may be less obvious), and other such as development time becomes more important.And here, the ASN. 1 would be a good choice will depend on the specific details of the project.A need to pay attention to the problem: when it comes to development time, the time of game developers is more important than the engine developer time, therefore, need to consider what kind of IDL developers prefer - one is what is said above, or the ASN. 1 (by the way, if they prefer simple custom IDL, so still can be used in the underlying the ASN. 1, from the IDL to the ASN. 1 compiler, because it's not really complicated).

    Summary: although people really don't like the ASN. 1, but it may be useful (according to the above, please determine).

    8 g. Remember Little Endian/Big Endian warned

    Big endian - is the address of the low byte will is stored in memory.On the contrary, Little endian - is the low low byte is stored in the memory address.

    When on the C/C + + implementation ()/parse_ compose_ * * () function (multibyte expression), it is important to note that the same integer different sequence of bytes in the different platforms.For example, in "little endian" - system (especially the X86), (uint16_t) 1234 storage said to 0 xd2, 0 x04, and in the "big endian" - system (such as powerful AIX), the same (uint16_t) said 1234 0 x04, 0 xd2.That's why if only write "unit16_t x = 1234; send (socket, & x, 2);"In little endian and big endian - send a different data platform.

    In fact, for the game, this is not a real problem.Because of the need to deal with the vast majority of the CPU is Little endian - (X86 is Little endian, ARM can be a Little endian, can also be a Big endian, IOS and Android is currently Little endian).However, in order to guarantee the correctness, remember and select the best a method using the following:

    ; Word by word section marshal data (i.e., sending the first x > > 8, and then the x & 0 XFF -

    whether it is Little endian or Big endian - -, the result is the same).

    ; Use # ifdef BIG_ENDIAN (or # ifdef __i386, etc.), on different machines to produce different

    versions.Note: strictly speaking, the Big endian macro - enough to run based on the

    calculation of marshalling;(especially the SPARC architecture), it is difficult to read without

    alignment data, so I can't run.ARMv7 and CPU, however, the situation is more complicated:

    although technically, not all instructions to support this deviation, as a result of the

    marshalling code compiler dislocation tend to use safety instruction code generation, so

    based on the analysis of the calculation can be run;However, I still wouldn't give the ARM

    now using this method.

    ; Using the function, such as htons ()/ntohs (), note: these functions to generate the so-called

    "network byte order", which is Big endian - (so) happened.

    The final option usually is often recommended in the literature, however, in the practical application effect is not obvious: on the one hand, because of all the marshalling encapsulation processing;The second option ((# ifdef BIG_ENDIAN)) is also a good choice (when in 99% of the target machine using Little endian, - could save some time).On the other hand, can't see any performance differences can be observed.What is more important, remember that the exact implementation does not matter much.

    Personally, when the focus on performance, the author prefers the following method: a "general" section of the word for word version (it can run anywhere, regardless of the byte order and does not depend on the ability to read unaligned data), and then to platform features based on the calculation of the professional version (for example X86), for example:

    uint16_t parse_uint16(byte*& ptr) { //assuming little-endian order on the wire #if defined(__i386) || defined(__x86_64__) || defined(_M_IX86) || defined(_M_X64)

     uint16_t ret = *(uint16_t*)ptr;

     ptr += 2;

     return ret;


     byte low = *ptr++;

     return low | ((uint16_t)(*ptr++)) <<8;



    In this way, will get a reliable can work anywhere version (below "# else"), and a high-performance version of a platform based on interest.

    As for the other programming languages (such as Java) : as long as the underlying CPU is still little endian or big endian -, such as the Java language are not allowed to watch the difference, so the question is not to exist.

    8 h. Remember Buffer Overwrites the and Buffer Overreads

    When parsing program, make sure they are not easy to be abnormal packet attack (abnormal packets cannot result in buffer overflow, for example).Details please refer to the Part of the VIIb # 57.Another is important to remember that it is not only buffer overwrites is dangerous: buffer overreads (for example, to a null-terminated string that is said to be made of a packet call strlen (), once the character is clearly not null-terminated character) can lead to the core dump of 0 xc0000005 exceptions (Windows), is likely to destroy your program.

    9. To have a single network layer with a well-defined interfaces

    No matter what to do to the network, it should have a separate library (in the other game engines within or adjacent) to encapsulate all the network related.Although the function of the library is simple - soon, it may be evolution is very complex.And the library should be separated from other engine enough.This means "should not be confused with 3 d network together; the separation of them as far as possible".In short, the network library should not depend on graphics library, and vice versa.Note: for those who believe that no one could write a tightly coupled with the network engine graphics engine - take a look at Gecko/Mozilla, you will be quite surprised.

    Warning: network interface library need to make appropriate adjustments according to the requirements of application (must not blindly imitate the TCP sockets or are using other system level API).In the game, the task is usually send/receive information (with or without guarantee delivery), and the corresponding API libraries should be reflect it.For a good (although not universal) instances of abstract is the Unity 3 d: their network API provides information or unwarranted state synchronization, both for the task of real-time games are good abstraction choices.

    There are other is (in addition to the packaging system calls to your abstract API) belong to the network layer?More than one way to do it, but usually include everything, they will transfer the network information to the main thread (see Part I of the # 1), and in situ treatment.Similarly, marshalling/unmarshalling (see # 8) also belong to the network layer.

    There is no doubt that any system level network calls will only appear in the network layer, and absolutely should not be used in other places.The whole idea is to encapsulate the network layer and provide a clean separation of concerns, isolating the application level communication has nothing to do with the fine.

    10. To understand the underlying what is going on

    When developing network engine, the use of some frameworks (such as TCP sockets) looks very attractive (at least at first glance), it will automatically do a lot of things, do not need to focus on developers.However, if you want to make a better user experience, things become difficult.In short: although using a framework is worry, but completely ignore it is not good.In practice, it means that as long as the team more than 2 people, usually need to have a dedicated web developer - he knows framework underlying what is going on.

    In addition, the overall project architect must know at least most of the limitations of the Internet makes ('s inherent non guarantee IP packets, for example, how to ensure the accurate delivery, typical of the round-trip time, etc.), and all team members must understand network is a transmission messages, these messages could be any delays (guaranteed message delivery) or lost (unwarranted message transmission).

    Can be summed up in the following form:

    The team members skills

    The team members Everything about the library and the underlying


    Overall project architect The usual network limited

    All team members News on the Internet, as well as potential dela

    y or potential loss

    11. Don't assume that all users are using the same version of the App (that is, to provide a way to extend the game protocol)

    Although the program will automatically upgrade (including network library, etc.), or to remember those who haven't upgrade the APP to the user.Although each application startup forced to upgrade, there are still users to upgrade the moment are using the Internet, there are also some found the ignore to upgrade method (ignore the upgrade for many reasons, usually don't like to update the change).Deal with the problem of two commonly used methods are:

    ; Provide a mechanism for App developers will App and a App version protocol binding, check

    it on the server, let use overdue client user leave, forcing them to upgrade.

    ; Provides a way in the form of graceful degradation treatment protocol, the difference

    between previous versions are provided no function in the agreement.

    Take the second road is difficult, but it can give the end user feel extra comfortable (if done very carefully.In general, two mechanisms should be provided in the engine,

    make the app developers can choose according to the requirements (in the long run, even in life cycle is an app, they often need, both).

    A handling method 2 is based on the observation, in a nearly mature app, most agreements of change and adding new fields in the protocol are involved.This means that you can in marshalling layer provides a generic functions, such as end_of_parsing_reached (), so that app developers can add new field at the end of the message, and use the following code to parse may have changed.

    if( parser.end_of_parsing_reached() )

     additional_field = 1;


     additional_field = parser.parse_int();

    If you use your IDL (see # 8 b) above, it should look like.






    Of course, in compose ()/parse () will change accordingly.

    This simple method, that is, at the end of the message to add additional fields, running is good, although need to game developers understand is how to extend the agreement.Of course, not all agreements have can use this way to deal with change, but if the app developers can use this method to deal with more than 90% of the updated agreement, and will be forced to update to reduce the number of ten times, the user will be very grateful (may not, depending on the update of negative tired).

Report this document

For any questions or suggestions please email