Large Internet application logging system

By Bruce Washington,2015-11-19 23:11
12 views 0
Large Internet application logging system

    Large Internet application logging system

    The outstanding characteristic of large Internet application application itself is the size of complicated structure, big user visits.Design log system, can help you to analyze traffic trends, help manage the network application;Help, the problems of the application quickly find problems, time available to the application of network.Design a set of perfect log system, used to record the application of internal behavior, is a very valuable work.

    Hope from the Angle of the design and implementation, this paper describes the build log system.Description of this article, based on the Java language, as well as the implementation using Java language libraries.Readers can according to their actual demand, replacing the realization of the other. Log system is introduced

    The simplest logging System is in the System to add System. Out. The println (" Some log ") statement.As shown in listing 1.

    1 public void someFunction() {

    2 System. out .println(enter method);

    3try {

    4throw new Exception(Some exception);

    5 } catch (Exception e) {

    6 System. out .println(Failed: expcetion + e + thrown out);


    8 System. out .println(Success);

    9 System. out .println(exit method);


    List 1 to use the System. The out. Println implementation of the most simple logging System

    This is a simple logging system records for two kinds of important information, are respectively the call stack method invocation information (line 1 and line 9), and the success of method invocation information (line 6 and line

    8).But because too humble, such implementation, and can't appear in actual large-scale network applications.

    For Java developers, familiar with the used to implement the logging system framework including, Log4J and Java. Java libraries bring bag util. Logging.

    1. Logger logger =

    2. Logger.getLogger(PackageMatcher.class.getSimpleName());

    3. public void someFunctionWithLogger() {

    4. logger.log(Level.INFO, enter method);

    5. try {

    6. throw new Exception(Some exception);

    7. } catch (Exception e) {

    8. logger.log(Level.SEVERE,

    9. Failed: expcetion + e + thrown out);


    11. logger.log(Level.INFO, exit method);


    Listing 2 USES Java. Util. Logging implementation of log system

    First of all, from the performance, the two frameworks can meet the needs of large-scale access, then add the necessary infrastructure, such as the concept of Level (line 4, line 8 and line 11), Logger parent/child relationship, etc.But the framework of the problem is that they did not provide the domain model (domain model).Such as the list does not distinguish between 1 said the two types of information.This approach to log system is still relatively rough, the call stack information and method call success information is output in the same log file.Need extra work to extract the corresponding information.

    Design the perfect logging system

    For a large Internet application, it is very important, is to measure website traffic and access trend of anomalies can accurately and timely record system problems, and use these data to help diagnose problems in the website.In order

    to meet these requirements, the logging system can be divided into several modules to record the relevant information.Figure 1 shows a logging system module chart

    Figure 1 large Internet application logging system

    In this design, the logging system has four subsystems.Path tracking system, local log system, centralized logging system, and the reference counting system.These four subsystems, record the different categories of information, to manage and debug, provide support.

    Local log system

    A large web application, were deployed on a large number of machine cluster above.For each physical machine, when running in the application of the module problems, first of all, on the machine to record the current error phenomenon.Listing 3 presents a recorded in the log segment on the unit

    2008-12-7 21:57:24 SomeClass someFunctionWithLogger

    INFO: enter method

    2008-12-7 21:57:24 SomeClass someFunctionWithLogger

    SEVERE: Failed: expcetion java.lang.Exception: Some exception thrown out.

    2008-12-7 21:57:24 SomeClass someFunctionWithLogger

    INFO: exit method

    List 3 are recorded in the log on

    Because this is a real time recording system, all system exceptions can be recorded accurately and on time.In general, a local logging system consists of four main parts, figure 2 shows the local log system framework

    Figure 2 local log system

    Can be seen from the diagram, the four parts are respectively entries (LogRecord), recorder (Logger), a processor (Handler), Filter (Filter).Recorder

can also have a father and son relationship (such as use the Decorator pattern

    implementation).When children nodes after processing the log, can continue to

    the parent node.Listing 4 shows a local log reference implementation framework

    of the system.

    1 public class Logger {

    // parent logger

    2private Logger parent;

    // add filter

    3public void addFilter(Filter filter) { 4.}

    // add handler

    5public void addHandler(Handler handler) { 6.}

    // log

    7public void log(LogRecord record) { 8.}


    10public interface Handler {

    11void addFilter(Filter filter);


    13public interface Filter {

    14void filter(LogRecord record);


    16public class LogRecord {

    // some fields


    List four local log system implementation framework of reference

    In fulfilling the local log system, can record the related information in LogRecord class design.According to the list 3 shows the output, for example, can add the Date in LogRecord (Java. Util. Date), IP (String []), the ClassName (String), MethodName (String), Level (int), the Message (String), and other domains.One Level for marking the log message Level, higher than MouJi other messages will be output.Handler and the Filter also do filtering, can provide great flexibility for logging system.

    Centralized logging system

    Local log system can record system on a machine running situation.Although the timely, accurate, but due to the module on a single machine in general is only a part of the whole system, so the fault is not global monitoring system.Need a log bus, all the log information is centralized, merge the same type of log and record the transaction information and so on.Figure 3 shows a centralized logging system architecture diagram.

    Figure 3 centralized logging system architecture diagram

    As shown in figure, centralized logging system can be designed as a client/server architecture.In each deployment application server, all have a logging system client, the client application in some key point call log system, through the log bus, the client will log information submitted to the backend server log system.Listing 5 gives the call logging system client sample code.

    1public void someMethodWithDistributedLogger() {

    2 // Log transactionstarted once log client was created

    3 Logger logger = CentralizedLoggerFactory. createLogger ();

    4 logger.setData(key, value);

    5try {

    6throw new Exception(Some exception);

    7 } catch (Exception e) {

    8 logger.setStatus(Status.FAIL,

    9new LogRecord(e.getMessage()));


    11 logger.log(Level.INFO, new LogRecord(exit method));

    12 logger.setStatus(Status.SUCCESS);

    13 // Log transaction ended


    List five centralized logging system client calling code

    When the factory method to create log system client, log transactionstarted, at the end of a method call, the end of the life cycle of the logging client instance, log transaction is over.A detail of the example code is, once the state is set (line 8-9), the subsequent state will not be back (line 12).

    Can also according to need, the centralized logging system design for the structure of the peer to peer.Logging system backend database can need according to the data aggregation, data mining, etc.

    Path tracking system

    The system used to track is a network system in production operation running path.It can include the record module and function invocation path, a user identity information, analysis the corresponding user data flow, etc.

    The realization of the path tracking system can refer to the third part of the local log system implementation.Listing 6 gives a use purposes - recording system run path:

    1public void doSomething(String value1, String value2) {

    2 tracer.traceEnterMethod( new Object[]{value1, value2});

    3 // do something

    4 tracer.tranceExitMethod( new Object[]{value1, value2});


    Listing 6 recording system running path

    Similar to the fourth part introduces the centralized logging system, the path tracking system can also increase the system bus and the back-end server support, enhance the system capability.

    Reference counting system

    Large site there will be a lot of modules, each module will consist of more smaller components.Such as the user interface module may have a search box, a drop-down box article, user input, etc.The background components such as user authority certification, business logic calculation.Statistics of these components of traffic information is a good way to understand the operation of system, to provide maintenance, performance tuning is very good help.

    There are two categories of reference information is valuable.Category known as static reference count, measure is a component in the building, the number of reference by other components.Another kind is called dynamic invocation count, measures the running process is called the number of times.

    Static reference number

    Calculation of static reference count is a quite difficult work, need in each reference to the code of the component, incrementing the reference count.If the users of the job to a component, such as missing counting, error happens, has increased the burden of the user.

    There are many methods to solve.Such as the use of AOP technology in place to increase the count for each reference component.In this paper, using the model builder building blocks.Each component inside the builder was build, will be called a counting method.

    1public static Component buildComponent(Component[] components) {

    2for ( int i = 0; i < components.length; i++) {

    3 addComponent(components[i]);

    4 com.corp.countStaticReference(

    5 components[i].getClass().getName());



    Listing 6 static reference counting in the Builder

    The dynamic invocation count

    Records the module number of dynamic invocation, relatively simple.Every is called module in general have a gateway function or a master control function.Record component dynamic invocation count, as long as in the function to add the corresponding count operations.

    1public void handleBody() {

    2 com.corp. countDynamicReference ( this .getClass().getName());

    3 // handleBody


    Listing 7 add count operations in main control function module

    Every time to enter this function, static countDynamicReference counting function is called, the function using the module name of the class as a key value, and the corresponding reference count will be increasing.

    In order to show the value of the static and dynamic invocation count can design an admin interface, display module count.

    With above all levels of the logging system, a network application inside the basic can take in everything in a glance.Different applications can also according to their different needs, customize different logging system shows the internal behavior of the system.

    Author introduction: Shen Yu is ebay software engineer, software engineering companies in China engaged in JSP application, the JSP engine research and development, enterprise network application.

Report this document

For any questions or suggestions please email