TUL6

Analysis for the Asian Telecommunications Provider

     Table of Contents

    Executive Summary ………………………………………………………………………3

    Parameters used in SAS EM ………………………………………………………………5

    Reviewing and Sampling …………………………………………………………..…..5

    Data Partitioning, Variable Selection, Replacement, and Transformation ………………5

    Modeling ………………………………………………………………………………6-10

    Conclusion ………………………………………………………………………………11

    Diagram of Data Flow……………………………………………………………………....12

     2

     Executive Summary

    Background

     An Asian telecommunication provider recently launched the first phase of a third generation (3G) mobile telecommunications network. The company now has both second generation (2G) and 3G subscribers. The company would like to use existing customer usage and demographic data to identify which 2G customers are most likely to upgrade or switch to the 3G technology. The sample dataset provided to the consulting firm has 20,000 2G network users and 4,000 3G network users with more than 200 data fields for each subscriber. Three quarters of the database (18,000 records of 15,000 2G and 3,000 3G) have target variable values available. These records should be used for training/validation/testing of models. The remaining 6,000 records presumably contain 5,000 2G and 1,000 3G customers. The target values are missing for this part of the database. The goal is to accurately predict as many 3G customers in the scoring database as possible.

Methodology

     The process follows this sequence: data review and sampling, data partitioning, variable selection, modeling, and model evaluation and selection. The model selection process used charts to determine specificity, sensitivity, and accuracy (or misclassification).

     The original database was skewed toward 2G customers at the rate of 83.4% to 16.6% for 3G customers. To properly train the predictive models a sampling of the main database was created with equal numbers of 2G and 3G customers. This allowed the models to learn both 2G and 3G customer characteristics more effectively. This database was partitioned into training, validation and testing sub-sets so that each model “checks” its learning and assumptions.

    Each customer record contained over 250 variables. Several variable selections methods were used to reduce the number of variables: manual variable selection, decision tree variable selection and SAS variable selection. After thorough evaluation of each variable selection process, we selected the SAS variable selection tool because it provided better results with the models for this dataset. The 250 variables were narrowed down to 31 for use with the models.

     Three types of models were used to predict the 3G customers: Logistic Regression (LR), Decision Trees (DT) and Artificial Neural Networks (ANN). A fourth ensemble model was also used, which is a composite or combining of the individual models. Decision Trees consistently outperformed both Logistic Regression and Artificial Neural Networks in terms of “true

    positives” prediction (or sensitivity based on ROC chart) and misclassification rates. Both two

    node and multiple node decision trees were evaluated and tested with the three node decision trees yielding the better results. The Ensemble model, based on two Decision Trees (Gini and Entropy) provided even better accuracy than any one single decision tree.

     Each model was evaluated using several criteria: the ROC chart (a measure of sensitivity and specificity), Captured Response Rate and the Lift Chart. The DT model consistently performed better than Logistic Regression and Artificial Neural Networks in terms of misclassification rate and, more importantly, sensitivity. The Ensemble model comprised of two Decision Trees performed better in all of the measurements than these models did separately.

Findings

     Consistent with the scope of the project, the Gini and Entropy Decision trees, combined in an ensemble model, more accurately predicted 3G customers within the original training, validation and testing data. ROC charts were used to evaluate sensitivity, misclassification rates were used as estimates of overall error rates, and cumulative and non-cumulative captured response charts provided general comparison of models’ performance.

     3

Conclusion

     We recommend the use of an Ensemble Model, which combines two Decision Tree models (Gini and Entropy) to correctly predict which 2G customers will be most likely to upgrade or switch to the 3G technology. The management team and the marketing department of the Asian Telecommunication Company can use the developed model to proceed with cost-benefit analysis. The final selection of a model depends on marketing budgets and management choice. The next recommended step would be project budgeting analysis based on suggested model data and using fixed, variable costs and profit per customer to find break-even point and profit/loss at different predicted 3G levels.

     4

Parameters used in the SAS EM

Seed ID: 12345

    Number of observations used: complete dataset: 18,000 records

    Number of dataset variables: 250

    Partitioning: 60% - training, 30% - validation, 10% - test

    SAS Enterprise Miner version 4.3

    Modeling methods: Chi-Square 3-Node Decision Tree, Entropy Reduction 3-Node Decision Tree, Gini Reduction 3-Node Decision Tree, Logistic Regression, Multi-Layer Perceptron (MLP),

    Artificial Neural Network (ANN), Ensemble Model

Reviewing and Sampling the Dataset

    The first order of business the group considered was the distribution of the target variable: Customer_Type (2G or 3G). The distribution of 2G customer types compared to 3G customer types was 15,000 records to 3,000 records in the training/testing dataset. The group further reduced the number of records used by creating a balanced dataset consisting of 3000 2G customers and 3000 3G customers. If this procedure was not completed, the dataset would have been biased towards predicting everybody as 2G customers.

Data Partitioning, Variable Selection, Replacement, and Transformation

     The group’s next step was to split the sample into 3 datasets: Training, Validation, and Testing. The training dataset consisted of 60% of the data, validation dataset consisted of 30% of the data, and the testing dataset consisted of the final 10% of the data. The group felt this split would reduce the amount of overtraining in the dataset, and would provide a general idea of how well the models we were creating were performing.

    The group tried two methods of reducing the 251 variables which were included in the datasets being used to create the models. Numerous methods were employed in reducing the amount of information provided to the group: Manual Variable Selection, Decision Tree Variable Selection, and the SAS Variable Selection tool. The SAS variable selection tool was used to determine the variables used in the model because it provided the best results in the initial models created to study the dataset, and provided the user with a workable number of variables. Many of the variables were eliminated because they had a low correlation with the Target Variable, Customer_Type. The final set of variables used in the modeling is shown in Table One.

    Table One: Variable Selection

    Variable Variable

    AGE AVG_CALL_MOB

    MARITAL_STATUS AVG_CALL_INTRAN

    CUSTOMER_CLASS AVG_CALL_T1

    LINE_TENURE AVG_VAS_GAMES

    HS_AGE AVG_VAS_SR

    HS_MODEL AVG_PK_CALL_RATIO

    LUCKY_NO_FLAG AVG_OP_CALL_RATIO

    LST_RETENTION_CAMP AVG_M2M_MINS_RATIO

    LOYALTY_POINTS_USAGE STD_BILL_SMS

    BLACK_LIST_FLAG STD_T1_CALL_CON

    VAS_CND_FLAG G_PAY_METD

    VAS_CNND_FLAG G_PAY_METD_PREV

    VAS_AR_FLAG G_TOP1_INT_CD

    AVG_DIS_1900 G_TOP2_INT_CD

     5

    AVG_USAGE_DAYS G_TOP3_INT_CD

    AVG_BILL_VOICED

     Replacement was not performed on the dataset, most of the variables were missing less than 5% of their values, and the only significant variable which was missing over 50% of its values, was excluded in the variable selection process.

     Transformation was performed on the remaining variables in the early stages of modeling. It was determined through testing many of the binning and normalization techniques did not yield improved results when modeled. Therefore, it was decided by the group to forgo the transformation process on final set of variables.

Modeling

     Three basic models were run: Logistic Regression, Artificial Neural Network, and Decision Trees. The Logistic Regression and ANN models consistently performed worse than the three decision tree models even as measures were taken to try to improve their misclassification rates (predictive ability). Various parameters were changed in order to improve the accuracy of these models, for example, the number of hidden nodes in the ANN models. Random seed numbers were also changed to see the effect of this variable in our results and to make sure no local minimums/maximums were observed in the ANN model.

     One of the analyses we did amongst the early testing of decision trees was to determine if the maximum number of branches to a node affected our results substantially. From our testing we were able to ascertain the maximum of three branches per node yielded the best results. Once our group decided to pursue the maximum three branch route, we discovered the various Decision Trees yielded similar results, and we were able to improve their predictive rates when various parameters were manipulated. Through testing, the parameters used in the final model (an Ensemble Model of the Entropy Reduction and Gini Reduction models) were completed. The parameters used in the decision trees are shown below in Table Two.

Table Two: Decision Tree Criteria

     Table Three shows a comparison of the six main models and how they predicted which 2G customers who would switch to 3G. The prediction across training, validation, testing data is fairly stable indicating there was little overfitting of the models. The Decision Trees consistently performed five to ten percent better in predicting the 3G customers than the Logistic Regression and ANN models. The four models utilizing Decision Trees were consistently around the twenty percent misclassification rate and they performed better in the ROC chart (specificity and sensitivity). When creating the Ensemble model, the individual models were grouped together in

     6

    various combinations to determine if any benefit was achieved through their coupling or staging. Based on these observations, our group used the combination of two Decision Tree models: Entropy Reduction and Gini Reduction to improve the results. Consequentially, we found that the combination of these two models provided a better overall result than each of the models created individually in terms of high accuracy, sensitivity, specificity, and misclassification rate across all three datasets.

Table Three: Model Comparison

     In the case of our group, we used multiple criteria to help us formulate which the “best model”. The ROC chart (shown in Table Four below) was the basis for determining which of the models performed best.

Table Four: ROC Chart

     When viewing this chart, it is important to remember the greater the slope of the curve when going from left to right, the more sensitive the model. In our case, the individual Decision Tree models, and the Ensemble models clearly outperform the Regression and Neural Network

     7

    models. If judging on this criterion solely, we would choose the Ensemble model as the best performer.

     Another criterion we compared our models against was the Capture Response Rate shown in Table Five. Once again the different Decision Tree models and Ensemble models clearly outperformed the Regression and ANN models. The larger the space between the model and the baseline, the greater the Captured Response Rate indicates the model(s) are performing better than average. For the Asian Telecommunications company, the goal is to correctly predict the 3G consumers in the Scoring dataset. Therefore, this graph must be used with great caution, with the reminder this is for the classification of 2G and 3G, not 3G solely.

Table Five: % Captured Response

     A third criterion to consider for selecting the correct model is to look at the Lift Chart as shown in Table Six.

     8

Table 6: Lift Chart Values

     When viewing this chart, look for the point at which the individual Model Lift Value lines cross the threshold of 1. At this point, the user can not obtain any significant lift in the model. In the case of these models, the Regression and ANN model cross the baseline at a greater percentile than the Decision Tree and Ensemble models. It is at the client’s discretion as

    to whether this can be considered important. If the client only targets the top 30 percentile of its consumer, then the models need to be judged at this percentile mark. In our case, we took this into consideration, but felt this criterion would not greatly influence our decision of which of the models to use. Using this criterion as the sole determinant for a successful model would end in disappointment. When interpreted in conjunction with the Captured Response chart in Table

     9

    Five it becomes clears the Decision Trees are more effective at predicting whether the customer will become a 3G user.

    As you can see in the classification matrix in the above Table, the Ensemble model was quite successful at predicting which category a customer would fall into. It excelled at predicting the 2G customers, and did quite well at classifying the 3G customers. The Ensemble was a relatively low rate of predicting 2G customers who actually became 3G customers.

Table 7: Classification Matrix

     10