Performance of Some Yang and Chang estimators in Logistic Regression Model.

In this paper we were able to recommend some LYC estimators in estimating the biasing parameter k, in which both Monte Carlo study and real life data was used to investigate the performance of these estimators. The MSE criterion was used in evaluating the performances of the estimators to know the best among them. In the simulation study it was seen that at small samples sizes (20, 30 and 40) and number of explanatory variable say, p= 2 and 3 the biasing parameter k with the Harmonic version has the lowest MSE at almost all degree of correlation. Likewise for sample size (200, 250 and 300) with number of explanatory variable say, p= 5 and 6 and the degree of correlation is low the biasing parameter k with the median version has the lowest and when ρ is high biasing parameter k with the Arithmetic mean and maximum version is the best option to use. Also when sample size (800, 900 and 1000) with number of explanatory variable say, p= 9 and 10 and the degree of correlation is low the biasing parameter k with the median version has the lowest and when ρ is high biasing parameter k with the Arithmetic mean is the one with lowest MSE. In addition, from the numerical example the biasing parameter k with the arithmetic mean and the median version has the two lowest MSE respectively.

Recommendations

Hence, based from our findings both in simulation and numerical examples we thereby recommend to researchers and scholars, when having the issue of multicollinearity in logistic model, let the LYC (biasing k with the version of Arithmetic mean, Harmonic mean and the Median version) estimator be used.

Acknowledgment

The authors are very grateful to Federal University of Technology, Akure and Ladoke Akintola University of Technology for their support, love and tutoring which also helped to improve this work's quality.

Conflict of interest

The author has no conflict of interest.

Introduction

    In multiple regression models, multicollinearity occurs when the explanatory variables are collinear, which was first introduced by Frisch [1]. This is a common occurrence in applied research. Using ordinary least squares (OLS) or maximum likelihood (ML) to estimate linear regression models and logit regression models leads to high variance and unstable parameter estimates. As a result, multicollinearity may lead to questions about the validity of the regression analysis conclusions. In the past few decades, ridge regression (RR) has been widely used as a corrective measure for linear regressions, with a lot of research focusing on estimating the ridge parameter k. Hoerl and Kennard [2,3]       initially proposed ridge regression. Several studies have been conducted on the subject, including Gibbons [4], Lawless and Wang [5], Dempster et al. [6], Hoerl and Kennard [2], Hoerl et al. [7], McDonald and Galarneau [8], Alkhamisi et al. [9], Alkhamisi and Shukur [10], Muniz and Kibria [11], Muniz et al. [12], and Månsson et al. [13], Lukman and Ayinde [14], Ayinde et al [15]  and others too. However, much attention has not been given into the logit model and those that worked on it are researchers like Schaeffer et al. [16], Schaeffer [17], Månsson and Shukur [18], Kibria et al [19], and few others. These researchers only focused on Ridge regression and no or little attention is given to other biasing parameter k emanating from other estimators too.

                The main focus of this paper is to propose some Logistic Yang and Chang (LYC) estimators based on the work of Kibra [20] and Kibra et al [19]. Since it is anticipated that these estimators will have lower mean squared error (MSE) than that of the logistic ridge regression (LRR) and ML. The MSE is computed in order to assess the estimators' performance.

                The work is structured as follows: we provide a description of the materials and statistical methods in Section. 2. Section 3 discusses the simulation and numerical findings. Section 4 provides a succinct overview and conclusions.

Materials and Methodology

On the basis of the research of Kibria [20] and Kibra et al. [19], we suggest a few LYC estimators in this section for estimating the biasing parameter k.

Logit Regression

Logit regression has always been one of the commonest method in statistic that is applied whenever the ith value of our so called dependent variable (y) of the regression model follows a Be (πi ) distribution with the following parameter value:

                                                                                                                                    (1)

Where  is the ith row of X, and is a data matrix, having p explanatory variables and such that is a  vector of coefficients. Using the Maximum Likelihood technique, which maximizes the following log likelihood, is one of the most used ways to estimate and can be expressed as:

L=                                                                                           (2)

This can be achieved by setting the first derivative of the above expression to be equal to zero. Hence, the ML estimates are found by solving the subsequent equation:

Setting the first derivative of the aforementioned equation (2) to zero will do this. Therefore, the following equation below must be solved in order to find the ML estimates:

                                                                                                                         (3)

The iterative weighted least square (IWLS) algorithm is used:

                                                                                                                        (4)

Where the following are the expression of  and  respectively

 and  is known to be a vector where the ith element equals

Since equation (3) is nonlinear in , we can express the MSE of the ML estimator as:

                                                   (5)

  is said to be the jth eigenvalues of the  matrix.

Logistic Ridge Regression Estimator

 Schaeffer et al. [16], proposed LRR estimator, as a substitute to the ML estimate that mitigates the problem of multicollinearity, Instead of directly estimating the coefficients of the regression model, the LRR estimator focuses on estimating the inverse of the covariance matrix. By doing so, the LRR estimator effectively reduces the impact of small eigenvalues caused by multicollinearity, resulting in a more reliable and robust estimation of the regression coefficients. The LRR estimator is defined as:

                                                                                                    (6)

k is the biasing parameter,  and is the  estimates derived by equation(4). The LRR estimator MSE can be expressed as:

                          (7)

Logistic Yang and Chang Estimator

The Logistic Yang and Chang (LYC) estimator, which is a special estimator of Liu and ridge estimator combined was proposed by Awwad et al [21] and it also handle the problem of multicollinearity effectively too. The estimator LYC is defined as:

                                              (8)

k and d is the biasing parameters,  and is the  estimates derived by equation(4). The LYC estimator MSE can be expressed as:

                                                                                                                                                                                 (9)

where is expressed as the jth element of and is known to be the eigenvector expressed as , where .

The Proposed Estimators.

A ridge parameter can be chosen in many different ways, however, a number of approaches have been put out for the linear RR model, and these have been extended to the logistic ridge regression model. In the classical RR a biasing parameter k from the works of Hoerl and Kennard [2,3] is as expressed as follows:

                                                                                                                                              (10)

The above biasing parameter was also adopted to have the following biasing parameter proposed by Kibria [20] as stated below

                                                                                                                                   (11)

Later on, both equation (10 & 11) was adopted into the LRR by Schaeffer et al. [16 and Kibra et al [19] respectively.

However the biasing parameter k for LYC can be gotten from the MSE

                             (12)

Given that d is fixed, an ideal value of k is the value that will be to minimize .

Then, by differentiating  w.r.t. k and equating to 0, we have k as below:

                                                                                                                  (13)

However, k depends on the unknown . For practical purposes, it will be replaced by its unbiased estimator . Hence, this will be obtained as:

                                                                                                                  (14)

As an operational estimator for k. Furthermore, when d = 1, the above equation returns back to the k proposed by Schaeffer et al. [17] which is expressed as:

Following the works of Schaeffer et al. [16], Kibra [20] and Kibra et al [19] the following biasing parameter k for Yang and Chang are proposed as:

                                                                               (15)

                                                                                  (16)

                                                                                     (17)

                                                                                          (18)

                                                                                           (19)

                                                                                                                        (20)

Furthermore, the optimal value for d can also be derived by minimizing  in equation (12)

Then, by differentiating  w.r.t. d and equating to 0, we have d as below:

                                                                                                 (21)

Noting that when k =0 the above equation leads to the Liu biasing estimator, the above estimator would be

                                                                                                                          (22)

The Monte Carlo Simulation

This paper's major goal is to determine how multicollinearity impacts ML, LRR and LYC Estimators. Therefore, the most important variable in the experiment is the degree to which the regressors were correlated. Hence, we generate the explanatory variables by using the following formula, which lets us adjust the correlation's strength:

    i=1, 2,…, n, j=1,2,…,p                                                          (23)

The degree of correlation between the explanatory variables is referred to as    and is also the pseudo random numbers from the standard normal distribution. The four various levels of considered are said to be 0.8, 0.9, 0.95 and 0.99. Likewise the dependent variable is also derived from the   distribution where

                                                                                                                                  (24)

We set , according to Newhouse and Oman [22] statement that if our MSE is a function of , and k if all the  explanatory variables used are fixed, we can then say  that the MSE is minimized when this coefficient are choose. Models consisting p=2 and 3 explanatory variables, sample sizes n=20, 30 and 40 are used, models, p=5 and 6 explanatory variables, sample sizes n=200,250 and 300 and models consists of p as 9 and 10 explanatory variables, sample sizes n=800,900 and 1000 are used. With this experimental design we will be able to examine which of the Yang and Chang biasing k parameters that will give an optimal result for the designs. From the works of Kibria [20],  Lukman et al [23,24], Oladapo et al [25], Muniz and Kibria [11], Idowu et al [26], Owolabi et al [27,28], Månsson and Shukur [18],  among others we can get more detailed information on simulation procedures.

 Results and Discussion based on Simulation

Findings from Monte Carlo investigation are all presented. The MSE values of all the estimators for Monte Carlo study are shown in Table 1- Table 3 and that of real life is in Table 4. Also, the outcome of varying the different factors we used in this study on the ML, LRR and LYC estimators are discussed too.

Table 1 Estimated MSE for different estimator when p=2 and 4

Bold values show the smallest MSE

In Table 1 is the estimated MSE values when n is 20, 30 and 40 for explanatory variables, p=2 and 3 it can be observed that the LYC with the biasing parameter k of the Harmonic version gives the lowest MSE in all cases except when n is 40, p=2 and ρ= 0.8 likewise for when n is 40, p=3 and ρ= 0.8.

Table 2: Estimated MSE for different estimator when p=5 and 6

Bold values show the smallest MSE

In Table 2 is the estimated MSE values when n is 200, 250 and 300 for explanatory variables, p=5 and 6 it can be observed that the LYC with the biasing parameter k of the median version gives the lowest MSE when the multicollinearity level ρ= 0.8 and 0.9, likewise for when ρ= 0.95 its seen that the LYC with the biasing parameter k of the arithmetic mean version gives the lowest MSE and when ρ= 0.99 its seen that the LYC with the biasing parameter k of the Maximum version gives the lowest MSE.

Table3: Estimated MSE for different estimator when p=9 and 10

Bold values show the smallest MSE

In Table 3 is the MSE values for n at 800, 900 and 1000 for explanatory variables, p=9 and 10 it can be observed that the LYC with the biasing parameter k of the median version gives the lowest MSE when the multicollinearity level ρ= 0.8 and 0.9, likewise for when ρ= 0.95 its seen that the LYC with the biasing parameter k of the arithmetic mean version gives the lowest MSE for both sample sizes 800 and 900. But when n =1000 its seen that the LYC with the biasing parameter k of the Median version gives the lowest MSE.

Real Life Data

Pena et al [29] used a logistic model to investigate the effects of temperature, pH, and soluble solids content on the response of Alicyclobacillus growth likelihood in apple juice. The matrix's eigenvalues are 13464.7990, 1715.9257, 56.5515, and 3.5445. As a result, the condition index (C.I) is 61.6342, indicating that multicollinearity exists in the model. Table 4 shows the estimated regression coefficient values from each estimator, as well as the accompanying mean squared error.

When there is multicollinearity, the ML estimator performs the least well, as expected. The selection of k and d (as shrinkage parameters) determines the efficiency of biased estimators. All of the proposed estimators performed admirably, and one of them has the minimum mean square error, which corresponds to the simulation outcome.

Table 4: Regression coefficients and MSE