References

ACPD

Atmospheric Chemistry and Physics Discussions

ACPD

1680-7375

Copernicus GmbH

Göttingen, Germany

10.5194/acpd-8-6409-2008

Technical Note: Review of methods for linear least-squares fitting of data and application to atmospheric chemistry problems

Cantrell

C. A.

National Center for Atmospheric Research, Atmospheric Chemistry Division, 1850 Table Mesa Drive, Boulder, CO 80305, USA

01 04 2008

8 2 6409 6436

This is an open-access article ditributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

This article is available from http://www.atmos-chem-phys-discuss.net/8/6409/2008/acpd-8-6409-2008.html

The full text article is available as a PDF file from http://www.atmos-chem-phys-discuss.net/8/6409/2008/acpd-8-6409-2008.pdf

The representation of data, whether geophysical observations, numerical model output or laboratory results, by a best fit straight line is a routine practice in the geosciences and other fields. While the literature is full of detailed analyses of procedures for fitting straight lines to values with uncertainties, a surprising number of scientists blindly use the standard least squares method, such as found on calculators and in spreadsheet programs, that assumes no uncertainties in the x values. Here, the available procedures for estimating the best fit straight line to data, including those applicable to situations for uncertainties present in both the x and y variables, are reviewed. Representative methods that are presented in the literature for bivariate weighted fits are compared using several sample data sets, and guidance is presented as to when the somewhat more involved iterative methods are required, or when the standard least-squares procedure would be expected to be satisfactory. A spreadsheet-based template is made available that employs one method for bivariate fitting.

References 1

Barker, D. R. and Diana, L. M.: Simple method for fitting data when both variables have uncertainties, Am. J. Phys., 42, 224–227, 1974.

Bevington, P. R.: Data reduction error analysis for the physical sciences, McGraw-Hill Book Company, New York, 1969.

Borcherds, P. H. and Sheth, C. V.: Least squares fitting of a straight line to a set of data points, Eur. J. Phys., 16(5), 204–210, 1995.

Bruzzone H., and Moreno, C.: When errors in both coordinates make a difference in the fitting of straight lines by least squares, Meas. Sci. Technol., 9, 2007–2011, 1998.

Cantrell, C. A., Edwards, G. D., Stephens, S., Mauldin, R. L., Zondlo, M. A., Kosciuch, E., Eisele, F. L., Shetter, R. E., Lefer, B. L., Hall, S., Flocke, F., Weinheimer, A., Fried, A., Apel, E., Kondo, Y., Blake, D. R., Blake, N. J., Simpson, I. J., Bandy, A. R., Thornton, D. C., Heikes, B. G., Singh, H. B., Brune, W. H., Harder, H., Martinez, M., Jacob, D. J., Avery, M. A., Barrick, J. D., Sachse, G. W., Olson, J. R., Crawford, J. H., and Clark, A. D.: Peroxy radical behavior during the Transport and Chemical Evolution over the Pacific (TRACE-P) campaign as measured aboard the NASA P-3B aircraft, J. Geophys. Res., 108(D20), 8797, doi:10.1029/2003JD003674, 2003.

Cecchi, G. C.: Error analysis of the parameters of a least-squares determined curve when both variables have uncertainties, Meas. Sci. Technol., 2, 1127–1128, 1991.

Chong, D. P.: Comment on "Linear least-squares fits with errors in both coordinates" by Reed, B. C.,(Am. J. Phys., 57, 642–646, 1989), Am. J. Phys., 59(5), 472–474, 1991.

Chong, D. P.: On the use of least squares to fit data in linear form, J. Chem. Ed., 71(6), 489–490, 1994.

Christian, S. D. and Tucker, E. E.: Analysis with the microcomputer. 5. General least-squares with variable weighting, Am. Lab., 16(2), p 18, 1984.

Christian, S. D., Tucker, E. E., and Enwall, E.: Least-squares analysis: A primer, Am. Lab., 18, 41–49, 1986.

Crawford, J., Davis, D., Olson, J., Chen, G., Liu, S., Gregory, G., Barrick, J., Sachse, G., Sandholm, S., Heikes, B., Singh, H., and Blake, D.: Assessment of upper tropospheric HOx sources over the tropical Pacific based on NASA GTE/PEM data: Net effect on HOx and other photochemical parameters, J. Geophys. Res., 104(D13), 16 255–16 273, 1999.

Faloona, I., Tan, D., Brune, W. H., Jaeglé, L., Jacob, D. J., Kondo, Y., Koike, M., Chatfield, R., Pueschel, R., Ferry, G., Sachse, G., Vay, S., Anderson, B., Hannon, J., and Fuelberg, H.: Observations of HOx and its relationship with NOx in the upper troposphere during SONEX, J. Geophys. Res., 105(D3), 3771–3783, 2000.

Fried, A., Crawford, J., Olson, J., Walega, J., Potter, W., Wert, B., Jordan, C., Anderson, B., Shetter, R., Lefer, B., Blake, D., Blake, N., Meinardi, S., Heikes, B., O'Sullivan, D., Snow, J., Fuelberg, H., Kiley, C. M., Sandholm, S., Tan, D., Sachse, G., Singh, H., Faloona, I., Harward, C. N., and Carmichael, G. R.: Airborne tunable diode laser measurements of formaldehyde during TRACE-P: Distributions and box model comparisons, J. Geophys. Res., 108(D20), 8798, doi:10.1029/2003JD003451, 2003.

Gonzalez, A. G., Marquez, A., and Fernandez Sanz, J.: An iterative algorithm for consistent and unbiased estimation of linear regression parameters when there are errors in both the $x$ and $y$ variables, Computers Chem., 16(1), 25–27, 1992.

Irwin, J. A. and Quickenden, T. I.: Linear least squares treatement when there are errors in both $x$ and $y$, J. Chem. Ed., 60(9), 711–712, 1983.

Jones, T. A.: Fitting straight lines when both variables are subject to error, I. Maximum likelihood and least-squares estimation, Math. Geo., 11(1), 1–25, 1979.

Kalantar, A. H.: Weighted least squares evaluation of slope from data having errors in both axes, Trends in Anal. Chem., 9(5), 149–151, 1990.

Kalantar, A. H., Kerrich's method for $y$=$\alphax$ data when both $y$ and $x$ are uncertain, J. Chem Ed., 68(5), 368–370, 1991.

Kalantar, A. H.: Straight-line parameters' errors propagated from the errors in both coordinates, Meas. Sci. Technol., 3, p 1113, 1992.

Kalantar, A. H., Gelb, R. I., and Alper, J. S.: Biases in summary statistics of slopes and intercepts in linear regression with errors in both variables, Talanta, 42(4), 597–603, 1995.

Krane, K. S. and Schecter, L.: Regression line analysis, Am. J. Phys., 50(1), 82–84, 1982.

Leduc, D. J.: A comparative analysis of the reduced major axis technique of fitting lines to bivariate data, Can. J. For. Res., 17, 654–659, 1987.

Lybanon, M.: A better least-squares method when both variables have uncertainties, Am. J. Phys., 52(1), 22–26, 1984a.

Lybanon, M.: Comment on "Least squares when both variables have uncertainties", Am. J. Phys., 52(3), 276–278, 1984b.

Lybanon, M.: A simple generalized least-squares algorithm, Comp. Geosci., 11, 501–508, 1985.

Macdonald, J. R. and W. J. Thompson, Least-squares fitting when both variables contain errors: Pitfalls and possibilities, Am. J. Phys., 60(1), 66–73, 1992.

MacTaggart, D. L. and Farwell, S. O.: Analytical use of linear regression. Part II: Statistical error in both variables, J. AOAC Intl., 75(4), 608–614, 1992.

Moreno, C. and Bruzzone, H.: Parameters' variances of a least-squares determined straight line with errors in both coordinates, Meas. Sci. Technol., 4, 635–636, 1993.

Moreno, C.: A least-squares-based method for determining the ratio between two measured quantities, Meas. Sci. Technol., 7, 137–141, 1996.

Neri, F., Saitta, G., and Chiofalo, S.: An accurate and straightforward approach to line regression analysis of error-affected experimental data, J. Phys. E: Sci. Inst., 22, 215–217, 1989.

Neri, F., Patanè, S., and Saitta, G.: Error-affected experimental data analysis: application to fitting procedure, Meas. Sci. Technol., 1, 1007–1010, 1990.

Olson, J. R., Crawford, J. H., Chen, G., Fried, A., Evans, M. J., Jordan, C. E., Sandholm, S. T., Davis, D. D., Anderson, B. E., Avery, M. A., Barrick, J. D., Black, D. R., Brune, W. H., Eisele, F. L., Flocke, F., Harder, H., Jacob, D. J., Kondo, Y., Lefer, B. L., Martinez, M., Mauldin, R. L., Sachse, G. W., Shetter, R. E., Singh, H. B., Talbot, R. W., and Tan, D.: Testing fast photochemical theory during TRACE-P based on measurements of OH, HO2, and CH2O, J. Geophys. Res., 109, D15S10, doi:10.1029/2003JD004278, 2004.

Olson, J. R., Crawford, J. H., Chen, G., Brune, W. H., Faloona, I. C., Tan, D., Harder, H., and Martinez, M.: A reevaluation of airborne HOx observations from NASA field campaigns, J. Geophys. Res., 111, D10301, doi:10.1029/2005/JD006617, 2006.

Orear, J.: Least squares when both variables have uncertainties, Am. J. Phys., 50, 912–916, 1982; Erratum, Am. J. Phys., 52, p 278, 1984.

Pasachoff, J. M.: Applicability of least-squares formula, Am. J. Phys., 48(10), p 800, 1980.

Pearson, K.: On lines and planes of closet fit to systems of points in space, Philos. Mag., 2(6), 559–572, 1901.

Reed, B. C.: Linear least-squares fits with errors in both coordinates, Am. J. Phys., 57(7), 642–646, 1989; Erratum, Am J. Phys., 58(2), p 189, 1990.

Reed, B. C.: Linear least-squares fits with errors in both coordinates. II: Comments on parameter variances, Am. J. Phys., 60(1), 59–62, 1992.

Riu, J. and Rius, F. X.: Univariate regression models with errors in both axes, J. Chemometrics, 9, 343–362, 1995.

Sheth, C. V., Ngwengwe, A., and Borcherds, P. H.: Least squares fitting of a straight line to a set of data points: II: Parameter variances, Eur. J. Phys., 17(6), 322–326, 1996.

Squire, P. T.: Comment on "Linear least-squares fits with errors in both coordinates" by B. C. Reed [Am. J. Phys. 57, 642–646 (1989)], Am. J. Phys., 58(12), p 1209, 1990.

Williamson, J. H.: Least-squares fitting of a straight line, Can. J. Phys., 46, 1845–1847, 1968.

York, D.: Least-squares fitting of a straight line, Can. J. Phys., 44, 1079–1086, 1966.

York, D.: Least squares fitting of a straight line with correlated errors, Earth Planet. Sci. Lett., 5, 320–324, 1969.

York, D., Evensen, N. M., López Martinez, M., and De Basabe Delgado, J.: Unified equations for the slope, intercept, and standard errors of the best straight line, Am J. Phys., 72(3), 367–375, 2004.