I. Introduction

1. There is a consensus as to which price index number formula is best when price and quantity/value information are available for the aggregation of heterogeneous items. The economic theoretic approach to index number formulas supports superlative index numbers, primarily the Fisher, Törnqvist, and Walsh indexes, all of which give similar answers. The axiomatic approach supports the Fisher index. Such findings are part of the internationally-accepted manuals on consumer, producer, and (forthcoming)² trade price indexes—ILO et al. (2004a and 2004b). What is less well known is that for the aggregation of homogeneous items, the unit value index is the best formula and superlative price index numbers are biased, and for the aggregation of heterogeneous items, superlative price index numbers are best and the unit value index is biased. This paper provides a formal decomposition of the difference between a unit value index and the Fisher, and other, price indexes, and addresses the question as to what is the appropriate formula for differentiated, broadly-comparable products that lie in the continuum between homogeneous and heterogeneous products.

2. The bias in superlative index numbers for homogeneous goods (or services) is a neglected and important index number issue. Say, for example, the price of good A was 10 in both the reference and current period and the price of good B was 12 in both periods, but there was a shift in quantities from say 6, for both A and B in the reference period, to 8 for A and 4 for B in the current period. A superlative, or any other index number formula for heterogeneous goods, would give an answer of unity, no overall price change. However, the correct answer for homogeneous goods would be a unit value fall of 3 per cent appropriately reflecting the shift in the quantity basket in the current period from the higher price level of 12 for A to the lower price level of 10 for B. The good, and cost of living with regard to this good, is, on average, now cheaper. The CPI Manual (ILO et al. 2004a, Chapter 20) and 2008 System of National Accounts (SNA) advocate the use of unit value indexes for homogeneous goods and services:

“When there is price variation for the same quality of good or service, the price relatives used for index number calculation should be defined as the ratio of the weighted average price of that good or service in the two periods, the weights being the relative quantities sold at each price. Suppose, for example, that a certain quantity of a particular good or service is sold at a lower price to a particular category of purchaser without any difference whatsoever in the nature of the good or service offered, location, timing or conditions of sale, or other factors. A subsequent decrease in the proportion sold at the lower price raises the average price paid by purchasers for quantities of a good or service whose quality is the same and remains unchanged, by assumption. It also raises the average price received by the seller without any change in quality. This must be recorded as a price and not a volume increase.” Commission of the European Communities and others, (2008), 2008 SNA, paragraph 15.69.

3. Index number theory recognizes that the appropriateness of each formula depends on whether the items aggregated are homogeneous or otherwise—Diewert (1995) and (Balk, 1998 and 2005). As matters stand the advice is to simply determine whether or not items are homogeneous and apply the appropriate formula. But what if the goods and services are broadly comparable, that is they are of different qualities such that some of the price dispersion is due to product differentiation and some due to say search costs or price discrimination? A superlative index would ignore any shift to higher or lower average (quality-adjusted) price levels, but a unit value index would wrongly treat changes in compositional mix of items of different quality as price changes, the familiar unit value bias. Given that these formulas will generally give quite different answers it is important to determine why they differ, the conditions under which each is suitable, and what to do when, as is likely to be the case, neither is.

4. The rationale for unit value and superlative indexes are outlined in Section II. Section III provides a formal analysis of how unit value and Fisher price indexes differ. In Section IV a solution is proposed to the problem of aggregating broadly comparable goods. An application using scanner data is provided in Section V with conclusions in VI.

5. The application of the results of this paper is in the determination of price and volume measures at the national and micro level for economic aggregates. It applies to consumer, commodity, producer (input and output), import, and export price indexes, as well as price indexes of capital goods, such as house price indexes. Since price indexes are used as deflators there is a concomitant application to volume indexes. The concern is with aggregation where price and quantity/value information is available for broadly comparable items, for example, for measuring the aggregate price and volume change of different qualities of automobiles, but not over automobiles and beef.

II. Superlative and Unit Value Indexes

III. The Difference Between a Unit Value and a Fisher Index

17. Párniczky (1974) and Balk (1998) respectively compare unit value indexes to the Paasche and Fisher price indexes. These seminal decompositions, while useful, undertook the decomposition in terms of quantity-weighted covariances of changes. However, quantity weighting implicitly assumes homogeneity, thus negating the analysis for this purpose and, further, the decompositions do not distinguish switches to levels. Both are issues of concern here. We provide a new decomposition.

18. We first define Laspeyres and Paasche price indexes, P_L and P_P, and a Laspeyres quantity index, Q_L, respectively as:

where $s_m^0$ were defined in equation (2) above as period 0 value shares; x_m is the m^th price relative, and y_m the m^th quantity relative, defined as: $x_m\equiv \frac{p_m^t}{p_m^0};y_m\equiv \frac{q_m^t}{q_m^0},m=1,\dots ,{M}$ items. It follows from equation (1), (3), and (5) that the ratio of a unit value index to a Fisher price index is given by:

Adopting a Bortkiewicz (1923) decomposition⁷:

where ${\rho }_{x,y}^{s_0}$ is the $s_m^0$-weighted correlation coefficient between price relatives and quantity relatives, x_m and y_m and ^cνs₀(x) and cν^s₀(y) are their $s_m^0$-weighted respective coefficients of variation, i.e. ${\sigma }_y/\overline{y}$ and ${\sigma }_x/\overline{x}$. It follows⁸

where ${\rho }_{p^0,q^t}$, and ${\rho }_{p^0,q^0}$ are the correlation coefficients between $p_m^0$ and $q_m^t$ and $p_m^0$ and $q_m^0$ respectively, cov(.) the covariances, and ${\widehat{\beta }}^t$ and ${\widehat{\beta }}^0$ the estimated OLS slope coefficients from the regressions $q_m^t=\alpha +{\beta }^t{p}_m^0+{\epsilon }_m$ and $q_m^0=\alpha \prime +{\beta }^0{p}_m^0+{\epsilon }_m^{\prime }$ respectively. Substituting (8) into (7):

The substitution effect between Fisher and Laspeyres, and Paasche and Fisher, is:

Substituting (10) and (11) into (6) yields a unit value to Fisher index ratio:

19. First, the difference between a unit value index and Fisher price index can be seen to depend on the two factors in (12): the first is the substitution effect between Fisher and Laspeyres, as given by (11), ${\left({\rho }_{x,y}^{s_0}c{\nu }^{s_0}(x)c{\nu }^{s_0}(y)+1\right)}^{\frac{1}{2}}$, also equal to the substitution bias between Paasche and Fisher indexes, since $\frac{P_F}{P_L}=\frac{P_P}{P_F}$.⁹

20. The second factor is the levels effect, that is the effect, for negatively sloping demand, of quantities shifting towards lower-priced transactions. The measure is based on the ratio of the slope coefficients from the regressions of $q_m^t$ on $p_m^0$ and of $q_m^0$ also on $p_m^0$. The two lines intercept have a common ${\overline{p}}^0$ by construction. Assume for reasons of exposition that they also have a common ${\overline{q}}^t={\overline{q}}^0$. Thus, as depicted in Figure 1, as the slope changes for the period t regression, the line $q_m^t$ rotates about the means ${\overline{q}}^t={\overline{q}}^0$ and ${\overline{p}}^0$. As the slope of the period t line say increases, above average prices have lower quantities and below average prices have higher quantities—the larger the increase, the greater the shift. The change in slopes capture a shift in levels.

View Full Size

Depiction of Levels Effect

Citation: IMF Working Papers 2009, 019; 10.5089/9781451871661.001.A001

• Download Figure
• Download figure as PowerPoint slide

21. Note this requires the two lines to intersect at the means, which they may not since it is likely that ${\overline{q}}^{t^{*}}\ne {\overline{q}}^0$, the two lines having different intercepts. However, the adjustment required to bring ${\overline{q}}^{t^{*}}$ (in this depiction down) to ${\overline{q}}^0$ at ${\overline{p}}^0$ is $\frac{{\overline{q}}^0}{{\overline{q}}^t}$. The measure in (12) includes a ratio of β^t to β⁰ with such an adjustment. Equation (12) successfully decomposes the difference between the formulas into these two effects, as we will illustrate in the empirical section.

22. Second, it is also apparent from (12) that for the unit value index to equal the Fisher price index it is necessary that: all price changes OR quantity changes are equal to each other, OR there is no (weighted) correlation between the base-period-weighted price and quantity changes AND mean quantities remain unchanged with no change to the demand slopes. These are extreme conditions. Having no dispersion in price or quantities or their changes is a negation of the index number problem, and while we do not expect the laws of economics to work perfectly, there is expected to be some relationship between price and quantity changes. There is further required in (12) an assumption of a static economy with parameter stability for demand functions and no change in average quantities purchased.

IV. What to do for Broadly Comparable Items

23. For homogenous items there is no problem: the answer is a unit value index. For heterogeneous items there is no problem: the answer is a superlative index such as a Fisher price index. However, many goods and services are broadly comparable: consumers in most product markets have a selection of differentiated items available to them, even if the differentiation is only due to the services provided by different outlets providing the same item (Hausman and Leibtag, 2008). Say there are two models, one with a feature worth an additional 5 percent and one without. Assume further that the price of the model with the feature is sold with a 10% premium and prices remain constant. We can expect a shift in quantities to the standard model and an effective drop in the average (unit value) price level. The extent of the fall would be calculated after stripping out of the prices in both periods the 5 percent quality difference. Note the answer depends on the price levels, not price changes. Index number theory might argue that the items are heterogeneous, and advise a Fisher answer of no price change. This is synonymous with treating each model as belonging to a separate stratum, that is, as being separate, heterogeneous, goods. But consumers and producers in the market that are indifferent to the two models, once the 5 percent quality-adjustment to price has been made, experience a fall in average prices that a quality-adjusted unit value index would pick up.

24. Quality adjustment factors can be applied to prices to render the comparison of prices of differentiated items akin to one of homogeneous items. We make use of (hedonic) quality-adjusted unit value indexes that remove the effects on prices of product heterogeneity, a proposal that goes back to Dálen (2001) and is formalized and empirically examined in De Haan (2004) and reiterated in De Haan (2007).¹⁰ Since a unit value index is appropriate for homogeneous items, a quality-adjusted unit value index must be appropriate for broadly comparable items. We consider first such a measure.

25. A hedonic regression (see Triplett, 2004) using data on m = 1,…,M matched models for periods τ = 0,t, of the price, $p_m^{\tau }$, on k = 1,…,K quality characteristics, $z_{km}^{\tau }$:

where $u_m^{\tau }$ are assumed to be normally distributed with mean and variance δ^τ and ${\xi }_{\tau }^2$ respectively. The heterogeneity-adjusted prices in each period relative to a reference numeraire item with mean characteristics ${\overline{z}}_{km}^{\tau }$ in each period are given by:

26. Bear in mind the models in each period are matched so that $z_{km}^{\tau }=z_{km}^0=z_{km}^t$. Note also that ${\beta }_k^0$ may or may not equal ${\beta }_k^t$ and (13) can be estimated on pooled data with a dummy variable for time and with the constraint that ${\beta }_k^t={\beta }_k^0={\beta }_k^{\tau }$, though it is preferable to estimate (13) separately for each time period without the constraint. The heterogeneity-adjusted unit value index is:

27. For goods and services with slight product differentiation we would advise a measure based on (15). Of course the quality adjustments need not use hedonic regressions. They may be much simpler due to say the addition of a single feature or option for which cost or market estimates of their value are available. Bear in mind that the items are matched in each period so there is no quality change over time. The above formula has abstracted from the measures in each period cross-sectional variation in price due to quality differences.

28. What of items that are comparable, say, models of television sets, washing machines, laptop computers, automobiles, whose price dispersion due to product differentiation is significant, as is the price dispersion due to factors that cannot be accounted for by the characteristics of the item?¹¹ There is an element in the price measure for which the quality-adjusted unit value reduces the problem to one of homogenous items, but there is also an element for which a Fisher price index given by (1) is appropriate.

29. There is a need for a weighted average of (15) and (1), but a problem as to what the weights should be. One approach is to consider what we mean by “comparable.”¹² Consider the case for, say, models of automobiles where a hedonic regression explains just about all of the price variation. The models are very different in this sense, compared to a data set, say, of the same model of automobile sold by different dealers for which a hedonic regression may explain a much smaller proportion of the price variation. A Fisher price index would be appropriate in the former case and unit value index for the latter, since quantities cannot be meaningfully added together for the former but can for the latter. Thus the weights for the heterogeneity-adjusted unit value index (15) might be the ratio of the sum of squared errors from the hedonic regression (SSE) to the total sum of squares (SST) and the weight for the Fisher price index given by (1) the ratio of the (explained) regression sum of squares (SSR) to SST. The weighted average is given by:

where ${\overline{w}}_U=\frac{SSE}{SST}$ and $(1-{\overline{w}}_U)=\frac{SSR}{SST}=R^2$. Note that the weights in (16) have a bar over them, they are an arithmetic mean of the weights from period 0 and period t for the hedonic regressions in equation (13) for binary comparisons between τ = 0,t.

30. The index given by (16) has the property that if all price variation is explained by the hedonic regression, the index is a Fisher index; if none of the price variation is explained by the hedonic regression, the index is a unit value index; as the percentage of price variation explained by the hedonic regression increases, so too will the weight given to the Fisher component.

31. The use of such weights is but one proposal. Consider the case of television sets. A hedonic regression could be estimated over all screen sizes with dummies for these screen sizes and variables for the other quality characteristics. But while the regression would attribute a say 30% premium for a 32 inch screen over a 14 inch one, while controlling for other variables, it is unlikely that consumers would consider the two models as substitutes when an allowance has been made for the screen size, and other variables. Thus the regression should be undertaken for similar goods in the sense that there is some substitutability.¹³

32. It might be argued that substitutability should be the main concept behind the weighting system. However, this is problematic. Substitutability can exist for goods and services for which quality-adjustments for unit values are not feasible and the concept of an average price not meaningful, for example, beef and chicken. However, as indicated above, the first step should be to identify a cluster of goods which are comparable and substitutable or exchangeable, for example, television sets of a similar screen size, and then use quality adjustments.

V. An Empirical Example Using Scanner Data

33. The empirical work utilizes monthly scanner data for television sets (TVs) from the bar-code readers of UK retail outlets from January 2001 to December 2001. Each observation is a model of a TV in a given month sold in one of four different outlet types: multiple chains, mass merchandisers (department stores), independents and catalogue stores. The sample was devised to only include models of TVs that were sold in all 12 months of the data. This has the advantage of replicating the matched model methodology employed by statistical offices for consumer and producer price indexes, as well as clarifying that the effects we identify are not due to new and old models of differing quality entering and leaving the market (Silver and Heravi, 2005). We further limit the sample to a narrow range of screen size, that is 10 and 14 inch TV screens, since it may be argued that larger TVs with larger screen sizes serve a different consumer need. The data set included series for 94 such models in each month accounting for sales of 0.37 million TVs worth £49 million.

34. Hedonic regressions were estimated for each month to remove the effect on price of cross-sectional product heterogeneity. The variable set for the regressions included: (i) 17 brand dummies; (ii) size of screen, 14 and 10 inch; (iii) Nicam stereo sound; (iv) on-screen text retrieval news and information panels from broadcasting companies, in order of sophistication: teletext and fastext; (v) three types of reception systems; (vi) continental monitor style; (v) flat & square and super-planar tubes; (vi) s-vhs socket; (vii) DVD playback or DVD recording; and (viii) the outlet types, multiple chains, mass merchandisers (department stores), independents and catalogue stores.

35. Table 1 and Figure 2 show Laspeyres, Paasche, Fisher price and unit value indexes. Laspeyres and Paasche generally act as upper and lower bounds on Fisher as expected from the negative value of the correlation between relative price and quantity changes, ${\rho }_{x,y}^{s_0}$, given in Table 1. The magnitude of the difference between both Fisher and Laspeyres and Paasche and Fisher price indexes is given by the substitution effect—equation (11). The magnitude of the substitution effect, as dictated by its components in equation (11), can be seen from the calculated values in Table 1 to be generally small. A notable exception is July during which the negative correlation increases in magnitude, as more price conscious consumers hit the summer sales, leading to an increase in the Laspeyres-Paasche gap. The unit value index for these differentiate products has quite dissimilar changes to the price indexes.

View Full Size

Unit Value and Price Indices for 14-inch TVs

Citation: IMF Working Papers 2009, 019; 10.5089/9781451871661.001.A001

• Download Figure
• Download figure as PowerPoint slide

Table 1.

Understanding the Differences Between Laspeyres, Paasche, and Fisher

Table 1.

Understanding the Differences Between Laspeyres, Paasche, and Fisher

	Laspeyres	Paasche	Fisher	Unit value	Substitution effect	${\rho }_{x,y}^{s_0}$	cv(x)	cv(y)
January	1.000	1.000	1.000	1.000
February	0.998	0.998	0.998	1.060	1.000	0.002	0.150	0.695
March	0.996	1.001	0.999	1.049	1.003	0.040	0.150	0.898
April	0.996	0.982	0.989	0.968	0.993	-0.037	0.153	2.562
May	0.999	0.983	0.991	0.985	0.992	-0.043	0.160	2.452
June	0.964	0.939	0.951	0.957	0.987	-0.075	0.179	1.908
July	0.959	0.922	0.940	0.994	0.980	-0.218	0.163	1.088
August	0.945	0.917	0.931	0.990	0.985	-0.142	0.170	1.271
September	0.941	0.961	0.928	0.972	0.987	-0.093	0.171	1.686
October	0.950	0.935	0.942	0.981	0.992	-0.046	0.187	1.863
November	0.948	0.925	0.936	1.000	0.987	-0.071	0.188	1.882
December	0.933	0.925	0.929	0.962	0.996	-0.017	0.197	2.667

View Table

Table 1.

Understanding the Differences Between Laspeyres, Paasche, and Fisher

	Laspeyres	Paasche	Fisher	Unit value	Substitution effect	${\rho }_{x,y}^{s_0}$	cv(x)	cv(y)
January	1.000	1.000	1.000	1.000
February	0.998	0.998	0.998	1.060	1.000	0.002	0.150	0.695
March	0.996	1.001	0.999	1.049	1.003	0.040	0.150	0.898
April	0.996	0.982	0.989	0.968	0.993	-0.037	0.153	2.562
May	0.999	0.983	0.991	0.985	0.992	-0.043	0.160	2.452
June	0.964	0.939	0.951	0.957	0.987	-0.075	0.179	1.908
July	0.959	0.922	0.940	0.994	0.980	-0.218	0.163	1.088
August	0.945	0.917	0.931	0.990	0.985	-0.142	0.170	1.271
September	0.941	0.961	0.928	0.972	0.987	-0.093	0.171	1.686
October	0.950	0.935	0.942	0.981	0.992	-0.046	0.187	1.863
November	0.948	0.925	0.936	1.000	0.987	-0.071	0.188	1.882
December	0.933	0.925	0.929	0.962	0.996	-0.017	0.197	2.667

36. Equation (10) shows the difference between the unit value and Paasche index, rounding aside, to be solely governed by the expression for the levels effect, as demonstrated in Table 2. The levels effect is a significant and quite volatile generally positive factor with the unit value index exceeding the Paasche price index by over 8 percent in November, but being very similar in May. Table 2 shows the substitution effect to generally have a countervailing effect to that of the levels effect bring the unit value index closer to the price indexes. For example, in June the two effects are almost offsetting, with the unit value index very close to the Fisher price index. But this is unusual; the levels effect generally dominates driven by the increasing dispersion in quantities over time.

Table 2.

Understanding the Differences Between Unit Value Indexes and Laspeyres, Paasche, and Fisher Price Indexes

Table 2.

Understanding the Differences Between Unit Value Indexes and Laspeyres, Paasche, and Fisher Price Indexes

	Ratio of unit value index to:			Substitution effect	Level effect
	Laspeyres	Fisher	Paasche
January	1.000	1.000	1.000
February	1.061	1.061	1.061	1.000	1.062
March	1.053	1.050	1.047	1.003	1.048
April	0.972	0.979	0.986	0.993	0.986
May	0.985	0.994	1.002	0.992	1.002
June	0.993	1.006	1.019	0.987	1.019
July	1.037	1.057	1.078	0.980	1.079
August	1.047	1.064	1.080	0.985	1.081
September	1.033	1.047	1.061	0.987	1.062
October	1.033	1.041	1.050	0.992	1.050
November	1.056	1.069	1.082	0.987	1.083
December	1.031	1.035	1.040	0.996	1.040

View Table

Table 2.

Understanding the Differences Between Unit Value Indexes and Laspeyres, Paasche, and Fisher Price Indexes

	Ratio of unit value index to:			Substitution effect	Level effect
	Laspeyres	Fisher	Paasche
January	1.000	1.000	1.000
February	1.061	1.061	1.061	1.000	1.062
March	1.053	1.050	1.047	1.003	1.048
April	0.972	0.979	0.986	0.993	0.986
May	0.985	0.994	1.002	0.992	1.002
June	0.993	1.006	1.019	0.987	1.019
July	1.037	1.057	1.078	0.980	1.079
August	1.047	1.064	1.080	0.985	1.081
September	1.033	1.047	1.061	0.987	1.062
October	1.033	1.041	1.050	0.992	1.050
November	1.056	1.069	1.082	0.987	1.083
December	1.031	1.035	1.040	0.996	1.040

37. The unit value index is compiled for heterogeneous TVs comprising 17 brands and several characteristics as detailed above. They are broadly comparable items. We estimated hedonic regressions for each month as price on the quality characteristics and quality-adjusted the prices as outlined in equations (14) and (15). The quality-adjusted unit value index is given in Figure 3. Between January and February the unit value index increased by about 6 percent, reflecting an increasing quantity of purchases directed to more expensive sets, better brands. Yet when we take out the effect of such quality differences, the change in the mix of the characteristics purchased, there is a fall in the prices of about 5 percent. Consumers are paying more on average for better sets, but given their valuation of what the improved mix in characteristics are worth, the result is an overall fall in average (unit) prices. Similarly, in November to December, the unit value index fell by about 4 percent, as the bundle of TVs purchased included a higher proportion of cheaper models, but the quality-adjusted unit value index actually increased reflecting the fact that the fall in average prices is not compensating for the fall in quality that gave rise to it.

View Full Size

Quality Adjusted Unit Value and Fisher Price Indices

Citation: IMF Working Papers 2009, 019; 10.5089/9781451871661.001.A001

• Download Figure
• Download figure as PowerPoint slide

38. Also given in Figure 3 are the Fisher index, equation (1) and the weighted average of the quality-adjusted unit value index and the Fisher index (equation (16). The average value of the R² for the hedonic regressions was 0.63, the quality-adjusted unit value index receiving the lower weight of 0.37, on average. As a result the weighted average tracks the Fisher price index more closely than the quality-adjusted unit value index.

VI. Conclusions

39. For the aggregation of homogeneous items, the unit value index is the best index and superlative index numbers biased, and for the aggregation of heterogeneous items, superlative index numbers are best index and unit value index numbers biased.

40. The factors determining the difference between unit value indexes and Laspeyres, Paasche and Fisher price indexes were established in Section III. They comprise a substitution bias (for unit value to Laspeyres and a countervailing one from Fisher to Laspeyres) and a levels effect. The conditions for the unit value index to equal the price indexes were established to be implausible.

41. For items that are very similar, a unit value index remains appropriate for it is necessary to capture the effect of a change in price levels, and price indexes do not properly do this. Quality adjustments to the prices to mitigate price dispersion due to the slight product heterogeneity would be appropriate.

42. The determination of whether or not an item is homogeneous is critical to the choice of index number formula, but in practice many items are broadly comparable, and neither a unit value nor a Fisher price index is appropriate. The more similar the items aggregated, the stronger the case for a heterogeneity-adjusted unit value indexes. It follows that an appropriate formula may be based on an average of a heterogeneity-adjusted unit value index and a Fisher price index. The weighting ascribed to each should be an indicator of the similarity of the items. A possible indicator explored in this paper is the extent to which the price variation can be explained by price-determining characteristics: the (explained) sum of squares from a hedonic regression. While the discussion has been phrased in terms of hedonic regression analysis the principles apply to simpler quality adjustments.