Margin of Error

This term with its mention of ‘error’ strikes terror into the heart of business executives when they are asked to make decisions based on market research findings.  It seems that every introductory statistics course covers ‘margin of error’ so that many business people know enough about this topic to know that it is important.  But should this subject be as important as the emotive term ‘error’ seems to make it?   Does it really need to be raised at nearly every meeting when market research results are presented?

Consider the financial information that business people usually take as gospel.  From the end of the financial year to when annual financial reports are provided, what do accountants do?  What they do is make adjustments within a set of agreed accounting rules.  Within these rules there is wide scope for different treatments that can produce fundamentally different financial results.  So what is the margin of error in financial results?  Businesses employ auditors to ensure that the margin of error in financial accounting is at an acceptable level.  And what about when accountants work outside the rules?  What was the margin of error for the financial results at WorldCom? 

So while it is important to ensure that market research information is reliable, it is just as important to ensure that all business information is reliable and to know what its margin of error is. 

Minimising Risk

I believe its best to not just look at the margin of error, but instead to look at the reliability of business information.  Risk is often a critical element for businesses to weigh up when making decisions.   Businesses invest heavily in gaining the information needed to minimise the risk involved in making decisions with a large financial impact.  But even with the very best information that is available, there is still no absolute certainty that any decision is the right one.  Consider a company interested in acquiring another business.  They may build models and test scenarios to determine the current and future value of a business but these models always have a margin of error.  In the end, a decision has to be made based on the best available information complemented by experience and intuition.  A decision based entirely on the wrong information has a high degree of risk, while if good information is available the risk is reduced.  When making a business decision we need to know how reliable the information is.   

Reliable Information

Many factors impact the reliability of market research information.  They range from the quality of questionnaire used, whether the people interviewed are users or decision makers, the survey response rate, sample selection, through to the precision of the sample.   

When businesses consider if they should base business decisions on what Customer Value Added information is telling them, they often ask about the sample size.  As sample size is one of the factors that determines the reliability of survey data, it is a good thing to ask about.  There are usually very good reasons (e.g. cost or time constraints) for using a sample to make inferences about the wider target population.  The downside from using a sample is that it opens up the possibility of measurements being less than 100% reliable.  On the other hand, a census where you don’t gather the views of close to 100% of the people may be less reliable than a random sample. 

There are questions of accuracy (is the sample representative of the population?) and precision (what margin of error are we willing to tolerate?) which need to be considered before sampling in order that the statistics will be sufficiently reliable for decision-making.  Having said this, a carefully selected sample covering only a small fraction of the population is usually sufficient for producing customer value performance measurements on which business decisions can be reliably based.  We usually use a cell size of 70 interviews for Customer Value Added surveys as statistical analysis confirms that this provides information at a precision of above 95%.  Each cell relates to a market sub-segment where information is required. Experience indicates that in practice the data reliability is far higher than that predicted by the statisticians.  In some situations where you are interviewing competitor customers, it just isn’t possible to interview more than 30 people.  In these situations, we accept a lower level of precision knowing that this level of interviews provides the best available information.

Random Sampling

Even more important than sample size is the need to have a random sample.   It is important to draw a sample from the population under consideration that is as close to a ‘slice’ of the entire population as is possible.  The way that this is done is by making a random selection from the entire population making a selection using randomly generated numbers.  Statistical text books say that a practical way of doing this is to use a systematic sampling technique such as interviewing every 100th person.  If we use these techniques and record the views of at least 30 people, then statisticians assure us that we have a sample that represents the views of the wider population.

A measurement can be precise but not accurate.  For example a set of kitchen scales may give a consistently high reading every time.  In CVA surveys we use random sampling to ensure that we don’t consistently capture the views of a slightly wrong selection of the entire population.  This ensures we capture unbiased data in the interviews.   

Ensuring we have both a high level of precision and a random sample means that business people are confident that CVA information is reliable and can be used to guide decision making.

Kawhia

My first visit to Kawhia was in the late 1950s.  I remember how remote it seemed, driving there with my parents over the twisty gravel roads.  Another memory was of Kawhia’s racing whale boats.  Each year these boats from the days of whaling are raced in Kawhia’s regatta.  Since then I’ve visited Kawhia from time to time and it still looks much the same.  Although the main road in from Otorohanga is now sealed, the road from the north is still gravel, being used every year for some exciting car driving in Rally New Zealand.    

Noticing that the Kawhia Museum had recently opened a new building housing an original 1880s kauri plank whale boat, I thought it was time to revisit Kawhia.  The museum was a delight, full of interesting local exhibits that were presented well.  The volunteer staff were helpful and informative.  The photo above shows the superbly restored whale boat in its purpose built gallery.

After we visited the museum we spied a new fish restaurant in the main street.  Although the restaurant was new it was like a traditional New Zealand fish and chip restaurant with locally caught fish, served simply but well.  And on the table was a bottle of DYC malt vinegar and one of those red plastic squeeze bottles shaped like a tomato and full of Watties tomato sauce. 

The Kawhia Regional Museum Gallery is open 11am to 4pm, Wednesday to Sunday from October to April and on Saturdays and Sundays for the rest of the year.   You can find it on the waterfront near the wharf at Kawhia, New Zealand in the old County Council buildings or on the web at http://www.kawhia.co.nz then select Museum.

Raglan Weather
Customer Value Management New Zealand Ltd is a sponsor of the new Raglan weather station.
For the latest Raglan weather conditions, forecast, news and web cam visit: http://www.raglanweather.co.nz