In my recent article ‘Why Astro‘, I highlighted the risks in using the GPS. Since writing that article, I am frequently asked “if astro / celestial navigation is to be used, which of the many systems is the best”. Sight reduction methods tend to fall under two categories, Formula and Tabular. (Computerised sight reduction systems will involve a combination of these methods i.e. mathematical calculation of data contained within a matrix held in a database). Therefore, in this article, I will discuss the relative merits of these two methods.
Please note that the terms ‘Astro Navigation’ and ‘Celestial Navigation’ are synonymous but for the rest of this article I will use the term astro navigation.
Sight Reduction. This is the process of reducing the data gathered from observations of celestial bodies down to the information needed to establish an astronomical position line. The two essential items of data that we need to begin the process of sight reduction are the azimuth and the altitude of the celestial body in question.
The azimuth gives us the direction of the celestial body from the calculated position.
When we measure the altitude, what we are really trying to establish is the zenith distance (Zenith Distance = 90o – Altitude) that is the distance to the geographical position of the body. We measure the altitude at our true position and we calculate the altitude at the DR position (or assumed position); this enables us to calculate the zenith distances at the two positions. The difference between the two zenith distances will give us the distance from the DR position to the true position measured along the direction line of the calculated azimuth.
Measuring the altitude and azimuth at the true position with a sextant and azimuth compass is relatively straightforward but calculating what they would have been at the DR or Assumed position is the real work of sight reduction.
Formula Methods. The traditional way of calculating the azimuth and zenith distance at the DR position is by spherical trigonometry. Before the advent of the electronic calculator, this would have been a very lengthy and time consuming method involving the use of tables of logarithms to make calculations involving the Haversine Formula. However, these days we can still make use of spherical trigonometry with the use of a scientific calculator and with the application of just two formulas derived from the Cosine Rule, one for the azimuth and one for the zenith distance. With just a little practice, it will be found that this method is quick and easy to apply. We usually refer to these methods as Formula Methods. Accuracy is the greatest advantage of formula methods; calculations are usually made to 3 or 4 decimal places but this can be extended if greater accuracy is required. Of course there is always the risk of human error when making mathematical calculations but with an electronic calculator, it takes very little time to double check.
TABULAR METHODS. During the twentieth century, tabular sight reduction methods were first devised and today there is such a proliferation of these methods that choosing one can be very confusing. Tabular methods do not require a knowledge of spherical trigonometry; they involve the use of sets of pre-computed tables of data from which the altitude and azimuth can be interpolated. The disadvantage of these tables is that they have to be entered with the latitude and Local Hour Angle rounded to the nearest degree so that calculation of the altitude and azimuth depends on interpolation and extrapolation. To achieve this, an ‘Assumed Position’ has to be chosen. This is a position where the latitude and longitude closest to the DR position have the following properties: The assumed latitude is the DR latitude rounded up to the nearest whole degree and the assumed longitude is the longitude closest to the DR longitude that makes the local hour angle a whole degree. In comparison, when we solve the problem directly by spherical trigonometry, we use the latitude and longitude of the DR or EP position and we make exact calculations without the inaccuracies of interpolation methods. Of course the greatest advantage of tabular methods is that the navigator does not require a knowledge of trigonometry and the only mathematical calculations needed involve simple arithmetic.
Comparison. The links below show two examples of calculations made to establish astronomical position lines from identical input data. The first example shows the calculations made using the cosine formula method and the second shows those made using the Rapid Sight Reduction Method (NP303). Please note that the sight reduction forms used in these examples are not standard but are designed as learning aids for use with exercises in my books.
Formula Sight Reduction Method
Findings. There is a difference of 1.558 nautical miles in the calculation of the intercepts produced by the two methods, that is to say there is a difference of 1.558 arcminutes in the two sets of calculations. In terms of distance, 1.558 nautical miles seems quite a lot but in terms of mathematical calculation, 1.558 arcminutes does not seem so great a difference (unless you are a rocket scientist of course). So how do we decide which method is the more accurate?
- On the one hand, it could be argued that the formula method is the more accurate of the two methods for the following reasons: There are accumulative and unavoidable errors caused by the addition and rounding-off of quantities taken from sight reduction tables whereas with formula methods; calculations are usually made to three or more decimal places thereby providing a greater degree of accuracy.
- On the other hand, it could be argued that sight reduction by the use of spherical trigonometry is time consuming and gives considerable scope for mathematical error. Because time and accuracy are of the essence in practical navigation, it is an advantage to be able to calculate altitude and azimuth by relatively simple table operations.
Summary. The arguments above are really inconclusive and it would seem that, from the point of view of accuracy, there is not a great deal of difference between the two methods. Yet, we are told that the accuracy of astronavigation position fixing is only plus or minus 1 nautical mile, so if it’s not the method, where does this level of inaccuracy stem from?
Errors That Occur No Matter Which Sight Reduction Method Is Used. If we are concerned about accuracy in astro navigation, it matters not which sight reduction method we use, the real danger of inaccuracy lies in other areas. Inaccuracy in calculations may be introduced by a number of contributory errors irrespective of the sight reduction method being used; these errors are summarized below.
Errors in the observed altitude. Even when the sextant altitude has been corrected for index error, semi-diameter and parallax, the resultant altitude reading may still be incorrect owing to a combination of other errors such as incorrect calculated values for dip and refraction. An error in the observed altitude will lead to an error in the observed zenith distance.
Refraction. A pronounced error in refraction is likely to occur when the altitude is below 15o. The dip being affected by refraction is the most likely cause of error; when atmospheric conditions are abnormal, the actual value of dip may differ from the tabulated value by up to 10′.
Deck-watch error. If the deck-watch error is incorrect, the GMT and the LHA will be incorrect. An error in the LHA will lead to an error in the calculated altitude and this will cause the position line to be displaced.
Errors in the D.R. position. Errors in the course and distance laid down on the chart may result from a combination of inaccurate plotting, compass error. the effects of wind and tidal stream and incorrect calculation of speed made good over the ground. An error in the DR position and resultant assumed position will lead to errors in the estimated longitude and hence the local hour angle and this in turn will lead to an error in the calculated altitude.
Nautical Almanac. There are accumulative and unavoidable errors caused by the addition and rounding-off of quantities taken from the almanac.
Errors In Observed Positions Derived From More Than One Position Line. Position lines obtained from two or more astronomical observations are not likely to pass through a common point. The reasons for this are firstly, the observations are not likely to be taken simultaneously since it is not possible to take sextant readings of three several celestial bodies at the same instant. The faster a vessel travels, the greater the movement of the observer between the three observations and the more significant this error becomes even when special methods of calculation such as ‘MOO’ are used. Secondly, observed altitudes are very seldom correct and therefore, the resultant observed zenith distances will not be correct. For these reasons, the resultant position lines will be displaced and a ‘cocked-hat’ will be formed and because the position within the triangle of the cocked-hat is arrived at by guess-work, it is unlikely to be correct.
Conclusion: Arguments concerning the relative accuracy of different sight reduction methods are not important, the real cause of inaccuracy in astro navigation is more likely to stem from the types of error described above and these can occur irrespective of the method being used. For the average yachtsman, sailing in the vast expanse of the ocean, an accuracy level of plus or minus 1 nautical mile is probably nothing to worry about but for those engaged in activities that require a greater level of accuracy such as surveying and naval operations, it is obviously a matter of concern.
Wish to learn more? The cosine formula method, it is comprehensively taught in my book ‘Celestial Navigation – Theory and Practice’. The Rapid Sight Reduction Method is comprehensively taught in my book ‘Astro Navigation Demystified’.
Other Books by Jack Case:
Applying Mathematics to Astro Navigation
Astronomy for Astro Navigation
Astro Navigation Demystified 
You must be logged in to post a comment.