Interesting, thanks for the effort.
Comments interspersed...
-----Original Message-----
>From: Jay Terleski <jayt@arraysolutions.com>
>
>Hi Jim,
>
>I ran your comments by Bob Clunn, the designer of the AIM and Power Aim
>Analyzers and he has responded below.
>
>Jay
>
>Bob Clunn's comments:
>
>
>It's interesting to see a detailed discussion of the issues concerning
>calibration
>and measurement.
>
>Statements like "works great" or "doesn't work" are subjective and depend on
>what the user is trying to accomplish and what kind of performance he expects.
>On
>eham.com reviews can be found ranging from 1 to 5 for the same product, the
>perception of quality depends on the circumstances.
I fully agree on subjective evaluations of "usefulness" and the tendency of
eham reviews to be distributed at the ends of the scale. However, here, we are
talking about quantitative measures of accuracy, not a subjective evaluation of
works for me or doesn't.
>
>"It's one thing to calibrate out a few degrees of phase or a few tenths of a
>dB shift to compensate for the non-ideal directivity in a probe or the
>loss/mismatch in a connector or test port cable. It's entirely another to
>calibrate out the response of a reactive device with S11 and S21 varying by
>orders of magnitude over the passband."
>
>
>
>The AIM uses a procedure that compensates for non-ideal parameters over a
>dynamic range of about 60+ dB. This is illustrated in the manual in the
>section about Custom Calibration starting on page 29. Several graphs show the
>effectiveness of the cal procedure in canceling the effect of a complex
>transmission network.
Indeed, and I have no quibble with the calibration technique used, nor with the
fact that it can potentially cancel over a 60 dB range. My concern is with the
effect on the measurement uncertainty of the resulting "calibrated measurement"
(i.e. the number you get AFTER subtracting/dividing out the network), because,
typically, this is not discussed.
That is, there's no "error bars" on the measurement display, and, because of
the calibration process, the size of the error bars is NOT the same across the
display, and, in fact, the bars could be larger than the height of the graph.
Most people, when presented with a spec like "the analyzer is accurate to 5%"
make the assumption that the uncertainty of the displayed calibrated
measurements are about that value everywhere. There's also often an unwritten
assumption that the spec is "5% of full scale", which, of course, is 20% of a
quarter scale measurement.
The problem is aggravated when making measurements with things like spectrum
analyzers that display the results on a dB scale, rather than a linear one
(such as is typical for a voltmeter), and even worse if the log function is
being done in software from a basically linear measurement. A system using a 12
bit A/D to make a linear measurement is only accurate to 1 part in 4000 at full
scale (admittedly MUCH better than an analog meter). For a full scale signal,
the uncertainty is small (.025% or 0.002dB) However, that 1 LSB uncertainty is
about -72dBFS. If you start looking at signals, say, 40 dB down from
saturation/Full Scale, (that is at 0.01 of full scale voltage), the uncertainty
is now 100 times bigger (0.2dB).
>
>"Basically, if you divide a not very accurate measurement by another not
>very accurate measurement, the result isn't very accurate."
>
>Agreed. Another serious pitfall is to subtract two large numbers, their
>difference can have a large percentage variance.
>
Exactly, so when you make that measurement that is going to compensate for the
60 dB loss (as you noted above), the uncertainty in the 60dB is huge. If it's
5% of full scale, at 60 db down, it's 1000 times as large, that is, the
uncertainty is 5 times larger than the measurement!
When you subtract out the 60dB loss at a given frequency, maybe you really
should have been subtracting 53 db or 67 dB, or somewhere in between. The
uncertainty of the calibrated measurement (displayed as 0dB, instead of the
uncalibrated -60dB) could be +/-5dB, as opposed to a +/- 0.5dB for a
measurement of the same signal, without the loss and corresponding compensation.
(presumably it's not that bad... the uncertainty of the actual measurement is
more likely some small fixed amount (noise and quantization error of the A/D,
for instance) and some measurand proportional value.
>"This kind of uncertainty analysis is what separates typical experimental
>gear and bench experiments from "lab-grade" instruments like a VNA from
>Agilent. If I make a measurement on an unknown device with a professional
>VNA, not only do I get an estimate of what the unknown device's
>characteristics are, but I also get an estimate of the uncertainty of that
>estimate."
>
>
>
>Knowledge of the uncertainty is nice to have. Some people, like Larry Benko,
>W0QE, have compared the
>AIM to HP equipment (HP8753B, 3 GHz. two port vector network analyzer, with
>HP85046A S-parameter test set) and found it to be comparable over it's
>frequency range.
Has W0QE actually done an uncertainty analysis, or just a comparison of
identical test articles.
Were the test conditions designed to evaluate the calibration process?
Certainly, point comparisons can provide a clue to the overall measurement
uncertainty, but, without knowing the circumstances, you have no way to know if
maybe you just got lucky. The situation is similar to testing SWR meters.
It's easy to get good accuracy near 1:1, much tougher at 10:1, so unless you
actually test with carefully designed cases, it doesn't tell you much.
And then, anytime you are comparing a measurement from one instrument to
another, you have some combination of uncertainties to deal with. Often this is
dealt with by the "three cornered hat" technique (measure A against B, B
against C, A against C), especially if all the uncertainties are of similar
magnitude.
I don't have equipment like this myself, but I will be glad
>to hear from anyone who does a comparison study.
Actually, a comparison study is the wrong approach. This is more a
mathematical analysis problem, and you can confirm the analysis with some well
chosen attenuator pads (which don't, themselves, have to be calibrated, just
stable). That is, you make some measurements without the pads of some
impedance test articles. Then, you put the pad on, run the cal, and make
measurements of the same test articles(through the pad). The uncertainty in the
measurements of the test articles should increase (i.e. get noisier) in a
predictable fashion.
There's quite a bit of literature out there on calibration and measurement
uncertainty if one wants to get to the gnat's eyelash. However, I think that
for something like the AIM4170, one could probably do the analysis on the basis
of the block diagram, and some information about things like the noise floor
into the A/D, etc. You're probably not looking to characterize the 5 digit of
the reflection coefficient, after all.
>The AIM compares quite favorably to other antenna analyzers in it's price
>range, according to data obtained by the ARRL lab.
>These comparisons can be seen on the w5big.com website.
Sure, but the ARRL lab didn't go out and to any sort of testing to quantify the
measurement uncertainty. They checked the meter against some standards and
said, yep, it reads about the same as we expected. They didn't run multiple
calibrations with different networks and verify that the calibration process
"cancels out" the network correctly.
Let's be clear.. I'm not knocking the AIM4170's performance. It does a great
job for the price, but I think that the performance of these devices in general
is somewhat oversold, particularly when using functions like "fixture
deembedding" (which is what calibrating out the filter basically is).
>
>"So, can you point us to an analysis of the accuracy of the calibration
>method? How much (in numbers) does the filter degrade the accuracy of the
>measurement (or, more correctly.. what's the uncertainty of the calibrated
>measurement after applying the calibration)."
>
>
>The effectivness of the calibration procedure in several situations is
>illustrated in the manual. For a particular situation, you can check the
>calibration with known loads to see if it's sufficient for your
>requirements. Depending on the degree of compensation required, the accuracy
>when measuring through a complex network, should be within 2 or 3 percent.
>Ideally it would depend
>only on the accuracy of the loads used for calibration, but there is some
>uncertainty in the raw data, so the compensation is not perfect.
OK, so you don't have the "analysis of accuracy". WHat you have is some
empirical evidence that it seems to work in a few cases, and an unsubstantiated
assertion that it "should be within 2-3 percent" (which, frankly, is tighter
than the advertised accuracy of 5%, but, I recognize that the actual
performance of the device is probably a lot better than 5%)
As you point out, there is some uncertainty in the raw data. The real
question, then, is what is that uncertainty?
This whole thing of the theoretical performance and uncertainty of the new crop
of VNAs and 1 port analyzers is actually pretty interesting, and I've been
meaning to do the analysis and write it up, but just haven't got around to it.
One thing is certain: they are sufficiently different from the kinds of
measurements hams typically do that "intuitive" approaches to the uncertainty
analysis are full of traps. It's sort of like the false precision one gets
with the cheap 4 1/2 digit multimeter with a specified accuracy of 5%. The
display may have 4 digits, but only the first two are meaningful. I think most
folks understand this by now, but with the new types of instruments, it's a bit
tougher to have an intuitive understanding of the effects of various things on
the measurement accuracy.
HP realized this, and provided an application that ran with the data extracted
from your 8510 VNA that actually calculated the measurement uncertainty for
your measurements. The newer PNAs do it internally, and the data sheet for the
VNA has little graphs that tell you what the measurement uncertainty is for
various stimulus levels and S21 or S11 values (for instance, if you're
measuring cable or attenuator loss, the measurement uncertainty gradually rises
as the loss goes up, so there's a graph of uncertainty vs Mag(S21))
WHat they don't do (yet.. or maybe they do, and I haven't found the option
button) is put error bars on the display. I'd love to have the phase(s21)
display have error bars, so I don't have to explain that the 1 degree ripple in
the graph is less than a 10th of the measurement uncertainty.
>
>
>"In fact, the Ap Note describing the 160m antenna measurements is quite
>suspicious, from a calibration standpoint, because you made use of
>"smoothing"..Those fluctuations with respect to frequency in the calibrated
>response before smoothing are often indicative of the calibration constants
>being derived from a noisy measurement of the standard. (the whole dividing
>a small noisy number by another small noisy number problem...)."
>
>Calibration constants for any instrument are derived from measurements that
>are noisy.
>
>During the calibration procedure, the AIM averages 16 readings at each
>frequency point. This a perfectly valid mathematical technique. Averaging
>is done in the hardware. Smoothing
>uses software to do a running average to further reduce the effect of
>measurement noise.
It's a valid mathematical technique, if the statistical behavior of the thing
you're averaging is understood well enough. The key is whether the errors are
uncorrelated. If the thing being averaged out is, say, white gaussian noise,
then you're in great shape. If the thing being averaged out is, say, a
spurious signal putting a sinusoidal error, then maybe not. If the averaging
is done before or after a nonlinear operation (e.g. linear to dB conversion) it
makes a difference.
This one is EASY to test. Make a series of measurements of the same test
article with different averaging sizes. Say, 100 measurements without
averaging, then 100 measurements averaging 10 samples, then 100 measurements
averaging 100 samples, etc. If you look at the variance of the collections of
100 measurements, they should change in proportion to the number of samples
being averaged. That is, the variance of the 100 measurements without
averaging should be 10 times that of the variance of the 100 measurments which
average 10 samples. There are more sophisticated tests that require less data
collection, but basically what they do is confirm your assumption that the
errors have a consistent distribution, and that they are uncorrelated.
Is the running average over frequency points, or at a single frequency? If
it's over multiple points, then how do you know that you aren't "averaging out"
actual differences in the signal you're measuring? If you have a device with
reasonably high Q, particularly if there's some transmission lines involved,
the bounces up and down could be "real", and by smoothing them out
>
>
>
>Comment added by Jay, The 160m Antenna in the application note performed
>exactly as the VSWR curve on the graph indicates as verified by using a
>transceiver, and our NIST traceable VSWR/ power meter.
>
>The intent of the application note was not to show .1% accuracy, but to
>demonstrate the usefulness of the hardware and software tools which we think
>are way beyond what any $500 instrument provides, I hope you agree with this
>statement. The average amateur can solve some typical amateur problems
>without resorting to high dollar equipment and tune up a 160m vertical.
Sure.. These small microprocessor based things are a godsend. When I got my
first MFJ 259, I thought "life is much better now" when doing mundane things
like tuning a multiband dipole at field day.
The issue I have is when people throw around terms like "lab grade" or start to
believe that because it works just like a Agilent PNA and has the same number
of digits on the computer display, it must be as accurate as the PNA. And,
hams aren't unique here. More than one junior engineer at work has come in
asking why the device differs from the datasheet by 1 dB, based on power levels
measured with a spectrum analyzer at 20dB above the noise floor, where the
absolute accuracy is probably 2 or 3dB, at best. (or worse, yet, the poor
engineer puts that data up on the screen in a review, and gets beaten up by the
review board...)
Enjoy all,
Jim, W6RMK
_______________________________________________
_______________________________________________
TowerTalk mailing list
TowerTalk@contesting.com
http://lists.contesting.com/mailman/listinfo/towertalk
|