Fundamentals of Arbitrary Waveform Generation Ch. 4

Choosing a DAC sampling rate close to the Nyquist limit allows saving memory, but will make filtering of unwanted images harder. The Agilent publication shows a scary example on page 116, where you can see how this degrades a signal even in the time-domain, because the reconstruction filter just doesn't work as well close to the Nyquist frequency. Also, digital equalizing will eat up more of the SFDR (spurious free dynamic range), and the noise power density of the quantization noise increases.

Using a "way too high" sampling rate is known as oversampling. It helps a lot.

Beware of record length granularity (RLG). For single shot signals, you can 0-pad (or otherwise pad) the waveform accordingly.

For periodic signals, 0-padding can lead to phase-jumps in loops. In that case, one way to deal with RLG is to adjust the sample rate of the waveform, but due to the fixed sample-rate of the DAC, this can lead to jitter, especially in digital signals. (I guess trueform is supposed to help here.) For sine waves, you may want this effect, because it can make sure that quantization noise is spread evenly across the first Nyquist band. The other way to deal with RLG is to use a record length where the waveform repeats an appropriate number of times. You can also combine both techniques.

A completely different method to combat periodic quantization noise is dithering, where you add some small, random value (uniform distribution within [-1/2 LSB, +1/2 LSB] to the ideal waveform before quantization (so before it is loaded into the AWG). If done intelligently, this add's a surprisingly small amount of noise, but spreads the quantization noise spectrum across the whole band. It can also increase the effective DAC-resolution for DC signals (like low-pass-filtered PWM).

Careful when digitally filtering a looped signal: Make sure the filter convolution is cyclic, too, otherwise you get "filter discontinuities". This is trivial for FIR, but IIR might need several passes to converge onto a final, filtered signal. When using sequences, things become even more complicated: Use link-segments!

Of course, you can also play back capture signals. Agilent recommends setting the sampling rate of the scope and AWG to the same value (or an integer fraction in between), both being as high as possible. The captured signal (captured with $f_\mathrm{ADC}$) has to be digitally filter such that it contains nothing above $f_\mathrm{DAC}/2$. It is then resampled. Playack creates sidebands in the other Nyquist band of the DAC, which are, as always, filtered by the output stage.

You can increase the fidelty of the played-back signal through averaging multiple waveforms, smoothing, and further digital filtering.

Side-Note: A scope's high-res mode usually means sampling the signal at a higher frequency, then binning it, and storing only the averages of the bins to memory. Analog noise acts as natural dithering, so this gives a better resolution.