Fundamentals of Arbitrary Waveform Generation Ch. 5
This chapter deals with the generation of serial signals for testing purposes.
Serial data is typically transported without a clock signal (this is reconstructed in the receiver). During generation, the most important defect of the signal is jitter.
Often, serial data is heavily degraded during propagation, due to cross-talk, attenuation, dispersion, reflection, and noise. Differential signalling helps with some of these problems.
The encoding of serial data is usually done with high/low levels, but PAM (pulse amplitude modulation) encodes more than one bit per sample through several levels. Data can be transmitted using "de-emphasis" (basically overdriving a bit if there was a transition) to compensate losses. Sometimes, data is transmitted by the polarity of short pulses, so that there is less DC on the line. Many protocols also specify an "idle level" between high and low.
Real-world signals are characterized, among other things, by high level, low level, rise time, fallilng time, noise, crosstalk, jitter, and frequency. Rise and fall times are usually measured at the 10%, 90% levels (or 20%, 80%, or some other combination).
Timing control of digital signal ist not really limited by the sampling rate. If you place a few (one is enough) samples on the edge, you can position the edge finely by adjusting the amplitude of the edge samples. But, quantization noise leads to (small) jitter in much the same way, especially if you have many samples per edge (and thus slope is small). The optimum samples per edge is pretty broad, where jitter is dominated by other, mostly sampling clock jitter and random noise.
Rule of thumb: Use two samples/edge for minimizing intrinsic jitter and getting full timing control. This limits serial baud rate to 1/5 of sampling frequency. Make sure to use a good "reconstruction filter" (sort-of low pass at the output chain.)
If you want to create jitter for testing purposes, you can modulate the 10 MHz reference (for low frequency jitter, internal PLL acts as filter), or, if your AWG supports it, you can feed it a modulated DAC frequency. Best is, of course, to calculate a jittered waveform.
There is such a thing as inter symbol jitter (ISI) in digital signals, which basically means that the last few bits (symbols) influence the edge of the current bit (symbol). A low-pass filter, for instance, adds this ISI.
Embedding is simulating the distortion of a signal path. "De-Embedding" is the opposite, where you predistort your signal such that it arrives cleanly. You need to measure/simulate the S21 parameter (sigh) of your circuit for this. It's more difficult for differential signals.
When simulating Gaussian noise, you have to clip it, because it is unbounded. It's easy to accidentally clip too much - the rare events are those that cause errors in the real world. Real noise is rarely Gaussian. It has thicker tails (a higher crest factor).
If you want to measure low bit-error rates under tested "random" noise, make sure your test-waveform is long enough. Man, they try to sell memory whereever they can... Or, you add noise programmatically, which of course they also have a product for.