GMFBridge and Timestamps

GMFBridge is a sample that shows how to separate DirectShow tasks into multiple graphs, so that some filters can be started and stopped independently from others and so that you can switch sources or renderers without stopping the graph.

Unless the graphs are all started and stopped in sync and share the same clock, they will have different time bases. Every sample that crosses a graph will need its timestamps adjusted for the new graph. However, there are several different ways to do this, appropriate to different situations. In this article, I’ve tried to describe the issues and explain how GMFBridge solves them.

Stream Time

As a little background, let me summarise the time handling in DirectShow very briefly.

Seamless File Playback

One common use for GMFBridge is to play a sequence of clips seamlessly. Each clip is opened in a separate source graph, and these are connected in turn to a single render graph. Within each source graph, the timestamps will go from zero to the clip’s duration, but in the render graph, the timestamps cannot suddenly go back to zero without a pause or seek. So we need to add on the previous timestamp so that they continue in sequence:

source		render
(clip 0)
0			0
33			33
66			66
100			100
...			...
10,000		10,000
(clip 1)
0			10,033
33			10,066
66			10,099

You can see this code in BridgeSource::OnNewConnection(), which will find the latest stop time used by the previous clip and use that as the baseline start time for the new clip. Each time there’s a new connection, we record the previous stop time and we add that on to all clips from then on.

Incidentally, if you stop and start the source graph while still connected to the bridge, this is also treated as a new connection, since the timestamps will start from 0 again.

Switching Live Sources

If you are switching between live sources, rather than files, this might not work so well. If the timestamps are adjusted to fit after the last timestamp of the previous source, then each time you switch sources, you might increase the latency slightly. In any case, this behaviour is not want you normally want with live sources. You want to switch to the next source now. It does not make sense to fit the current source after the previous source if both are live.

In this case, you can use BridgeAtDiscont to indicate to GMFBridge that the next clip is not seamless, but instead starts immediately. The bridge will get the current stream time, add on a latency to allow time for the samples to pass through the graph, and use that as the baseline time for the new graph. So all samples will be offset by this baseline.

You can review this code also, in BridgeSource::OnNewConnection(). If this is a discontinuity rather than a seamless change, the baseline time is set to the current stream time plus 300ms (as an allowance for how long the samples will take to flow through the graph).

As an aside, note that the base class StreamTime method only works when the graph is running. It simply subtracts the stream time offset from the current reference clock time. However, when the graph is paused, the stream time is not advancing, so you need to record the stream time at the moment of pause and use that until the next Run() call.

Preview Then Record

GMFBridge is often used to turn on and off the recording of a video without affecting the preview. A source graph contains the device capture filter, with the preview pin connected to a renderer. The source’s capture pin is connected to the bridge. When only preview is required, the graph is run and the capture pin’s samples are discarded at the bridge. When recording is turned on, the bridge is connected to a recording graph (containing multiplexor and file writer). Capture samples are then delivered over the bridge to the multiplexor and recording starts without the preview being affected at all.

When recording starts, the preview graph may have been running for a while. So the first timestamp arriving at the bridge will be well past zero. But the capture graph has just started, and it is expecting samples starting at zero. All the logic described above, whether using contiguous or discontiguous time bases, relies on the first sample from a new clip being based at zero.

To resolve this, the bridge sink (at the end of the source graph) will convert the timestamps to zero-based when connected to the bridge; the other end of the bridge will then convert the timestamps into the time base used in the downstream graph. This code is in BridgeSink::AdjustTime().

To demonstrate, the following table shows the timestamps that might be generated if two graphs were bridged when both graphs were already running, assuming that we use BridgeAtDiscont(true).

At first the graph is disconnected, so the samples are dropped

source	in bridge	render
0
33
66
100

Then the graph is connected. So the next sample is passed to the bridge, but adjusted to zero as the start of a new sequence. In the render graph at this point, the stream time is 1000ms, so the first sample is timestamped at 1000ms (now) plus 300ms latency.

133		0			1000 + 300 + 0  = 1300
166		33			1000 + 300 + 33 = 1333
200		67			1000 + 300 + 67 = 1367
...

Live Timing

Despite all these timestamp adjustments, there are still cases that GMFBridge does not handle well. Consider this: you have two cameras, and a single microphone, and you are streaming the output over the network. You need to switch between the cameras, so each of those is in a separate graph connected by a bridge. In the downstream graph, the video stream from the bridge is connected to the encoder and transmitter. You can switch seamlessly between cameras by bridging one or other of your source graphs to the encoder graph. You only have a single microphone, so that is connected directly in the encoder graph.

The problem is that every time the bridge connection is changed, the time base will change slightly. If the audio were being switched as well as the video, this would not matter, since the adjustments would be made to both audio and video timestamps. But with the audio in the encoder graph (staying unchanged), the sync between the audio and the video is drifting off, getting worse each time the bridge is changed.

Since the video and audio are live, we don’t really want to alter the timestamps at all. This can be done if all the graphs use the same clock. Then all that’s needed is to adjust for the stream time offset, to allow for the graphs starting at slightly different times. This can be done by adding on the stream time offset in the sink filter when entering the bridge, which converts the timestamp to an absolute reference clock time, and then subtracting the stream time offset in the source filter on leaving the bridge.

I have a version of GMFBridge that implements this alternative timing model, that is currently in testing. I will publish it once it has been tested.

Update: this is now available here

1st July 2011