MAC OS 视频处理
Core Video Programming Guide Core Video Concepts
Core Video is a new model for handling digital video in Mac OS X. It provides two major features to simplify video processing:
; A standard buffering model that makes it easy to switch between uncompressed video frames (such
as from QuickTime) and OpenGL.
; A display synchronization solution.
This chapter describes the concepts behind these features.
Figure 1-1 The Core Video pipelineThe Core Video pipeline
The movie’s frame data comes from your video source (QuickTime, for example) and is assigned to a visual context. The visual context simply specifies the drawing destination you want to render your video into. For example, this context can be a Core Graphics context or an OpenGL context. In most cases, a visual context is associated with a view in a window, but it is possible to have offscreen contexts as well.
After you specify a drawing context, you are free to manipulate the frame as you wish. For example, you can process your frame using Core Image filters or specify warping effects in OpenGL. After doing so, you hand off the frame to OpenGL, which then executes your rendering instructions (if any) and sends the completed frame to the display.
Within the Core Video pipeline, the most important facets for developers are the display link, which handles display synchronization, and the common buffering model, which simplifies memory management when moving frames between various buffer types. Most applications manipulating video need to use only the display link. You need to worry about using Core Video buffers only if you are generating (or compressing) video frames.
The Display Link
To simplify synchronization of video with a display’s refresh rate, Core Video provides a special timer
called a display link. The display link runs as a separate high priority thread, which is not affected by interactions within your application process.
In the past, synchronizing your video frames with the display’s refresh rate was often a problem,
especially if you also had audio. You could only make simple guesses for when to output a frame (by using a timer, for example), which didn’t take into account possible latency from user interactions, CPU
loading, window compositing and so on. The Core Video display link can make intelligent estimates for when a frame needs to be output, based on display type and latencies.
Figure 1-2 shows how the display link interacts with your application when processing video frames. Figure 1-2 Processing video frames with the display link
; The display link calls your callback periodically, requesting frames.
; Your callback must then obtain the frame for the requested time. You get this frame as an OpenGL
texture. (This example assumes that your frames come from QuickTime, but you can use any video
source that can provide frame buffers.)
; You can now use any OpenGL calls on the texture to manipulate it.
If for some reason the processing takes longer than expected (that is, the display link’s estimate is off), the video graphics card can still drop frames or otherwise compensate for the timing error as necessary. Buffer Management
If your application actually generates frames for display, or compresses incoming raw video, you will need to store the image data while doing so. Core Video provides different buffer types to simplify this process.
Previously, there was a lot of overhead if you wanted to, for example, manipulate QuickTime frames using OpenGL. Converting between various buffer types and handling the internal memory housekeeping was a chore. Now, with Core Video, buffers are Core Foundation-style objects, which are easy to create and destroy, and easy to convert from one buffer type to another.
Core Video defines an abstract buffer of type CVBuffer. All the other buffer types are derived from the CVBuffer type (and are typed as such). A CVBuffer can hold video, audio, or possibly some other type of data. You can use the CVBuffer APIs on any Core Video buffer.
; An image buffer is an abstract buffer used specifically to store video images (or frames). Pixel
buffers and OpenGL buffers are derived from image buffers.
; A pixel buffer stores an image in main memory.
; A Core Video OpenGL buffer is a wrapper around a standard OpenGL buffer (or pbuffer), which
stores an image in video (graphics card) memory.
; A Core Video OpenGL texture is a wrapper around a standard OpenGL texture, which is an
immutable image stored in graphics card memory. Textures are derived from a pixel buffer or an
OpenGL buffer, which contains the actual frame data. A texture must be wrapped onto a primitive
(such as a rectangle, or a sphere) to be displayed.
When using buffers, it is often useful to manage them in buffer pools. A buffer pool allocates a number
of buffers that can then be reused as needed. The advantage here is that the system doesn’t have to devote extra time allocating and deallocating memory; when you release a buffer, it goes back into the pool. You can have pixel buffer pools in main memory and OpenGL buffer pools in video memory.
You can think of a buffer pool as a small fleet of cars bought for corporate use. An employee simply takes a car from the fleet when needed and returns it when she’s done with it. Doing so requires much less overhead than buying and selling a car each time. To maximize resources, the number of cars in the fleet can be adjusted based on demand.
In a similar fashion, you should allocate OpenGL textures using a texture cache, which holds a number
of textures that can be reused.
Figure 1-3 shows a possible implementation of the frame processing that occurs under the hood when processing QuickTime movies, showing the use of a number of buffers and buffer pools to store video data as it progresses from compressed file data to the actual pixel images that appear onscreen. Figure 1-3 Decompressing and processing a QuickTime frame
The steps in the frame processing are as follows:
; QuickTime supplies the video data stream that will be turned into individual frames. ; The frames are decompressed using the specified codec. A pixel buffer pool is used to hold key
frames, B frames, and so on, which are needed to render individual frames.
; Individual frames are stored as OpenGL textures in video memory. Additional image processing for
the frame (such as de-interlacing) can be done here, with the results being stored in an OpenGL
; When you request a frame from Core Video (in response to the display link callback), the OpenGL
buffer contents are converted to an OpenGL texture that is then handed to you.
What’s in a Frame?
A video frame often has information associated with it that is useful to the system that displays it. In Core Video, this information is associated with a video frame as an attachment. Attachments are Core
Foundation objects representing various types of data, such as the following common video properties: ; Clean aperture and