This article provides an introduction to the concepts and resources provided by the Avaya Client SDK to support integration of video features into your application.
A developer is able to control the Z-order of just two surfaces in an application - the window surface (the upper surface) and a surface view (the lower surface). Android framework developer Dianne Hackborn confirmed this:
I strongly recommend against using multiple surface views in a window. The way surface view is implemented is that a separate surface is created and Z-ordered behind its containing window, and transparent pixels drawn into the rectangle where the SurfaceView is so you can see the surface behind. We never intended to allow for multiple surface views.
Prior to 2.0 multiple surfaces on top of each other were not supported, and any tricks you use to try to make this work are likely to break in the future or on different devices. (The framework made no guarantees about what order they would be placed in the window manager, it was just happenstance, and could change based on various conditions that may or may not have the same result in the future.)
As of 2.0, there is a hack you can use to at least control the Z-order of two surface views: setZOrderMediaOverlay.
With this limitation, a GLSurfaceView must be shared with all activities within the application that need one; as it must be treated like a singleton. To this end, we need a way to select which "scene" we show, or render, in the GLSurfaceView at any given time.
This has several benefits:
Each scene, referred to here as abstract class Plane, implements three java interfaces - Positionable, Renderable, and Touchable. The logic for rendering and interacting with each plane is in each concrete implementation of Plane. There are a few reusable implementations of Plane.
The abstract class PipPlane is used for managing a Positionable object as part of a picture-in-picture layout. Gesture recognition is implemented to allow for dragging and throwing the Pip toward a desired corner, with a smooth animation to the final resting place.
Since Planes are also Positionable, instances of PipPlane can be nested to allow for picture-in-picture-in-picture (local video, over remote video, over content) for use in Tablet user experiences.
The class VideoPlaneLocal extends PipPlane. This class manages an instance of VideoLayerLocal; which is Positionable and Renderable, and implemented natively. This plane handles maintaining the aspect ratio of the video layer and scaling the Pip to best fit the available space. A setPlane method allows placing a child Plane beneath the floating local video layer. This allows the underneath plane to be changed at will without disrupting the local video layer.
The class VideoPlaneRemote extends Plane. This class manages a single instance of VideoLayerRemote; which is Positionable and Renderable, and implemented natively. This plane handles maintaining the aspect ratio of the video layer while filling the bounds of the Plane. VideoPlane has a delegate interface to notify its owner of touch events - single and double taps, long presses, drag and drop, etc.
Plane could be extended further to manage multiple instances of VideoLayerRemote when needing to render multiple streams within a single plane. The layers would need to be positioned to maintain each layer's aspect ratio.
The class BitmapPlane simply renders an android.graphics.Bitmap at a position within the plane by setting an android.view.Gravity constant. BitmapPlane is used in combination with VideoPlaneLocal to show a static image in place of VideoPlaneRemote, when the remote video is on hold, or otherwise not being received.
The root plane of the view hierarchy is set in an instance of PlaneViewGroup. This class, which extends ViewGroup, can be placed within a standard view hierarchy. Standard android.view.View classes can be added to the PlaneViewGroup, and will be drawn in the window surface above the GLSurfaceView. Views overlaid should support transparency where appropriate.
In an application that makes use of a TabActivity, the PlaneViewGroup can be added to the TabActivity's content view, and be available to all tabs. This allows multiple tabbed activities to share the single instance of PlaneViewGroup.
@Override
protected void onCreate(Bundle savedInstanceState)
{
super.onCreate(savedInstanceState);
planeViewGroup = new PlaneViewGroup(this);
tabHost = getTabHost();
tabHost.getTabContentView().addView(
planeViewGroup,
new LayoutParams(
LayoutParams.MATCH_PARENT,
LayoutParams.MATCH_PARENT));
}
In an application that uses fragments, the PlaneViewGroup can be set as the content view of the base Activity:
@Override
protected void onCreate(Bundle savedInstanceState)
{
super.onCreate(savedInstanceState);
planeViewGroup = new PlaneViewGroup(this);
planeViewGroup.setVisibility(ViewGroup.INVISIBLE);
setContentView(planeViewGroup);
}
In either case, when an activity or fragment that uses the PlaneViewGroup is started or stopped, the PlaneViewGroup visibility should be updated by the activity or fragment, in case other fragments do not need (or know about) the PlaneViewGroup:
private PlaneViewGroup getPlaneViewGroup()
{
return ((ScpTest) getActivity()).getPlaneViewGroup();
}
@Override
public void onStart()
{
super.onStart();
getPlaneViewGroup().setPlane(videoPlaneLocal);
getPlaneViewGroup().setVisibility(ViewGroup.VISIBLE);
}
@Override
public void onStop()
{
getPlaneViewGroup().setVisibility(ViewGroup.INVISIBLE);
getPlaneViewGroup().setPlane(null);
super.onStop();
}
PlaneViewGroup manages the GLSurfaceView as a child, and provides an implementation of GLSurfaceView.Renderer (PlaneRenderer). Touch and Render events are passed to the root plane.
The PlaneRenderer class manages a list of Renderable objects that have been registered by a Plane. Planes are notified of the PlaneRenderer when they are set as the root plane of the PlaneViewGroup. The plane can then register one or more Renderables that are interested in onSurfaceCreated and onSurfaceChanged events. These events allow renderables to setup any OpenGL resources - such as shaders, textures, projection matrix - in the OpenGL context. This allows all planes that have been shown (and could be shown again) to keep their assets loaded in the OpenGL context.
All user interface objects should be drawn using traditional means in the window surface. This surface should have a transparent alpha channel anywhere the GLSurfaceView should be allowed to shine through. This allows all UI controls to sit above the GLSurfaceView and be completely isolated from it.
In this framework, uncompressed video frames flow through a pipeline of objects that are sources and sinks. An object that produces video frames (capturer, decoder) is a VideoSource. An object that consumes video frames (encoder, renderer) is a VideoSink. Video flows from a VideoSource to a VideoSink. All VideoSource objects implement a single method:
public void setVideoSink(VideoSink videoSink);
This connects a sink to a source, much like connecting a television to a dvd player. In fact, a renderer can be connected directly to a capturer for a simple loopback unit test. At the moment, video frames are transferred natively by C++ objects. The VideoSink base class appears empty, but is backed by a native instance of IVideoSink; which receives IVideoFrame objects.
The VideoCaptureController.Params class is used to control a camera. It has the following key methods:
public VideoSource getVideoSource();
public void setLocalVideoLayer(VideoLayerLocal localVideoLayer);
public boolean hasVideoCamera(final VideoCamera cameraType);
public void useVideoCamera(final VideoCamera cameraType,
final Runnable callback);
The getVideoSource method returns the VideoSource object to attach a VideoSink. The setLocalVideoLayer method allows connecting a VideoPlaneLocal object for picture-in-picture video preview of what's being captured. The hasVideoCamera method lets us know if the device supports a front or back camera. The useVideoCamera method starts capturing from a specified camera, allows switching seamlessly to another camera, and stops capturing when passed a null camera.
The abstract class VideoLayer extends VideoSink and implements Positionable and Renderable interfaces. There are two concrete classes provided that can be used to render video - VideoLayerRemote and VideoPlaneLocal. The implementations of these classes is entirely native in C++.
Instances of VideoLayer support a VideoLayerListener:
public interface VideoLayerListener
{
/**
* Notifies when the video frame size has changed.
*
* @param width the width of the video frame
* @param height the height of the video frame
*/
void onVideoFrameSizeChanged(int width, int height);
}
The provided VideoPlane classes register for this event, so that the aspect ratio is known when positioning the video layer.
This class is optimized to render video frames received from a remote participant. The video frame is rendered and scaled to the extent of the layer's bounds; which is calculated by the VideoPlaneRemote class to conform to the video frame aspect ratio.
This class is optimized to render video frames received from the local video camera. It supports five additional methods:
public native void setHidden(boolean hidden, double duration, double time);
public native void setMirrored(boolean mirrored);
public native void setRotated();
public native void setCornerRadius(float radius);
public native void setBorderWidth(float width);
The setHidden method is called by VideoPlaneLocal in response to calling the setLocalVideoHidden method; which should be called in response to using a video camera. Toggling the visibility of the layer can be animated over time with a fade effect. The following Runnable is executed when the camera is finally started or stopped, so that the Pip stays on-screen while the camera is actually on:
videoCaptureController.useVideoCamera(videoCamera, new Runnable()
{
public void run()
{
videoPlaneLocal.setLocalVideoHidden(videoCamera == null);
}
});
The setMirrored and setRotated methods are called by the videoCaptureController. Mirroring the layer for the front-facing camera is a requirement, as not doing so will cause a 45-degree tilt of the device to appear as a 90-degree tilt to the user. Rotating the layer will perform a 3D animation with perspective when switching (flipping) from the front camera to the back camera of the device. These effects start when the next video frame is delivered; so they need to be coordinated with stopping/starting the capture device. The capture controller has the appropriate timing information to make the animations smoothly. The setCornerRadius and setBorderWidth methods may be called by the user to stylize the Pip. By default, there is a border width of 4px and a corner radius of 16px.
The VideoInterface instance (retrieved from the MediaServiceInstance object) provides the following key methods:
/**
* Retrieves a VideoSink for the specified channel.
* This is used to attach a VideoSource (capturer).
*/
public VideoSink getLocalVideoSink(int channelId);
/**
* Retrieves a VideoSource for the specified channel.
* This is used to attach a VideoSink (renderer).
*/
public VideoSource getRemoteVideoSource(int channelId);
The getLocalVideoSink method returns the VideoSink representing the encoder of the channel. This sink is normally attached to the capturer's source object:
VideoSink sink = videoInterface.getLocalVideoSink(channelId);
videoCaptureController.getVideoSource().setVideoSink(sink);
The getRemoteVideoSource method returns the VideoSource representing the decoder of the channel. This source is normally attached to a VideoLayerRemote object:
VideoSource source = videoInterface.getRemoteVideoSource(channelId);
source.setVideoSink(videoLayerRemote);
The following public classes in this package currently implement the Destroyable interface:
It is important that all objects instantiated that implement Destroyable be destroyed when appropriate; in order to release native C++ resources. It's easiest to maintain a list of objects to destroy in each activity or fragment:
private List destroyables = new ArrayList();
@Override
public void onDestroy()
{
for (Destroyable destroyable : destroyables)
{
destroyable.destroy();
}
super.onDestroy();
}
public class ScpTest extends Activity
{
private PlaneViewGroup planeViewGroup = null;
public PlaneViewGroup getPlaneViewGroup()
{
return planeViewGroup;
}
@Override
protected void onCreate(Bundle savedInstanceState)
{
super.onCreate(savedInstanceState);
planeViewGroup = new PlaneViewGroup(this);
planeViewGroup.setVisibility(ViewGroup.INVISIBLE);
setContentView(planeViewGroup);
}
@Override
protected void onStart()
{
super.onStart();
planeViewGroup.onStart();
}
@Override
protected void onStop()
{
super.onStop();
planeViewGroup.onStop();
}
}
public class VideoPlaneTestFragment extends Fragment
{
private List destroyables = new ArrayList();
private VideoLayerLocal videoLayerLocal;
private VideoLayerRemote videoLayerRemote;
private VideoPlaneLocal videoPlaneLocal;
private VideoPlaneRemote videoPlaneRemote;
private VideoCamera videoCamera = VideoCamera.Front;
private VideoCaptureController videoCaptureController;
private PlaneViewGroup getPlaneViewGroup()
{
return ((ScpTest) getActivity()).getPlaneViewGroup();
}
@Override
public void onCreate(Bundle savedInstanceState)
{
super.onCreate(savedInstanceState);
// create the renderers
videoLayerLocal = new VideoLayerLocal();
videoLayerRemote = new VideoLayerRemote();
// fetch the VideoInterface from the MediaEngineInstance
VideoInterface videoInterface = getEngine().getVideoInterface();
int videoChannelId = getCurrentCall().getVideoChannels().get(0)
.getChannelId();
// fetch the local video sink from the video channel
VideoSink sink = videoInterface.getLocalVideoSink(videoChannelId);
// fetch the remote video source and connect it to the sink
VideoSource source = videoInterface
.getRemoteVideoSource(videoChannelId);
source.setVideoSink(videoLayerRemote);
// create the local video source and connect it to the sink
videoCaptureController = new VideoCaptureController();
videoCaptureController.setLocalVideoLayer(videoLayerLocal);
videoCaptureController.getVideoSource().setVideoSink(sink);
// create the local video plane
videoPlaneLocal = new VideoPlaneLocal(getActivity());
videoPlaneLocal.setLocalVideoLayer(videoLayerLocal);
// create the remote video plane
videoPlaneRemote = new VideoPlaneRemote(getActivity());
videoPlaneRemote.setRemoteVideoLayer(videoLayerRemote);
// set the remote video plane as a child beneath the local
// video plane
videoPlaneLocal.setPlane(videoPlaneRemote);
// maintain a list of all objects we need to destroy
destroyables.add(videoCaptureController);
destroyables.add(videoLayerLocal);
destroyables.add(videoLayerRemote);
destroyables.add(sink);
destroyables.add(source);
}
@Override
public void onDestroy()
{
for (Destroyable destroyable : destroyables)
{
destroyable.destroy();
}
super.onDestroy();
}
@Override
public void onStart()
{
super.onStart();
getPlaneViewGroup().setPlane(videoPlaneLocal);
getPlaneViewGroup().setVisibility(ViewGroup.VISIBLE);
onCameraSelected(VideoCamera);
}
@Override
public void onStop()
{
onCameraSelected(null);
getPlaneViewGroup().setVisibility(ViewGroup.INVISIBLE);
getPlaneViewGroup().setPlane(null);
super.onStop();
}
protected void onCameraSelected(final VideoCamera VideoCamera)
{
this.VideoCamera = VideoCamera;
videoCaptureController.useVideoCamera(VideoCamera, new Runnable()
{
public void run()
{
videoPlaneLocal.setLocalVideoHidden(
VideoCamera == null);
}
});
}
}