Build a mobile video chat framework with the Nex Gen Media Server (NGMS) API

Digital Marketing

Introduction

Nex Gen Media Server is best known as a multipurpose streaming media server for delivering live and stored video to a variety of devices. The same media server can be integrated into a mobile application to facilitate real-time video communication. Here we will use the NGMS API to make it easy to create a video chat client for Android devices using the Android Software Developer Kit (SDK) and Google’s Android Native Developer Kit (NDK).

We will use NGMS to provide all RTP services for network transport, video frame compression and decompression. NGMS will run as a natively compiled shared object used by our Android app. We will use a Java Native Interface (JNI) bridge layer to invoke native methods from Java application code.

previous requirements

This article assumes that you have a basic understanding and at least an intermediate level of experience building Android apps. I won’t go over the Android project and configuration details in an iDE like Eclipse. If you’ve built Android apps before, or at least followed some sample tutorials, you should be able to use this tutorial to build a streaming video client.

If you haven’t already, you’ll need to download the Google Android Software Developer Kit (SDK). In this example I used Revision 15 on a 32-bit Linux machine. The Java Application Layer is used to create the actual Android application. The application code will interact with a JNI layer to invoke the NGMS API routines.

You will also need to download Google’s Android NDK (Native Developer Kit). In this example I used Revision 6b on a 32-bit Linux machine. The NDK will be used to create the native interface layer that serves as the link between the native code and the Java application code. You should also have a basic understanding of the C programming language.

You should first start by creating a basic Android app using an IDE like Eclipse. In this example, the application will be called ngmsclient. The Java package name will be called com.example.ngmsclient and will target Android OS 2.3 or higher. The root directory of the project should contain a folder called “jni” which will contain the native sources used to interact with the Java application code. The following example assumes you have a basic Android app up and running and will show you how to integrate an ExampleChat class into your Android project.

the native layer

The NGMS core built-in library is written in C and is packaged as a shared object file. Since the Android operating system is based on Linux, the NGMS core library will run natively within your Android app space. You will need to obtain the NGMS core library for Android OS from the ngmsvid.com website. The essential components are the library files libngms.so, libxcode.so, and the header file ngmslib.h. The ngms bundled.so files will be bundled together with your ngmsclient.apk to provide video chat services.

The JNI Layer
We will build a shared library called ngmsglue.so that will serve as a link between the ngms API and our Java application code. The Java Native Interface (JNI) directory structure within the ngmsclient project directory should look like this.


jni/Android.mk
jni/ngmsglue.c
jni/ngms/include/ngmslib.h
jni/ngms/lib/libngms.so
jni/ngms/lib/libxcode.so

ngmsglue.c will contain the code to control the NGMS input and output functions. The code will maintain two separate NGMS API contexts, one for stream output parameters and one for stream input parameters. You should note that the naming convention for each function must match the Java package and class name from which you are calling the native code.

ngmsStartReceiver is called to initialize the NGMS Capture Input API to listen on port 5004 for a network video stream encapsulated over MPEG-2 TS. Each successive video frame will be automatically demultiplexed and decoded into RGB565 pixel format. The entire frame can then be read by calling ngmsReceiveFrame.

ngmsStartSender is called to initialize the NGMS Stream Output API to send encoded video frames to a remote host over port 5004. ngmsTransmitFrame is called to encode and transmit a single raw frame in NV21 pixel format.


/*
* ngmsglue.c
*
* JNI layer to access the NGMS streaming and capture API
*
*/

#include
#include
#include
#include
#include "ngmslib.h"

/**
* Holds the configuration for the NGMS RTP receiver and decoder
*/
static NGMSLIB_STREAM_PARAMS_T ngmsReceiver;

/**
* Holds the configuration for the NGMS RTP sender and encoder
*/
static NGMSLIB_STREAM_PARAMS_T ngmsSender;

/**
* Starts the NGMS capture service
*/
jint Java_com_example_ngmsclient_ExampleChat_ngmsStartReceiver (JNIEnv* _env, jobject _thiz)
{

char xcoderConfig[512];
char sdpConfig[512];
int i;

/*
* Before doing anything call ngmslib_open to open the NGMS service
*/
i = ngmslib_open(&ngmsReceiver);
if (i!= NGMS_RC_OK)
{
__android_log_print(ANDROID_LOG_ERROR, "ngms", "ngmslib_open failed with error %d", i);
return -1;
}

/*
* Construct an xcoder configuration key value string
*/
sprintf(xcoderConfig, "vc=rgb565,vx=320,vy=240");

/*
* Construct an SDP configuration for the RTP receiver
* Here we setup NGMS to read data over a MPEG-2 Transport Stream.
*/
sprintf(sdpConfig, "sdp://"
"m=video 5004 RTP/AVP 33n"
"a=rtpmap:33 MP2T/90000n"
"a=fmtp:33n");

/*
* Customize the capture configuration parameters
*/
ngmsReceiver.inputs[0] = sdpConfig;
ngmsReceiver.strxcode = xcoderConfig;
ngmsReceiver.islive = 1;
ngmsReceiver.noaud = 1;

/*
* Start the RTP receiver
*/
i = ngmslib_stream(&ngmsReceiver);

if (i!= NGMS_RC_OK)
{
__android_log_print(ANDROID_LOG_ERROR, "ngms", "ngmslib_stream failed with error %d", i);
ngmslib_close(&ngmsReceiver);
return -1;
}

return 0;
}

/**
* Stops the NGMS RTP capture service
*/
jint Java_com_example_ngmsclient_ExampleChat_ngmsStopReceiver (JNIEnv* _env, jobject _thiz)
{

/*
* Stop the RTP receiver
*/
ngmslib_close(&ngmsReceiver);

return 0;
}

/**
* NGMS Callback operation which blocks until a complete video frame has been received and decoded
*/
jint Java_com_example_ngmsclient_ExampleChat_ngmsReceiveFrame (JNIEnv* _env, jobject _thiz, jbyteArray frameBytes)
{
jstring str;

jboolean copy;
jbyte* buffer = (*_env)->GetByteArrayElements(_env, frameBytes, ©);
jsize max = (*_env)->GetArrayLength(_env, frameBytes);
int frameSize;

frameSize = ngmslib_readVidFrame(&ngmsReceiver, buffer, max, NULL);

(*_env)->ReleaseByteArrayElements(_env, frameBytes, buffer, JNI_ABORT);

return frameSize;

}

/**
* Starts the NGMS streaming service
*/
jint Java_com_example_ngmsclient_ExampleChat_ngmsStartSender (JNIEnv* _env, jobject _thiz, jstring remoteAddress)
{

char xcoderConfig[512];
char filter[512];
char destination[512];
const char *p;
int i;

/*
* Before doing anything call ngmslib_open to open the NGMS service
*/
i = ngmslib_open(&ngmsSender);

if (i!= NGMS_RC_OK)
{
__android_log_print(ANDROID_LOG_ERROR, "ngms", "ngmslib_open failed with error %d", i);
return -1;
}

/*
* Get the destination host string
*/
p = (*_env)->GetStringUTFChars(_env, remoteAddress, 0);

sprintf(destination, "rtp://%s:5004", p);

(*_env)->ReleaseStringUTFChars(_env, remoteAddress, p);

/*
* Customize the stream configuration parameters
*/
ngmsSender.output = destination;
sprintf(xcoderConfig, "vc=h264,vp=66,vb=250,vx=320,vy=240,vgmax=2000,vgmin=1500,vfr=15,vcfrout=-1,vth=1,vl=1,vsc=1,vf=2");
ngmsSender.strxcode = xcoderConfig;
sprintf(filter, "type=yuv420sp, ngmsSender.strfilters[0] = filter;
ngmsSender.inputs[0]= "/dev/dummyvideo";
ngmsSender.noaud = 1;

/*
* Start the RTP sender
*/
i = ngmslib_stream(&ngmsSender);

if (i!= NGMS_RC_OK)
{
__android_log_print(ANDROID_LOG_ERROR, "ngms", "ngmslib_stream failed with error %d", i);
ngmslib_close(&ngmsSender);
return -1;
}

return 0;
}

/**
* Stops the NGMS streaming service
*/
jint Java_com_example_ngmsclient_ExampleChat_ngmsStopSender (JNIEnv* _env, jobject _thiz)
{

/*
* Stop the RTP sender
*/
ngmslib_close(&ngmsSender);

return 0;
}

/**
* Encodes and transmits a single video frame using local RTP transport
*/
jint Java_com_example_ngmsclient_ExampleChat_ngmsTransmitFrame (JNIEnv* _env, jobject _thiz, jbyteArray frameBytes)
{

jboolean copy;
jbyte* buffer = (*_env)->GetByteArrayElements(_env, frameBytes, ©);
jsize size = (*_env)->GetArrayLength(_env, frameBytes);
int i;

i = ngmslib_onVidFrame(&ngmsSender, buffer, size);

(*_env)->ReleaseByteArrayElements(_env, frameBytes, buffer, JNI_ABORT);

return i;
}

We will create the Android.mk makefile that is used by the Android NDK to build our ngmsglue library.


#Example Android.mk Makefile

LOCAL_PATH:= $(call my-dir)

include $(CLEAR_VARS)

LOCAL_MODULE:= ngmsglue
LOCAL_SRC_FILES:= ngmsglue.c
LOCAL_C_INCLUDES:= ngms/include
LOCAL_LDFLAGS:= -Lngms/lib
LOCAL_LDLIBS:= -llog -lm -lngms -lxcode

include $(BUILD_SHARED_LIBRARY)

To compile the ngmsglue.so shared library just do the following.


cd [YOUR NGMSCLIENT PROJECT DIR]/jni
${ANDROID_NDK_HOME}/ndk-build

This should produce the output file. [YOUR NGMSCLIENT PROJECT DIR]/libs/armeabi/ibngmsglue.so

You may need to manually copy the ngms/lib/libngms.so and ngms/lib/libxcode.so libraries to the libs/armeabi output directory because the contents of this directory will be automatically packaged into the project.apk application.

The Java Application Layer

The Java Application Layer consists of the application level logic for the Android application we are building. In order to use the native code from the previous section, we will need to create a simple app using the Android SDK. This example demonstrates the use of the ExampleChat class which can be used by your own application to provide real-time, interactive video streaming. It doesn’t show all the steps required to create a complete Android video streaming app. An intermediate level Android developer should be able to complete the task by building on the ExampleChat class and including it in their full application.

The ExampleChat class shown below is used to control the NGMS stream sender and the NGMS stream receiver. The startSender method can be called to start streaming video from the native camera using the camera’s preview mechanism. Each frame of video output from the camera is transmitted to the NGMS API, where it will be encoded and transmitted to the remote party using native RTP encapsulation. The starReceiver method can be called to start receiving video from a remote instance of the same application. startReceiver starts a thread that performs a lock poll for the next available received and decoded video frame. Together, these two methods can be used to establish a two-way video stream between two remote instances of the same application over a network.


/*
* ExampleChat.java
*
*/

package com.example.ngmsclient;

import android.hardware.Camera;
import android.hardware.Camera.Parameters;
import android.hardware.Camera.PreviewCallback;
import android.graphics.ImageFormat;

public class ExampleChat
{
private boolean myReceiverRunning;
private Camera myCamera;
private PreviewCallback myCameraCallback;

/**
* Change this value to the destination where the video will be streamed
*/
private static final DESTINATION = "10.0.0.1";

/**
* method to initialize and start sending video from the local camera
*/
public void startSender () throws Exception
{

myCamera = Camera.open();

Parameters parameters = myCamera.getParameters();
parameters.setPreviewFormat(ImageFormat.NV21);
parameters.setPreviewFrameRate(15);
parameters.setPreviewSize(320, 240);

myCamera.setParameters(parameters);

/**
* Call the NGMS native method to perform streaming initialization
*/
ngmsStartSender(DESTINATION);

myCameraCallback = new PreviewCallback()
{
public void onPreviewFrame(byte[] frameBytes, Camera camera)
{
/**
* Call the NGMS native routine to encode and transmit the video frame
* to the remote destination using RTP transport.
*/
ngmsTransmitFrame(frameBytes);
}
}

myCamera.startPreview();
myCamera.setPreviewCallback(myCameraCallback);

}

/**
* method to stop sending video
*/
public void stopSender () throws Exception
{
ngmsStopSender();
}

/**
* method to start receiving remote video
*/
public void startReceiver () throws Exception
{

/**
* Call the NGMS native method to perform input initialization
*/
ngmsStartReceiver();

/**
* Start the thread which will poll for received video frames
*/
new Thread(myVideoFrameReceiver).start();

}

/**
* method to stop receiving remote video
*/
public void stopReceiver () throws Exception
{
myReceiverRunning = false;
ngmsStopReceiver();
}

/**
* Thread used to check for decoded video frames
* received by the NGMS native layer
*/
private Runnable myVideoFrameReceiver = new Runnable()
{
public void run()
{
/**
* Allocate a buffer to hold a video frame 320 x 240 pixels
* in RGB565 format, taking 16 bits (2 bytes) per pixel.
*/
byte[] frameBytes = new byte[ 320 * 240 * 2 ];
int size;
myReceiverRunning = true;

while(myReceiverRunning)
{
/**
* Block until a frame has been received and decoded by the native layer
*/
size = ngmsReceiveFrame(frameBytes);
if(size > 0)
{
/**
* The frameBytes array represents a video frame in RGB565 format.
* These bytes can be converted into a Bitmap object for display
* on the screen.
*/
}
}
}
}

/**
* Static initialization to load native libraries into address space
*/
static
{
System.loadLibrary("xcode");
System.loadLibrary("ngms");
System.loadLibrary("ngmsglue");

}

/**
* Define native method prototypes
*/
public native int ngmsStartReceiver ();
public native int ngmsStopReceiver ();
public native int ngmsReceiveFrame (byte[] frameBytes);

public native int ngmsStartSender (String remoteAddress);
public native int ngmsStopSender ();
public native int ngmsTransmitFrame (byte[] frameBytes);

}

Once you have integrated the ExampleChat class into your own Android project, make sure that all three native libraries exist in the project’s /libs directory so that they are included in the output of your .apk file.


libs/armeabi/libngms.so
libs/armeabi/libxcode.so
libs/armeabi/libngmsglue.so

This produces a sample video output to the destination IP address 10.0.0.1. To successfully allow two clients to stream video bidirectionally, you will need to change this IP address to match your remote client.

conclusion

The example above demonstrates how to stream live video from the camera object in real time and how to receive a remote live stream and extract each frame of decoded video for display. The same process flow can be applied to adding streaming audio to create a complete video chat application. 


To view the Nex Gen Media Server (NGMS) Embedded API Guide, go to http://ngmsvid.com/develop.php

Leave a Reply

Your email address will not be published. Required fields are marked *