End Point

News

Welcome to End Point's blog

Ongoing observations by End Point people.

Streaming Live with Red5 Media Server:Two-Way

I already wrote about the basics of publishing and broadcasting with Red5 Media Server. Let's fast forward to the advanced topics and create a video conference now!

Getting Ready

First, a word about the technology stack: a little bit of Java6/Java EE will be used for the server-side work (Red5 is written in Java), ActionScript2/Adobe Flash CS6 will be the primary tool for the client side development, and OS X Mountain Lion is my operating system.

Red5 Server comes with the set of sample applications that provide the source code for about everything you may want to achieve. The primary challenge is to unleash the power of it, since the samples fall extremely short of documentation! The "fitcDemo" application will serve as a base for all our customization.

Originally I made all the development in Red5 RC 1.0 version where fitcDemo was present. Unfortunately, when I downloaded the latest Red5 1.0.1 release yesterday it was simply not there! The source code was still in the repo, just outdated and not working. Well, I did all the work for Red5 team, so you can just download fitcDemo.war from my repo and drop it into the "webapps" directory of Red5 1.0.1 installation - and you are good!

You will then find the video conference demo at http://localhost:5080/demos/videoConference.html

Here is how it looks out of the box:

Here is how we want it to look!

The goal is to make our Red5 conference look as neat as Google Hangout.

Sleek Subscribers

Default video conference has five subscribers statically positioned on the stage. It's way more fun to have the subscribers added and removed on the fly as they connect to the server. So let's do that! I have the complete tutorial code based on Red5 1.0.1 The final version is in my Github repo, so I will be explaining parts of it further. Open videoConference.fla in Flash. I used CS6 for all the FLA/ActionScript editing in the tutorial.


Edit the VideoConference clip in the Library, to remove everything in it and add a ScrollPane with the following properties:


Edit the VideoPool clip, to remove everything in it as well and drag a Broadcaster clip to the top left corner:


Broadcaster and Subscriber clips should be modified too and have only the video component visible. You may look into the complete source code for details. Don't forget to change the Publish Settings to Flash Player 8; ActionScript2 and publish.

Modify Connector.as to point to the Red5 server. It is important to specify the IP address of the host machine rather than just "localhost", so other computers can join the hangout over network.

public static var red5URI:String = "rtmp://192.168.0.5/fitcDemo";

Make a slight change to configUI() to make it auto connect:

public function configUI():Void 
 {
  connection  = new Connection();
        
  connection.addEventListener("success", Delegate.create(this, onComplete));
  connection.addEventListener("onSetID", this);
  
  connection.addEventListener("newStream", this);
  connection.addEventListener("close", Delegate.create(this, onComplete));
  connected = connection.connect(red5URI, getTimer());
  dispatchEvent({type:"connectionChange", connected: connected});
 }
 
 private function onComplete(evtObj:Object):Void
 {
  dispatchEvent({type:"connectionChange", connected: evtObj.connected});
 }

Get rid of all the default Subscriber variables in the class and change configUI():

private function configUI():Void 
{
  subscriberList = [];
}

Open VideoPool.as and modify getVideoContainer() so that every subscriber will be dynamically created and positioned on stage. The row will have 4 streams with the broadcaster stream (your computer's camera) being the first in the first row, and more rows will be created in the scroll pane to accommodate more streams.

private function getVideoContainer(p_id:Number):Subscriber
{  
  var d:Number = 1;
  if (subscriberList.length > 0) {
    d = subscriberList[subscriberList.length - 1].getDepth() + 1;
  }
  
  var positionX:Number = broadcaster._x + ((subscriberList.length + 1)%4) * broadcaster._width;
  var positionY:Number = broadcaster._y + (broadcaster._height * Math.floor((subscriberList.length + 1)/4));
  
  attachMovie("org.red5.samples.livestream.videoconference.Subscriber", "subscriber_" + p_id, d, {_x:positionX, _y:positionY});
  
  var s:Subscriber = _level0.instance1.scrollPane.content["subscriber_" + p_id];
  subscriberList.push(s);
  return s;
}

The new Subscriber instance for the particular stream will be generated when subscribe() method is called.

public function subscribe(p_id:Number):Void
{
  if(p_id == "undefined" || isNaN(p_id) || p_id == "") return;
  
  var s:Subscriber = getVideoContainer(p_id);
  
  s.subscribe("videoStream_" + p_id, connection);  
}

If the subscriber is added, at some point it may need to be removed! Add removeSubscriber() function that will delete the disconnected subscriber and reposition all the other subscribers and invalidate ScrollPane.

public function removeSubscriber(s:Subscriber): Void {
  for(var i= 0; i < subscriberList.length; i++) { 
    if (Subscriber(subscriberList[i]).videoStream == s.videoStream) {
      subscriberList.removeItemAt(i);
    }
  } 
  for (var i= 0; i < subscriberList.length; i++) {
    var positionX:Number = broadcaster._x + (subscriberList.length%4) * broadcaster._width;
    var positionY:Number = broadcaster._y + (broadcaster._height * Math.floor(subscriberList.length/4));
    var mc:Subscriber = Subscriber(subscriberList[i]); 
    mc._x = positionX;
    mc._y = positionY;
  }
  s.removeMovieClip();
  _level0.instance1.scrollPane.invalidate();
}

Look into Subscriber.as. When the subscriber disconnects the stream fires an "unpublishNotify" event, that eventually makes a call to the removeSubscriber() function.

public function subscribe(p_subscriptionID:String, p_connection:Connection):Void 
{
  ...
  stream.addEventListener("unpublishNotify", Delegate.create(this, streamStop));
  ...
}

public function streamStop(evtObj:Object):Void
{
  videoPool.removeSubscriber(this);
  publish_video.clear();
  stream.close();
}

Finally open VideoConference.as. This is a controller class, the Red Queen of all the above classes! It manages all the incoming subscribing streams and the broadcasting stream. When your computer is ready to broadcast, the camera is up and the broadcasting stream received its id, VideoConference sends requests to get all the publishing streams to the url on the server and process them as the subscribers.

private function configUI():Void 
{
  ...
  videoPool.broadcaster.addEventListener("onSetID", this);
  ...
}

private function setID(p_id:Number, p_connection:Connection):Void
{
  //set local videoID
  videoID = Number(p_id);

  // set connection
  connection = p_connection;

  getStreams();
}

private function getStreams():Void
{
  connection.call("getStreams", this.result);
}

Since VideoPool with all the streams loads into the ScrollPane component now, VideoConference needs to be updated with the correct VideoPool reference:

private var videoPool:VideoPool;
private var scrollPane:ScrollPane;
...
private function configUI():Void 
{
  ... 
  scrollPane.setStyle("borderStyle", "none");
  videoPool = VideoPool(scrollPane.content); 
  
  videoPool.broadcaster.registerController(this);
  videoPool.broadcaster.addEventListener("connected", this);
  videoPool.broadcaster.addEventListener("disconnected", this);
  videoPool.broadcaster.addEventListener("onSetID", this);
}

We could wrap the last four lines in some kind of configUI() function on VideoPool, but haven't I told you? Refactoring will be your homework assignment!

When the new subscriber shows up, we add him or her and update ScrollPane.

private function processQue():Void
{
  if(streamQue.length <= 0) 
  {
    clearInterval(si);
    return;
  }
  
  var id:Number = Number(streamQue.shift().split("_")[1]);

  videoPool.subscribe(id);

  scrollPane.invalidate();
}

Almost ready! Publish the swf file and copy to the website folder. If you want to re-compile ActionScript classes only, use MTASC compiler and MX libraries from my Github repo:

> cp videoConference.swf classes/videoConference.swf
> cd classes
> export PATH="$HOME/mtasc-mx/bin:$HOME/mtasc-mx/bin/std:$PATH"
> mtasc -version 8 -cp "." -swf videoConference.swf -mx org/red5/samples/livestream/videoConference/videoConference.as -v
> cp videoConference.swf $HOME/Sites/red5-hangout/videoConference.swf

Include into the webpage:


Stylish Spotlight

The "Spotlight" component is the larger video of the "talking" person that shows up when one of the smaller previews is clicked. simpleSubscriber.fla is the ideal base for this component:

Subscriber and Broadcaster will respond to the "click" event and call JavaScript function with the video stream name as a parameter. The video stream name is just a string like "videoStream_12" denoting the 12th stream accepted by "fitcDemo" application.

In Broadcaster.as:

private function configUI():Void 
{   
  this.onRelease = function(){
    _global.tt('Broadcaster ' + this.videoStream + ' clicked.'); 
    if (ExternalInterface.available) {
      ExternalInterface.call("spotlight", this.videoStream);
    }
  }
} 

In Subscriber.as

public function configUI():Void 
{
  this.onRelease = function() {
    ExternalInterface.call("spotlight",this.videoStream);
  }
};

JavaScript will embed the SimpleSubscriber clip and pass along the video stream name that should be played via flashvars.

On the page


In SimpleSubscriber Main.as:

private var streamName:String;
...
private function configUI():Void 
  { 
    streamName = String(_level0.streamName); 
    ... 
  }
private function connectionChange(evtObj:Object):Void
{  
  if(evtObj.connected) 
  {
   
    stream = new Stream(evtObj.connection);
   
    stream.play(streamName, -1);
   
    publish_video.attachVideo(stream); 
  }
}

Publish simpleSubsciber.swf and place it into the website folder. To recompile the ActionScript part only:

> mtasc -version 8 -cp "." -swf simpleSubscriber.swf -mx org/red5/samples/livestream/subscriber/Main.as -v
> cp simpleSubscriber.swf $HOME/Sites/red5-hangout/simpleSubscriber.swf

By the way the -version 8 flag for mtasc was added specifically to compile ExternalInterface, otherwise, these libraries would not be found.

Chat in the absence of Sound

One thing I really appreciate about the Google Hangout architecture is that it does not just make the whole page a bulky <embed> or <object> and lock everything into the external component. I love how they use the familiar and friendly JavaScript and HTML to add some interactive features. That's why I decided to break the single VideoConference component into pieces as well and tie them together on a web page.

The original conference had its audio support commented out with the note about performance issues. Bummer! This will surely need to be addressed, but for now I decided to use the existing chat and make it separate. Create a blank document chat.fla and drag Chat clip from the VideoConference Library to the stage.

Remove any reference to VideoConference and VideoPool from Chat.as, create its own GlobalObject and Connection and paste the sound settings from VideoConference.as:

private var red5URI = "rtmp://192.168.0.5/fitcDemo";
...
public function configUI():Void 
{  
  streamID = Number(_level0.streamName.split("_")[1]);
    
  ...
  
  loadProfile("videoConference");
  
  var my_nc:Connection = new Connection();
  my_nc.connect(red5URI);

  chatID = "videoConferenceChat";
  connected = so.connect(chatID, my_nc, false);
  
  ...

  sndTarget = this.createEmptyMovieClip("sndTarget", this.getNextHighestDepth());
  snd = new Sound (sndTarget);
  snd.attachSound("newChatMessage");
  snd.setVolume(80);
   
  soundPlay._visible = false;
  soundMute.addEventListener("click", Delegate.create(this, updateMute));
  soundPlay.addEventListener("click", Delegate.create(this, updateMute));
  soundMute.tooltip = "Mute new chat sound";
  soundPlay.tooltip = "Un-mute new chat sound";
}

Initialize the chat in Broadcaster.as after the stream is initialized:

private function onSetID(evtObj:Object):Void
{
  ...
  ExternalInterface.call("chat", this.videoStream);
} 

Publish or recompile:

> mtasc -version 8 -cp "." -swf chat.swf -mx org/red5/samples/livestream/videoConference/Chat.as -v
> cp chat.swf $HOME/Sites/red5-hangout/chat.swf

Add the chat to the web page:


<div id="chat" style="display:none">
</div>

And enjoy chatting with yourself for a while in different browser tabs:

Final result

After a bit of styling and herding my test participants, here is what I got:

Questions, questions

There are problems to be addressed yet.

First, there is no audio. It's commented out in the original code with the note about performance issues. Silence may be golden, but the grimace-enhanced chat experience is totally not acceptable for production!

Second, the whole performance talk brings up the other important questions: How many subscribers can this setup handle? Is it possible to achieve the better video quality? Why is the video stream choppy at times?

Finally, every application needs a mobile presence. Will it be possible to port the client to mobile devices using AIR? How to port to iPads and iPhones?

The flash source code in its entirety is located in my Github repo. Please, keep in mind, that the code is not perfect, and the logic can be better organized between classes with event handlers. The demo application can be found here. MTASC compiler and MX libraries for compilation are available here.

I would love to hear your feedback on this!

12 comments:

Anonymous said...

Hello Marina! I want thanks you for your work)

you see, i must pass my diploma in this year, and summary of my diploma is :'Video chat using RoR+REd5'

can you tell me how i can integrate Red5 in RoR application?
Thanks a lot)

Anonymous said...

Hi,

What is the relation between videoconference.fla and the classes you modified. You have not used this file during the compilation.

Can I develop a flash builder project for the videoconference application using the classes available in videoconference.

Tinku Pandey said...

Hi,

First of all, I landed in your blog after googling for "red5 two way". I was looking for quality tuts for red5 and when asking for help in StackOverflow, I found nothing but cynicism. I read both your posts on red5. And regarding audio you can use Xuggler to do stream merges, I found the following link: https://groups.google.com/forum/#!msg/xuggler-users/s6zUCwYy98k/5J-YIkCSaEIJ.

Had a very bad time with setting up red5 until I stopped working on it for a few months.

The bug I posted got accepted in red5 main page but never saw any action.
https://code.google.com/p/red5/issues/detail?id=350

Cant thank you enough. I will surely update you with my progress and share some code as well :)

Keep writing!

Gaurav said...

Hello,

Thanks for your work.

I am using red5 included simpleBroadcaster and simpleSubscriber sample in one of my project.

I am getting a voice lag. If I speak in mic then the subscriber get voice after 1-3 seconds.

Can you help me what could be the reason for this ?

I am using Red5 1.0.1 final.

Nguyen Quoc Doan said...
This comment has been removed by the author.
Nguyen Quoc Doan said...

hi Marina!
can you uploadd the video revording all works should be done! there is something i can't follow.
Thanks a lot.

Williams said...

Hi, I have a simple question. what software I need to send signal tv live, any live videos, from computer studio tv to red5 server?. In others words, live streaming with source signal in another computer.

Dodla Sreekanth said...

Hello Marina!
we are developing video conference website. i use your code its not working can you please help me.

Advanced thanks

Денис Викторович said...

Good day Marina!
I want to work together to offer the idea of a random selection of party, in the likeness of Chatroulette.
my mail denver00700@mal.ru
Waiting for an answer.

Adolfo Exposito said...

Hi Marina!
It's a good new to have a fitcDemo application that can be used on Red5 1.0.1.
How can I install it on my Red5?

Thanks for your time.
adolfo.exposito@calculosdigitales.com

K-Funk (DJ/Producer/Webmaster/Creative Mind) said...

Hello Marina - this is really cool! I recently installed Red5 on a CentOS server, with the goal of streaming my DJ sessions live on my web page. I've got all the components I need installed on my server - JDK, Red5, JWplayer - and Adobe Flash Media Live Encoder to broadcast - however, due to my noobishness, I am not able to get a connection to my server thru FMLE - I'm looking for a pro such as yourself to help finish the job, and I would of course pay you for your time - is this something that would interest you? If so, please contact me at 'djkfunk' AT 'gmail' DOT 'com'. Thanks, Kevin.

Ingrid Sedé said...

Can you please help me out with the workflow?
I've downloaded all of the files from https://github.com/ftert/red5-flash . I opened the videoconference.fla that has the new elements on stage (your file that is). The next step would be to alter the as files according to the tutorial in the "red5-flash\trunk\classes\org\red5\samples\livestream\videoconference" folder? Those files are the classes that are referenced from within the videoconference.fla? So by publishing the .fla the swf will hold the changes? I should then use the mtasc utility to make it all compatible with red5? Please help me couse i'm getting crazy over this. :-)