Re: [Ros-kinect] multiple kinects working

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Ros-kinect] multiple kinects working

rusu
Administrator
Nink,

Interesting - ok I have to admit that I haven't tried 3 kinects yet on my system. I think that each of them might
require a separate USB controller, as we had issues with 2 on the same controller earlier today. Can you make sure
they're plugged in different controllers?

Cheers,
Radu.


On 11/19/2010 05:41 PM, Nink wrote:

> Radu. I can't seem to get the third kinect to start. I am using the latest ROS build and on seperate bus for each one.  I have tried plugging in one at a time, different orders etc but 3rd kinect still crashes each time. It sees the third kinect tries to initialize and that's about it.
>
> Any advice appreciated
> Sent from my BlackBerry
>
> -----Original Message-----
> From: Radu Bogdan Rusu<[hidden email]>
> Sender: [hidden email]
> Date: Fri, 19 Nov 2010 16:15:56
> To:<[hidden email]>
> Reply-To: [hidden email]
> Cc: Aurel W.<[hidden email]>
> Subject: Re: multiple kinects working
>
> Again, in our tests the interference is minimal (if any) with 2 kinects. We'll post a video up later to show this in
> case people still have doubts ;) We got about 10-20 of them by now in the building, so we might try a crazy 20-Kinect
> registration experiment ;)
>
> Cheers,
> Radu.
>
>
> On 11/19/2010 03:01 PM, Aurel W. wrote:
>> Another big issue, were interference is quite heavy, is when you face
>> two kinects in a way, that one ir camera has another laser projector
>> in the field of view. This results in some lens flare. So I guess
>> adding shutters and projector modulation may still be necessary for
>> good results in multi kinect setups.
>>
>> On 19 November 2010 23:54, Nink<[hidden email]>   wrote:
>>> What I find real intersting put two kinects on top of each other and cover the infared for one but leave camera uncovered.  Black screen.   They know what signal they sent out.  There is no issue the folks at primesence were smart.
>>>
>>> I have a quad core i7 running now and its humming nicely with 2 kinects.  Trying to add third I have some pci cards I am playing with at the moment to get the third one up.  Goal is 6.  Peaks at 50 percent then idles at 20 percent.  Ill tweet a pic on @nink.
>>> Sent from my BlackBerry
>>>
>>> -----Original Message-----
>>> From: "Aurel W."<[hidden email]>
>>> Sender: [hidden email]
>>> Date: Fri, 19 Nov 2010 23:40:22
>>> To:<[hidden email]>
>>> Reply-To: [hidden email]
>>> Subject: multiple kinects working
>>>
>>> Hello,
>>>
>>> in the last days there has been quite some discussion about using
>>> multiple kinects. Main reason is, we could get rid of occlusions and
>>> really use it for full real time 3d reconstruction. A first starting
>>> point would be to use an stereo setup.
>>>
>>> Sorry for this long post, but I kept working on this like crazy in the
>>> past two days so I have quite some stuff to say, hope it turns out
>>> useful to you. Interesting stuff on the bottom ;)
>>>
>>> When you think of, how the depth sensor works one issue comes
>>> immediately to your mind, A structured light pattern is projecteder in
>>> IR, while the viewing frustum of the projector has some slight offset
>>> to the one of an ir camera, but they are axis aligned. From just this
>>> it is possible to reconstruct depth. However, if you would use
>>> multiple kinects, there could be interferences, when the structured
>>> light is visible for the other kinect or when the laser projector is
>>> directly visible without any reflection. So it was common sens, that
>>> multiple kinects won't work by now.
>>>
>>> Some ideas to address this issue were stated, not all of them are
>>> practical. Some comments on those ideas:
>>>
>>> * Time multiplexing with 1/30s time slots. This basically means that
>>> in a stereo setup, you would subsequently capture one frame at one
>>> kinect and disable the laser projector on the other kinect. First keep
>>> in mind that all devices are running asynchronously. Depending on the
>>> exposure time for the ir-camera, one frame from one kinect can
>>> "overlap" two frames in the other kinect. In the worst cast of 1/30s
>>> exposure time, we would need to wait for an entire frame, before
>>> starting to capture. This means, in a stereo setup we would drop from
>>> the orifinal 30fps, to 7.5fps. lame. In addition, we don't know yet,
>>> if the device allows us to enable and disable the ir projector frame
>>> by frame. If not, someone would need to do this by altering hardware
>>> and switching the diode separately with a micro controler (or the
>>> built in motor controler, if you want to be clever :)
>>>
>>> * Frequency multiplexing. Using different narrow band ir lasers and
>>> filters. This would definitely work, if you get the optics right. Tho
>>> quite complicated imho.
>>>
>>> * Polarization. Won't work. A reflection on an arbitrary surface won't
>>> preserve polarization well enough. The reason why polarization
>>> projection screens for e.g. work, is because they use a special
>>> coating. This won't work on human skin or other materials tho.
>>>
>>> Because all this ideas have some sort of issue I tried something
>>> different. Normally someone would use synchronized devices, very short
>>> exposure times and time multiplexing. But that won't be possible with
>>> the kinect. My approach now was to add separate shutters to both, the
>>> camera and projector and to do time multiplexing within one single
>>> frame of the ir camera. This can be done with a high speed shutter.
>>>
>>> For the ir-projector the best approach would be to modulate the laser
>>> diode accordingly. For the camera, some sort of mechanical or lcd
>>> shutter is needed. Shutters would be added to both kinects and the
>>> shutters have to be synchronized of course, while the ir-cameras of
>>> the kinects are running asynchronously. Of course, the exposure time
>>> of one single frame is shorter now.
>>>
>>> To find out, how the depth sensor would behave, if i add a shutter, i
>>> made different experiments.This is only with one kinect by now. One
>>> setup looked like this:
>>>
>>> http://atommuell.mum.jku.at/~aurel/kinect_roling_shuter_proto_2.jpg
>>>
>>> The depth sensor gives slightly more errors here, because of the short
>>> exposure time, but in general the sensor seems to be pretty robust
>>> when doing such things as adding rolling shutters.
>>>
>>> The next step was to get a second kinect and to build an actual
>>> prototype, where both sensors are stereo separated by shutters.
>>> However, I was quite surprised when i pointed two plain kinects on the
>>> same object and started capturing. My intention was, and as it was
>>> discussed all around the net by now, that things would be completely
>>> screwed up. Both sensors were working quite good with some added
>>> errors. See for yourself:
>>>
>>> http://atommuell.mum.jku.at/~aurel/two_kinects_same_direction_c_both.png
>>>
>>> Compared to using just one kinect:
>>>
>>> http://atommuell.mum.jku.at/~aurel/two_kinects_same_direction_c_single.png
>>>
>>> Interference is imho very low, much lower than expected. Also an
>>> interesting test, to lower the luminance of one of the projects to
>>> 50%:
>>>
>>> http://atommuell.mum.jku.at/~aurel/two_kinects_same_direction_c_both_one_proj_shutter.png
>>>
>>> I don't think that time multiplexing is completely of the table now,
>>> but multiple kinects are working way better than expected, so maybe
>>> the hardware could be used without any modification in some cases. We
>>> also would have to see about interference when using more than two
>>> kinects.
>>>
>>> that's it for now,
>>>
>>> aurel
>>>
_______________________________________________
Ros-kinect mailing list
[hidden email]
https://code.ros.org/mailman/listinfo/ros-kinect
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Ros-kinect] multiple kinects working

Nink
I purchased 2 pci usb cards and Libusb tells me all on seperate bus.  Joshb had a great idea send initiation code to each card before we start collecting data to make sure we can at least activate the ir projector.  

 If someone can do me a big favour and just create a cut down version of ROS libfreenect that just sends the inital request to activate the kinect would be great.

Thanks
Sent from my BlackBerry

-----Original Message-----
From: Radu Bogdan Rusu <[hidden email]>
Date: Fri, 19 Nov 2010 21:14:23
To: Nink<[hidden email]>
Reply-To: Radu Bogdan Rusu <[hidden email]>
Cc: ros-kinect<[hidden email]>; [hidden email]<[hidden email]>
Subject: Re: multiple kinects working

Nink,

Interesting - ok I have to admit that I haven't tried 3 kinects yet on my system. I think that each of them might
require a separate USB controller, as we had issues with 2 on the same controller earlier today. Can you make sure
they're plugged in different controllers?

Cheers,
Radu.


On 11/19/2010 05:41 PM, Nink wrote:

> Radu. I can't seem to get the third kinect to start. I am using the latest ROS build and on seperate bus for each one.  I have tried plugging in one at a time, different orders etc but 3rd kinect still crashes each time. It sees the third kinect tries to initialize and that's about it.
>
> Any advice appreciated
> Sent from my BlackBerry
>
> -----Original Message-----
> From: Radu Bogdan Rusu<[hidden email]>
> Sender: [hidden email]
> Date: Fri, 19 Nov 2010 16:15:56
> To:<[hidden email]>
> Reply-To: [hidden email]
> Cc: Aurel W.<[hidden email]>
> Subject: Re: multiple kinects working
>
> Again, in our tests the interference is minimal (if any) with 2 kinects. We'll post a video up later to show this in
> case people still have doubts ;) We got about 10-20 of them by now in the building, so we might try a crazy 20-Kinect
> registration experiment ;)
>
> Cheers,
> Radu.
>
>
> On 11/19/2010 03:01 PM, Aurel W. wrote:
>> Another big issue, were interference is quite heavy, is when you face
>> two kinects in a way, that one ir camera has another laser projector
>> in the field of view. This results in some lens flare. So I guess
>> adding shutters and projector modulation may still be necessary for
>> good results in multi kinect setups.
>>
>> On 19 November 2010 23:54, Nink<[hidden email]>   wrote:
>>> What I find real intersting put two kinects on top of each other and cover the infared for one but leave camera uncovered.  Black screen.   They know what signal they sent out.  There is no issue the folks at primesence were smart.
>>>
>>> I have a quad core i7 running now and its humming nicely with 2 kinects.  Trying to add third I have some pci cards I am playing with at the moment to get the third one up.  Goal is 6.  Peaks at 50 percent then idles at 20 percent.  Ill tweet a pic on @nink.
>>> Sent from my BlackBerry
>>>
>>> -----Original Message-----
>>> From: "Aurel W."<[hidden email]>
>>> Sender: [hidden email]
>>> Date: Fri, 19 Nov 2010 23:40:22
>>> To:<[hidden email]>
>>> Reply-To: [hidden email]
>>> Subject: multiple kinects working
>>>
>>> Hello,
>>>
>>> in the last days there has been quite some discussion about using
>>> multiple kinects. Main reason is, we could get rid of occlusions and
>>> really use it for full real time 3d reconstruction. A first starting
>>> point would be to use an stereo setup.
>>>
>>> Sorry for this long post, but I kept working on this like crazy in the
>>> past two days so I have quite some stuff to say, hope it turns out
>>> useful to you. Interesting stuff on the bottom ;)
>>>
>>> When you think of, how the depth sensor works one issue comes
>>> immediately to your mind, A structured light pattern is projecteder in
>>> IR, while the viewing frustum of the projector has some slight offset
>>> to the one of an ir camera, but they are axis aligned. From just this
>>> it is possible to reconstruct depth. However, if you would use
>>> multiple kinects, there could be interferences, when the structured
>>> light is visible for the other kinect or when the laser projector is
>>> directly visible without any reflection. So it was common sens, that
>>> multiple kinects won't work by now.
>>>
>>> Some ideas to address this issue were stated, not all of them are
>>> practical. Some comments on those ideas:
>>>
>>> * Time multiplexing with 1/30s time slots. This basically means that
>>> in a stereo setup, you would subsequently capture one frame at one
>>> kinect and disable the laser projector on the other kinect. First keep
>>> in mind that all devices are running asynchronously. Depending on the
>>> exposure time for the ir-camera, one frame from one kinect can
>>> "overlap" two frames in the other kinect. In the worst cast of 1/30s
>>> exposure time, we would need to wait for an entire frame, before
>>> starting to capture. This means, in a stereo setup we would drop from
>>> the orifinal 30fps, to 7.5fps. lame. In addition, we don't know yet,
>>> if the device allows us to enable and disable the ir projector frame
>>> by frame. If not, someone would need to do this by altering hardware
>>> and switching the diode separately with a micro controler (or the
>>> built in motor controler, if you want to be clever :)
>>>
>>> * Frequency multiplexing. Using different narrow band ir lasers and
>>> filters. This would definitely work, if you get the optics right. Tho
>>> quite complicated imho.
>>>
>>> * Polarization. Won't work. A reflection on an arbitrary surface won't
>>> preserve polarization well enough. The reason why polarization
>>> projection screens for e.g. work, is because they use a special
>>> coating. This won't work on human skin or other materials tho.
>>>
>>> Because all this ideas have some sort of issue I tried something
>>> different. Normally someone would use synchronized devices, very short
>>> exposure times and time multiplexing. But that won't be possible with
>>> the kinect. My approach now was to add separate shutters to both, the
>>> camera and projector and to do time multiplexing within one single
>>> frame of the ir camera. This can be done with a high speed shutter.
>>>
>>> For the ir-projector the best approach would be to modulate the laser
>>> diode accordingly. For the camera, some sort of mechanical or lcd
>>> shutter is needed. Shutters would be added to both kinects and the
>>> shutters have to be synchronized of course, while the ir-cameras of
>>> the kinects are running asynchronously. Of course, the exposure time
>>> of one single frame is shorter now.
>>>
>>> To find out, how the depth sensor would behave, if i add a shutter, i
>>> made different experiments.This is only with one kinect by now. One
>>> setup looked like this:
>>>
>>> http://atommuell.mum.jku.at/~aurel/kinect_roling_shuter_proto_2.jpg
>>>
>>> The depth sensor gives slightly more errors here, because of the short
>>> exposure time, but in general the sensor seems to be pretty robust
>>> when doing such things as adding rolling shutters.
>>>
>>> The next step was to get a second kinect and to build an actual
>>> prototype, where both sensors are stereo separated by shutters.
>>> However, I was quite surprised when i pointed two plain kinects on the
>>> same object and started capturing. My intention was, and as it was
>>> discussed all around the net by now, that things would be completely
>>> screwed up. Both sensors were working quite good with some added
>>> errors. See for yourself:
>>>
>>> http://atommuell.mum.jku.at/~aurel/two_kinects_same_direction_c_both.png
>>>
>>> Compared to using just one kinect:
>>>
>>> http://atommuell.mum.jku.at/~aurel/two_kinects_same_direction_c_single.png
>>>
>>> Interference is imho very low, much lower than expected. Also an
>>> interesting test, to lower the luminance of one of the projects to
>>> 50%:
>>>
>>> http://atommuell.mum.jku.at/~aurel/two_kinects_same_direction_c_both_one_proj_shutter.png
>>>
>>> I don't think that time multiplexing is completely of the table now,
>>> but multiple kinects are working way better than expected, so maybe
>>> the hardware could be used without any modification in some cases. We
>>> also would have to see about interference when using more than two
>>> kinects.
>>>
>>> that's it for now,
>>>
>>> aurel
>>>
_______________________________________________
Ros-kinect mailing list
[hidden email]
https://code.ros.org/mailman/listinfo/ros-kinect
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Ros-kinect] multiple kinects working

rusu
Administrator
In reply to this post by rusu
Nink,

I only have 2 Kinects here, but I'll try with 3 or more on Monday at work.

Are you using ROS-kinect btw? We've already tested this and it works great there. You can easily do device selection as
a command line parameter or a ROS launch parameter, then pop up individual views of each camera in
rviz/image_view/pcl_visualization, etc. I am pretty sure that if you use that, you can get it to work for 3 cameras too,
unless there's some hardware limitation that we're not aware of.

Also make sure that whatever libfreenect fork you're using includes the multiple cameras patch that we wrote.

Cheers,
Radu.


On 11/21/2010 01:43 PM, Nink wrote:

> Hi Radu
>
> I ahve not made any progress on the multiple kinects.  Although it
> identifies 3 kinects it doesnt address each one individually so I
> think we are trying to initialise the same one over again.  I have
> tried to just init all 3 but no go.  Perhaps if we had a way to select
> what device we want to address as a switch on glview (say K1 K2 K3 or
> K4 etc) to ensure it knows what device we want to communicate with.
>
> Here is my config as you can see they are all on a seperate bus and
> all high speed
>
>    Bus 03.Port 1: Dev 1, Class=root_hub, Driver=ehci_hcd/4p, 480M
>      |__ Port 1: Dev 6, If 0, Class=hub, Driver=hub/3p, 480M
>          |__ Port 1: Dev 8, If 0, Class=vend., Driver=, 480M
>          |__ Port 2: Dev 7, If 0, Class=vend., Driver=, 12M
>          |__ Port 3: Dev 9, If 0, Class=vend., Driver=, 480M
> /:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=ehci_hcd/6p, 480M
>      |__ Port 6: Dev 3, If 0, Class=hub, Driver=hub/3p, 480M
>          |__ Port 1: Dev 5, If 0, Class=vend., Driver=, 480M
>          |__ Port 2: Dev 4, If 0, Class=vend., Driver=, 12M
>          |__ Port 3: Dev 6, If 0, Class=vend., Driver=, 480M
> /:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=ehci_hcd/6p, 480M
>      |__ Port 5: Dev 3, If 0, Class=hub, Driver=hub/3p, 480M
>          |__ Port 1: Dev 5, If 0, Class=vend., Driver=, 480M
>          |__ Port 2: Dev 4, If 0, Class=vend., Driver=, 12M
>          |__ Port 3: Dev 6, If 0, Class=vend., Driver=, 480M
>
> On Nov 19, 7:15 pm, Radu Bogdan Rusu<[hidden email]>  wrote:
>> Again, in our tests the interference is minimal (if any) with 2kinects. We'll post a video up later to show this in
>> case people still have doubts ;) We got about 10-20 of them by now in the building, so we might try a crazy 20-Kinect
>> registration experiment ;)
>>
>> Cheers,
>> Radu.
>>
>> On 11/19/2010 03:01 PM, Aurel W. wrote:
>>
>>> Another big issue, were interference is quite heavy, is when you face
>>> twokinectsin a way, that one ir camera has another laser projector
>>> in the field of view. This results in some lens flare. So I guess
>>> adding shutters and projector modulation may still be necessary for
>>> good results in multi kinect setups.
>>
>>> On 19 November 2010 23:54, Nink<[hidden email]>    wrote:
>>>> What I find real intersting put twokinectson top of each other and cover the infared for one but leave camera uncovered.  Black screen.   They know what signal they sent out.  There is no issue the folks at primesence were smart.
>>
>>>> I have a quad core i7 running now and its humming nicely with 2kinects.  Trying to add third I have some pci cards I am playing with at the moment to get the third one up.  Goal is 6.  Peaks at 50 percent then idles at 20 percent.  Ill tweet a pic on @nink.
>>>> Sent from my BlackBerry
>>
>>>> -----Original Message-----
>>>> From: "Aurel W."<[hidden email]>
>>>> Sender: [hidden email]
>>>> Date: Fri, 19 Nov 2010 23:40:22
>>>> To:<[hidden email]>
>>>> Reply-To: [hidden email]
>>>> Subject:multiplekinectsworking
>>
>>>> Hello,
>>
>>>> in the last days there has been quite some discussion about using
>>>> multiplekinects. Main reason is, we could get rid of occlusions and
>>>> really use it for full real time 3d reconstruction. A first starting
>>>> point would be to use an stereo setup.
>>
>>>> Sorry for this long post, but I kept working on this like crazy in the
>>>> past two days so I have quite some stuff to say, hope it turns out
>>>> useful to you. Interesting stuff on the bottom ;)
>>
>>>> When you think of, how the depth sensor works one issue comes
>>>> immediately to your mind, A structured light pattern is projecteder in
>>>> IR, while the viewing frustum of the projector has some slight offset
>>>> to the one of an ir camera, but they are axis aligned. From just this
>>>> it is possible to reconstruct depth. However, if you would use
>>>> multiplekinects, there could be interferences, when the structured
>>>> light is visible for the other kinect or when the laser projector is
>>>> directly visible without any reflection. So it was common sens, that
>>>> multiplekinectswon't work by now.
>>
>>>> Some ideas to address this issue were stated, not all of them are
>>>> practical. Some comments on those ideas:
>>
>>>> * Time multiplexing with 1/30s time slots. This basically means that
>>>> in a stereo setup, you would subsequently capture one frame at one
>>>> kinect and disable the laser projector on the other kinect. First keep
>>>> in mind that all devices are running asynchronously. Depending on the
>>>> exposure time for the ir-camera, one frame from one kinect can
>>>> "overlap" two frames in the other kinect. In the worst cast of 1/30s
>>>> exposure time, we would need to wait for an entire frame, before
>>>> starting to capture. This means, in a stereo setup we would drop from
>>>> the orifinal 30fps, to 7.5fps. lame. In addition, we don't know yet,
>>>> if the device allows us to enable and disable the ir projector frame
>>>> by frame. If not, someone would need to do this by altering hardware
>>>> and switching the diode separately with a micro controler (or the
>>>> built in motor controler, if you want to be clever :)
>>
>>>> * Frequency multiplexing. Using different narrow band ir lasers and
>>>> filters. This would definitely work, if you get the optics right. Tho
>>>> quite complicated imho.
>>
>>>> * Polarization. Won't work. A reflection on an arbitrary surface won't
>>>> preserve polarization well enough. The reason why polarization
>>>> projection screens for e.g. work, is because they use a special
>>>> coating. This won't work on human skin or other materials tho.
>>
>>>> Because all this ideas have some sort of issue I tried something
>>>> different. Normally someone would use synchronized devices, very short
>>>> exposure times and time multiplexing. But that won't be possible with
>>>> the kinect. My approach now was to add separate shutters to both, the
>>>> camera and projector and to do time multiplexing within one single
>>>> frame of the ir camera. This can be done with a high speed shutter.
>>
>>>> For the ir-projector the best approach would be to modulate the laser
>>>> diode accordingly. For the camera, some sort of mechanical or lcd
>>>> shutter is needed. Shutters would be added to bothkinectsand the
>>>> shutters have to be synchronized of course, while the ir-cameras of
>>>> thekinectsare running asynchronously. Of course, the exposure time
>>>> of one single frame is shorter now.
>>
>>>> To find out, how the depth sensor would behave, if i add a shutter, i
>>>> made different experiments.This is only with one kinect by now. One
>>>> setup looked like this:
>>
>>>> http://atommuell.mum.jku.at/~aurel/kinect_roling_shuter_proto_2.jpg
>>
>>>> The depth sensor gives slightly more errors here, because of the short
>>>> exposure time, but in general the sensor seems to be pretty robust
>>>> when doing such things as adding rolling shutters.
>>
>>>> The next step was to get a second kinect and to build an actual
>>>> prototype, where both sensors are stereo separated by shutters.
>>>> However, I was quite surprised when i pointed two plainkinectson the
>>>> same object and started capturing. My intention was, and as it was
>>>> discussed all around the net by now, that things would be completely
>>>> screwed up. Both sensors were working quite good with some added
>>>> errors. See for yourself:
>>
>>>> http://atommuell.mum.jku.at/~aurel/two_kinects_same_direction_c_both.png
>>
>>>> Compared to using just one kinect:
>>
>>>> http://atommuell.mum.jku.at/~aurel/two_kinects_same_direction_c_singl...
>>
>>>> Interference is imho very low, much lower than expected. Also an
>>>> interesting test, to lower the luminance of one of the projects to
>>>> 50%:
>>
>>>> http://atommuell.mum.jku.at/~aurel/two_kinects_same_direction_c_both_...
>>
>>>> I don't think that time multiplexing is completely of the table now,
>>>> butmultiplekinectsare working way better than expected, so maybe
>>>> the hardware could be used without any modification in some cases. We
>>>> also would have to see about interference when using more than two
>>>> kinects.
>>
>>>> that's it for now,
>>
>>>> aurel
_______________________________________________
Ros-kinect mailing list
[hidden email]
https://code.ros.org/mailman/listinfo/ros-kinect
Loading...