Skip to content

Commit 93f06a0

Browse files
committed
release
0 parents  commit 93f06a0

File tree

5 files changed

+295
-0
lines changed

5 files changed

+295
-0
lines changed

LICENSE

+21
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
MIT License
2+
3+
Copyright (c) 2024 Xuxin Cheng, Jialong Li, Shiqi Yang, Ge Yang and Xiaolong Wang
4+
5+
Permission is hereby granted, free of charge, to any person obtaining a copy
6+
of this software and associated documentation files (the "Software"), to deal
7+
in the Software without restriction, including without limitation the rights
8+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9+
copies of the Software, and to permit persons to whom the Software is
10+
furnished to do so, subject to the following conditions:
11+
12+
The above copyright notice and this permission notice shall be included in all
13+
copies or substantial portions of the Software.
14+
15+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21+
SOFTWARE.

README.md

+85
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,85 @@
1+
## 📺 Open-TeleVision: open-source tele-operation with vision 🤖
2+
Xuxin Cheng*, Jialong Li*, Shiqi Yang, Ge Yang,
3+
Xiaolong Wang
4+
5+
UC San Diego, MIT
6+
7+
<p align="center">
8+
<img src="./img/television.jpg" width="80%"/>
9+
</p>
10+
11+
## Introduction
12+
Stream head, hand, wrist data from VisionPro or Meta Quest 3. Stream real-time stereo video from camera to VR Devices.
13+
14+
## Installation
15+
16+
```bash
17+
conda create -n tv python=3.8
18+
conda activate tv
19+
pip install 'vuer[all]==0.0.31rc7 opencv-python numpy'
20+
```
21+
Install zed sdk: https://www.stereolabs.com/developers/release/
22+
23+
## Local streaming
24+
Apple does not allow WebXR on non-https connections. To test the application locally, we need to create a self-signed certificate and install it on the client. You need a ubuntu machine and a router. Connect the VisionPro and the ubuntu machine to the same router.
25+
1. install mkcert: https://github.com/FiloSottile/mkcert
26+
2. check local ip address:
27+
28+
```
29+
ifconfig | grep inet
30+
```
31+
Suppose the local ip address of the ubuntu machine is `192.168.8.102`.
32+
33+
3. create certificate:
34+
35+
```
36+
mkcert -install && mkcert -cert-file cert.pem -key-file key.pem 192.168.8.102 localhost 127.0.0.1
37+
```
38+
39+
4. open firewall on server
40+
```
41+
sudo iptables -A INPUT -p tcp --dport 8012 -j ACCEPT
42+
sudo iptables-save
43+
sudo iptables -L
44+
```
45+
or can be done with `ufw`:
46+
```
47+
sudo ufw allow 8012
48+
```
49+
5.
50+
```python
51+
self.app = Vuer(host='0.0.0.0', cert="", key="")
52+
```
53+
54+
6. install ca-certificates on VisionPro
55+
```
56+
mkcert -CAROOT
57+
```
58+
Copy the rootCA.pem via AirDrop to VisionPro and install it.
59+
60+
Settings > General > About > Certificate Trust Settings. Under "Enable full trust for root certificates", turn on trust for the certificate.
61+
62+
settings > Apps > Safari > Advanced > Feature Flags > Enable WebXR Related Features
63+
64+
7. open the browser on Safari on VisionPro and go to `https://192.168.8.102:8012?ws=wss://192.168.8.102:8012`
65+
66+
8. Click `Enter VR` and ``Allow`` to start the VR session.
67+
68+
## Network Streaming
69+
For Meta Quest3, installation of the certificate is not trivial. We need to use a network streaming solution. We use `ngrok` to create a secure tunnel to the server. This method will work for both VisionPro and Meta Quest3.
70+
71+
1. Install ngrok: https://ngrok.com/download
72+
2. Run ngrok
73+
```
74+
ngrok http 8012
75+
```
76+
3. Copy the https address and open the browser on Meta Quest3 and go to the address.
77+
78+
## Citation
79+
```
80+
@misc{cheng2022open,
81+
title={Open-TeleVision: open-source tele-operation with vision},
82+
author={Cheng, Xuxin and Li, Jialong and Yang, Shiqi and Yang, Ge and Wang, Xiaolong},
83+
year={2024}
84+
}
85+
```

TeleVision.py

+141
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,141 @@
1+
from vuer import Vuer
2+
from vuer.schemas import ImageBackground, Hands
3+
from multiprocessing import Process, Array, Value, shared_memory
4+
import numpy as np
5+
import asyncio
6+
7+
class TeleVision:
8+
def __init__(self, img_shape, stereo=False):
9+
self.stereo = stereo
10+
11+
self.app = Vuer(host='0.0.0.0', cert="./cert.pem", key="./key.pem", queries=dict(grid=False))
12+
13+
self.app.add_handler("HAND_MOVE")(self.on_hand_move)
14+
self.app.add_handler("CAMERA_MOVE")(self.on_cam_move)
15+
self.app.spawn(start=False)(self.main)
16+
17+
self.img_shape = (2*img_shape[0], img_shape[1], 3)
18+
self.img_height, self.img_width = img_shape[:2]
19+
self.shm = shared_memory.SharedMemory(create=True, size=np.prod(self.img_shape) * np.uint8().itemsize)
20+
self.shm_name = self.shm.name
21+
self.shared_image = np.ndarray(self.img_shape, dtype=np.uint8, buffer=self.shm.buf)
22+
self.shared_image[:] = np.zeros(self.img_shape, dtype=np.uint8)
23+
24+
self.left_hand_shared = Array('d', 16, lock=True)
25+
self.right_hand_shared = Array('d', 16, lock=True)
26+
self.left_landmarks_shared = Array('d', 75, lock=True)
27+
self.right_landmarks_shared = Array('d', 75, lock=True)
28+
29+
self.head_matrix_shared = Array('d', 16, lock=True)
30+
self.aspect_shared = Value('d', 1.0, lock=True)
31+
32+
self.process = Process(target=self.run)
33+
self.process.start()
34+
35+
def run(self):
36+
self.app.run()
37+
38+
async def on_cam_move(self, event, session):
39+
try:
40+
with self.head_matrix_shared.get_lock(): # Use the lock to ensure thread-safe updates
41+
self.head_matrix_shared[:] = event.value["camera"]["matrix"]
42+
with self.aspect_shared.get_lock():
43+
self.aspect_shared.value = event.value['camera']['aspect']
44+
except:
45+
pass
46+
47+
async def on_hand_move(self, event, session):
48+
try:
49+
with self.left_hand_shared.get_lock(): # Use the lock to ensure thread-safe updates
50+
self.left_hand_shared[:] = event.value["leftHand"]
51+
with self.right_hand_shared.get_lock():
52+
self.right_hand_shared[:] = event.value["rightHand"]
53+
with self.left_landmarks_shared.get_lock():
54+
self.left_landmarks_shared[:] = np.array(event.value["leftLandmarks"]).flatten()
55+
with self.right_landmarks_shared.get_lock():
56+
self.right_landmarks_shared[:] = np.array(event.value["rightLandmarks"]).flatten()
57+
except:
58+
pass
59+
60+
async def main(self, session, fps=60):
61+
session.upsert @ Hands(fps=fps, stream=True, key="hands")
62+
while True:
63+
display_image = self.shared_image
64+
65+
if not self.stereo:
66+
session.upsert(
67+
ImageBackground(
68+
display_image[:self.img_height],
69+
format="jpeg",
70+
quality=80,
71+
key="left-image",
72+
interpolate=True,
73+
aspect=1.778,
74+
distanceToCamera=2,
75+
position=[0, -0.5, -2],
76+
rotation=[0, 0, 0],
77+
),
78+
to="bgChildren",
79+
)
80+
else:
81+
session.upsert(
82+
[ImageBackground(
83+
display_image[:self.img_height],
84+
format="jpeg",
85+
quality=40,
86+
key="left-image",
87+
interpolate=True,
88+
aspect=1.778,
89+
distanceToCamera=2,
90+
layers=1
91+
),
92+
ImageBackground(
93+
display_image[self.img_height:],
94+
format="jpeg",
95+
quality=40,
96+
key="right-image",
97+
interpolate=True,
98+
aspect=1.778,
99+
distanceToCamera=2,
100+
layers=2
101+
)],
102+
to="bgChildren",
103+
)
104+
await asyncio.sleep(1/fps)
105+
106+
def modify_shared_image(self, img, random=False):
107+
assert img.shape == self.img_shape, f"Image shape must be {self.img_shape}, got {img.shape}"
108+
existing_shm = shared_memory.SharedMemory(name=self.shm_name)
109+
shared_image = np.ndarray(self.img_shape, dtype=np.uint8, buffer=existing_shm.buf)
110+
shared_image[:] = img[:] if not random else np.random.randint(0, 256, self.img_shape, dtype=np.uint8)
111+
existing_shm.close()
112+
113+
@property
114+
def left_hand(self):
115+
with self.left_hand_shared.get_lock():
116+
return np.array(self.left_hand_shared[:]).reshape(4, 4, order="F")
117+
118+
@property
119+
def right_hand(self):
120+
with self.right_hand_shared.get_lock():
121+
return np.array(self.right_hand_shared[:]).reshape(4, 4, order="F")
122+
123+
@property
124+
def left_landmarks(self):
125+
with self.left_landmarks_shared.get_lock():
126+
return np.array(self.left_landmarks_shared[:]).reshape(25, 3)
127+
128+
@property
129+
def right_landmarks(self):
130+
with self.right_landmarks_shared.get_lock():
131+
return np.array(self.right_landmarks_shared[:]).reshape(25, 3)
132+
133+
@property
134+
def head_matrix(self):
135+
with self.head_matrix_shared.get_lock():
136+
return np.array(self.head_matrix_shared[:]).reshape(4, 4, order="F")
137+
138+
@property
139+
def aspect(self):
140+
with self.aspect_shared.get_lock():
141+
return float(self.aspect_shared.value)

img/television.jpg

230 KB
Loading

zed_example.py

+48
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
import numpy as np
2+
np.set_printoptions(precision=2, suppress=True)
3+
4+
import time
5+
import cv2
6+
from TeleVision import OpenTeleVision
7+
import pyzed.sl as sl
8+
9+
grd_yup2grd_zup = np.array([[0, 0, -1, 0],
10+
[-1, 0, 0, 0],
11+
[0, 1, 0, 0],
12+
[0, 0, 0, 1]])
13+
14+
resolution = (720, 1280)
15+
16+
# Create a Camera object
17+
zed = sl.Camera()
18+
19+
# Create a InitParameters object and set configuration parameters
20+
init_params = sl.InitParameters()
21+
init_params.camera_resolution = sl.RESOLUTION.HD720 # Use HD720 opr HD1200 video mode, depending on camera type.
22+
init_params.camera_fps = 60 # Set fps at 30
23+
24+
# Open the camera
25+
err = zed.open(init_params)
26+
if err != sl.ERROR_CODE.SUCCESS:
27+
print("Camera Open : "+repr(err)+". Exit program.")
28+
exit()
29+
30+
image_left = sl.Mat()
31+
image_right = sl.Mat()
32+
runtime_parameters = sl.RuntimeParameters()
33+
tv = OpenTeleVision(resolution)
34+
35+
while True :
36+
start = time.time()
37+
38+
if zed.grab(runtime_parameters) == sl.ERROR_CODE.SUCCESS:
39+
zed.retrieve_image(image_left, sl.VIEW.LEFT)
40+
zed.retrieve_image(image_right, sl.VIEW.RIGHT)
41+
42+
rgb_left = cv2.cvtColor(image_left.numpy(), cv2.COLOR_BGRA2RGB)
43+
rgb_right = cv2.cvtColor(image_right.numpy(), cv2.COLOR_BGRA2RGB)
44+
45+
tv.modify_shared_image(np.vstack((rgb_left, rgb_right)))
46+
47+
end = time.time()
48+
zed.close()

0 commit comments

Comments
 (0)