- Put HoloLens2 into developer mode and ensure access to the device portal.
Note. This is unnecessary if installing from the store
-
Familiarise with Unity Robotics Hub. Use the guide there for the pick and place tutorial to create either a Docker container or VM running Ubuntu with the ROS bits installed. The location of this will need to be accessible from the HoloLens.
-
Run the Pick and Place Player tool to send commands to the demo
The demo HoloLens app has been published to the HoloLens store but cannot be discovered by keyword search. You must have a direct link to the listing and also a promo code. Please contact Pete Daukintis for further details.
Clone the repo and open the pick-and-place folder in Unity (2020.3.28f1)
Warning. Don't use a higher version of Unity for now as there is an issue that won't allow the execution of the app to go past the spinning balls loading screen (This is reported as fixed in 33f1 but not yet tested with this repo). Conversely don't use an older version as there is an issue with TLS certificates and SignalR.
Locate the Scenes folder in Assets and open the KHIBuildDemo scene
Navigate to your Unity build settings, select Universal Windows Platform and press Switch Platforms. When complete configure your build settings as below (but ensuring that the KHIBuildDemo is selected):
Before building you need to configure ROS settings to reference your VM/container:
Or alternativey, to adjust this at runtime without needing to rebuild you can edit and re-upload a file on the HoloLens disk containing the IP address setting. A default one will be created if one doesn't exist and a connection will be made to the first one on the list automatically. The hand menu will allow switching the connection at run time:
{
"IpOptions" : {
"US": {
"IpAddress": "20.225.161.122",
"Port": 10005
},
"Germany": {
"IpAddress": "localhost",
"Port": 10005
},
"UK": {
"IpAddress": "192.168.0.116",
"Port": 10005
}
}
}
Save this to a file named rosip.json
And upload this to User Folders\LocalAppData\KHI-HoloLens_x.x.x.x_arm64__pzq3xp76mxafg\LocalState\
This will persist between application installs.
IMPORTANT The URDF Importer has not been configured correctly for running in a Universal Windows App at this time. In order to fix this I have forked the repo and taken a dependency on that fork here https://github.com/peted70/URDF-Importer.
The actual fix is to edit the Unity.Robotics.UrdfImporter.asmdef (navigate to it from the Unity Project panel - it will be in a package cache somehwere outside the project). Open it in a text editor and add
WSAto theincludePlatformsarray.Missing this step will result in a
URDFImporternot found inUnity.Robotics
Then choose Build, select an output folder and when the process is complete you will have a VS solution file with a C++ UWP project in. You can run/debug from here but what I usually do is go to the VS Project menu and select Publish > Create App Package...
That will build and package he UWP app. You can sideload the app using the HoloLens Developer Portal.
When the app starts the robot arm enclosure in the image above is a placeholder which is used to place the content where you want it to be positioned. This may need to be aligned with a real-world enclosure or simply may be a position in your environment that has the required space. Since application 'actions' will be used to carry out the menu operations we will cover those next.
An action is an operation that the application will carry out but is abstracted from how it is invoked.
For example, an arbitrary action may be executed by pressing a button in a menu of uttering a voice command.
Each of the following actions can be invoked by interacting with buttons on the menu or by issuing a voice command whilst the menu is visible.
-
Show Menu
-
Hide Menu
-
Plan Move. This will send the robot joint data to the ROS endpoint and will receive back a trajectory which will execute immediately moving the robot arm
-
Toggle Sensors. Hide/Show the sensor gauges
-
Reposition. Replace the current anchor by choosing another one
-
Create Anchor. Create a spatial anchor at the position of the placeholder enclosure
-
Locate Marker. FInd a QR code marker. This will display a 3-axis hologram at the top left of the QR code when located
-
Align Marker. Align the enclosure with the previously discovered QR code marker
-
Toggle Enclosure. This will set the visibility of the enclosure model on/off as required
-
'Dolls House'. Switched the application between Table Top and Actual Size mode. (See Application Modes)
The placeholder enclosure will be movable using hand interactions; rotating about the y-axis and movable in 3D space using both near and far interactions. The placeholder enclosure will remain upright and when you have moved it into the desired location bring up the hand menu by rotating your palm towards you (or saying Show Menu) and select the command to Create Anchor (or say Create Anchor). The placeholder enclosure will be replaced with the real content that will be parented to another gameObject that is associated with a spatial anchor. The anchor ensures stability for the holograms and also the position relative to the physical surroundings will be persisted through app runs. If you are not happy with the position you can use the Reposition button in the hand menu (or say Reposition whilst the menu is visible) to get the moving placeholder enclosure back.
The placeholder enclosure can be automatically positioned using QR codes.
In general, if you place a QR code into a 'known' position in 3D space (in our case, at a specific point on the placeholder enclosure) then we can associate a hologram in the Unity scene with the real, physical enclosure.
As this has just been implemented each phase of locating a QR code and aligning the hologram with it is kept separate. This will not be the case moving forwards. For now, the following actions can be carried out to 'automatically' position the enclosure:
- Show Menu
- Find Marker (wait until the 3-axis widget appears in the top left of the QR code marker)
- Align Marker (this will align the hologram of the enclosure to where the QR code marker is in the real world)
- Create Anchor
The app supports mutiple QR codes which currently map to modes in the application (See Application Modes)). These are configured in the Placing script:
Where each 'mode' has a transform QR Marker Local Position specifying the position in local object space that the object will be aligned with, a QR Marker Real Position which is used for testing the alignment in the Unity editor. An additional rotation may be specified and in this case is used to orient the robot enclosure as expected on the table top.
When the Align Marker action is issued the object to be aligned will animate to the
QR Marker Real Positionwhich in the editor is specified by the transform above but in the real world will be specified by the actual QR marker (printed on some paper).
When you have determined a good place on the physical enclosure to place a QR marker you will need to position it in exactly the same spot (in local coordinates) on the placeholder enclosure model:
In order to test the alignment code there is also a game object representing the real-world QR code. This can be positiond in the scene and when the Align Marker action is invoked the placeholder enclosure whould be aligned with this game object in the Unity editor.
The sample can be configured to use Local Spatial Anchors or Azure Spatial Anchors. This is currently defined at build time by checking the checkbox below defined in a Unity script named AnchorSwitcher.cs:
Creating an anchor will result in a Unity Game Object being created which will have it's transform determined by the anchor data. This transform will change as the HoloLens understands more about it's environment. The content will get re-parented beneath the anchor node for the current application mode. In table top mode there is a transform that scales the content down which is added as a child of the anchor node.
Anchors will not work in Unity so in order to debug the anchors there are two paths
- Holographic Remoting to debug in Unity but have anchors created on the HoloLens
- Use the managed debugger to run the app fully on HoloLens and debug remotely
These are anchors that are stored locally on the disk of the HoloLens. They don't require a network to function but will get removed if the app is uninstalled.
To remove the anchor manually go to the HoloLens developer portal and choose the option
System > Map Manager > Reset System Map, Anchors and Spatial Mapping Database
There is a rudimentary implementation of Azure Spatial Anchors that is funcitonally the same as the local anchor implementation. It stores anchor IDs in a local file on the HoloLens and to remove the anchor you can delete/edit that file via the Developer Portal:
Note. This anchor id could be stored via a cloud API and shared to iOS/Android versions of the app so that all devices could share a coordinate system.
The robot arm and product track can be fully simulated by replaying data recorded from the hardware. This is in the form of running a Python script which sends data to the ROS container, which is in-turn replayed to the HoloLens app.
By default the iOT sensor data is simulated with random values updating every second. To override this see iOT Sensors below.
The cubes represent a source and target position for the robot arm to move and you can pick these up with your hand and reposition them on the table. Then if you turn one of your palms towards you a menu will be shown.
If you 'tear' off the menu with your opposing hand it will become anchored in the scene.
There are sensor data UI elements which rotate around the table depending on the user's viewpoint:
Note. These widgets are updated using data from a SignalR server. This functions both in the editor and on a HoloLens2. Note that the frequency of updates is determined by an Iot Hub that is outside the scope of this project and is currently updated every minute supplying data payload:
You can view the data at https://iotconsumerfunction.azurewebsites.net/api/index
The data that is shown in the HoloLens app is filtered by device id. The device id to use is defaulted to
92592c88ca5e0a1ad7fd670b74665977d78f81201634b5b5c73a49943853a9a80d9e45ea14e7d4ac9b5a24937b903ecd1091e110578d92923fef0afab19fcbd8
but this can be overridden by uploading a file called iot.json (in the same way as can be done for ROS IP Address Overrides with contents as follows:
Please see Iot Consumer Function for details of the data pipeline.
The indicator and IP address whow the current connection status and if you press the 'Plan' button a ROS request will be made sending the robot arm angles, source position and target position. The ROS nodes running on the server will then calculate a move and send that back to the robot arm which should begin the move. You can reposition the cubes and issue further requests.
The app is now configured to work in two separate modes; actual size, where the robot arm and enclosure are at 1:1 size with the real robot and other hardware. Both modes support manual manipulation or QR code alignment to position the holograms. The mode can be switched using the 'Dolls House' voice command or hand menu button.
This mode is designed as either a full-size demo or to align with the physical hardware. This is the original mode and most of the implementation and documentation so far refers to this mode.
This mode is designed to be viewed on a flat surface such as a table or desk. The placement hologram is a scaled-down version of the real sized version and when using QR code aligment the coordinate system is facing upwards so that the QR marker can be placed flat on a table and the resulting hologram will be aligned accordingly.
Note. the plan trajectory feature does not currently work in table top mode
This tool is designed to allow a user to send commands to the Hololens app via the ROS containers.
Learn more here Pick and Place Player tool.