Software Setup
This is not an exhaustive resource for how to do every step in setting up your software, however we cover the majority of important information here.
Last updated
This is not an exhaustive resource for how to do every step in setting up your software, however we cover the majority of important information here.
Last updated
Important notes before moving further into testing additional packages
It is not necessary to run the Stereo DNN in order to test the functionality of the TrailNet DNN for path navigation. Below is the rqt_graph that shows the flow of subscribing and publishing of topics between nodes. It is advantageous to validate the ability of the drone to carry out just the navigation package tasks on its own because it is less likely that the TX2 will overheat or processes will crash if both the TrailNet DNN and the Stereo DNN were running at the same time. From our experience, modifications to the frame are necessary in order to install the heatsync and fan (or fans) in order to keep the TX2 below 60 degC. If the board goes past 60 degC it will automatically reduce its clock speed in order to prevent damage, thus artifically reducing the true capabilities of the Jetson. We want to eventually combine several packages and perform even more complex navigation techniques, so we suggest making the frame modifications suggested in the "Hardware Setup" section if you want to test beyond this point.
Although these steps are for Jetpack 4.2.2 the same process can be followed for Jetpack 4.3 just make sure you flash the correct carrier board firmware for that version of Jetpack.
Download the Nvidia SDK Manager
Download the J120 Firmware 2.2 for the TX2
or
Download the Obritty Firmware
Start the SDK Manager
Select Jetpack 4.2.2 and make sure you have the host machine unchecked.
First we need to populate the ~/nvidia/nvidia_sdk folder.
You need to only check the Jetson OS components for download and installation.
Note you cannot choose the "Download now. Install later" option because we need the ~/nvidia/nvidia_sdk files to be populated so we can migrate our firmware into these files for the J120.
We will install the SDK components later on.
Once the OS image is downloaded and the /nvidia/nvidia_sdk folder is populated we can skip the flash and then navigate to the carrier board firmware directory.
Check the README file in the firmware folder, and verfiy that the correct paths and files are copied over into the /nvidia/nvidia_sdk folder.
For example the J120 would be something like this
after running the ./apply_binaries.sh you should not get any errors, and then you are good to move to the next step.
Re-Flashing with the J120 firmware patch installed
Now you need to turn off the TX2 on the J120 and boot back into recovery mode.
connect a micro-usb cable to the J120 and to your computer (preferably the one that came with the TX2 that has the little green controller symbol on it).
connect power to the J120 and the green LED should not come on indicating the OS has not booted.
then, you need to boot into recovery mode.
--> hold the REC (Recovery) button and then press the power button (still holding REC)
--> press the reset button once while STILL holding the REC button and then wait 2 seconds until releasing the REC button.
Now to verify the device is connected to the HOST PC you can run 'lsusb' and if NvidiaCorp shows up as a listed USB device you are set.
Another way to verify the TX2 has booted in recovery mode is to hook up the HDMI to a display, nothing should appear on the display if properly booted into recovery mode
Then follow the sdkmanager steps as we did before. ONLY install the Jetson OS components for download and installation. NOT the sdk components.
It should indicate that the OS image is ready with a green check mark because we already populated and flashed before, but now we modified that image and it will re-flash.
OS Boot up and Set-up
Once the sdkmanager has completed the flash you should see the TX2 boot up if you are connected to an HDMI.
verify that the J120 patch worked by checking to see if both the top and bottom USB ports are working.
note that only the upper USB port is USB3, the bottom is USB2.
also note that the micro-usb port does not work so you wont see it appear using lsusb anymore, only in forced recovery mode will it appear using lsusb. You cant use this port in normal operation.
while we are here we can configure our username/password then once those basics are finished on the setup wizard you can move to the next step.
Installing the SDK components from sdkmanager
Now this next step was traditionally done using an ethernet connection but now they have updated it to where the usb to micro-usb connection acts as usb-ethernet.
If you remember, our carrier board disabled the micro-usb port and in order to install the SDK components we cannot be in forced recovery mode. That mode is only used for flashing!!
So here is where it gets silly
Power down the TX2 and remove it from the J120 carrier board.
Get out the old development board and install the TX2 onto there and make sure you bring your antenna's with you because we need to connect to the internet.
Once you have everything moved over, boot up and login
open a terminal and run
Now connect the micro-usb to your HOST PC and run the sdkmanager.
run ifconfig on the TX2 and your host machine to ensure the IP's are 192.168.55.1 and 192.168.55.100 respectively.
run lsusb and see if the NvidiaCorp device shows up, it should.
ONLY check the sdk components for download and installation.
after the download is complete a window will appear asking for your username and password with a defualt IP address that correlates to the serial connection you have over usb w/ the TX2.
continue the installation and hopefully since we already updated the packages earlier we wont have any failed packages. (Maintain internet connection during install just to be safe)
Complete
run nvcc --version and see if CUDA is there!
you can verify any other packages a swell, but that's all. Easy! ;D
Jetsonstats is a great package that allows you to check all of the critical information and health on the TX2 quickly.
here you can change the power mode to MAX N and start the jetson_clocks service
Alternatively you can run
Now create your catkin workspace environment
If you are completely new to ROS please take the time to read through these tutorials
If you want to really understand how ROS works and why it is so applicable in robotics applications read this book ( at minimum the first 4-5 chapters)
Download the ZED SDK https://www.stereolabs.com/developers/release/
Its up to you what options you would like to install/not install
Navigate to this
Add the following to a new line in the file
Note if you want to connect to the internet, you will need to comment out this line in order to do so. This forces it to only be able to broadcast a hotspot connection.
Although the screenshot below is from ubuntu 16.04, setting up a new wifi connection and setting the mode to hotspot on ubuntu 18.04 is similar.
Once you have made the changes to the bcmdhd.conf configuration file and created a new wifi connection you can restart the TX2. After it boots up you should notice the hotspot has started. You can check to see if you can connect from another local machine.
In this example lets assume that
+Drone's IP: 10.42.0.1
+Host PC IP: 10.42.0.2
On the TX2
On the Host PC (With ROS melodic installed using the same steps provided beforehand)
Now when you are connected to the TX2's hotspot from the Host PC you will be apart of the ROS network and can publish/subscribe or view information about topics/nodes etc.
Use these commands to install the joy package, start a ROS master and then run the joy node with a controller connected to the Host PC.
Download the AppImage here
Following this guide
We are using the FrSky Taranis QX7 transmitter with the FrSky R-XSR SBUS 2.4GHz Micro receiver so these steps may differ for other transmitter/receiver pairings, if you are interested in other pairings check this out. The Taranis QX7 and R-XSR Manuals are linked below for reference
Go to SETUP and create a new model and configure the the internal RF and external RF settings as shown below:
Go to the INPUTS page and add the following switch channels 05-08
Once you have the switches assigned to a channel, you can configure the switches in QGC in order to change flight modes, arm/disarm ect. We suggest using switch SH as a emergency kill switch.
Assuming everything is properly downloaded, you can being launching ROS nodes. You may run into an error when launching a ROS node like "FCU: DeviceError:serial:open: Permission Denied". To remedy this, run
Following mtbsteve's 'Testing of the Installation' wiki page, you can test the ZED2 launcher and ROS to RTSP launcher.
In order to view the RTSP stream in QGroundControl on the Host PC, navigate to General Settings > Video. Choose RTSP Video Stream and set the RTSP URL to 'rtsp://10.42.0.1:8554/<mountpoint>'
Mountpoints can be configured in ~/catkin_ws/src/ros_rtsp/config/stream_setup.yaml by following the instructions here. For initial testing, just use the /zedimage mountpoint. If any changes are made to the catkin workspace, the workspace must be rebuilt; for example
To test MAVROS and the connection between the flight controller and the Host PC, run the px4 controller node. QGC should connect to the drone.
To test the Darknet-YOLO object detection node, run the following
You can view this in RVIZ or via RTSP on the Host machine by changing the mountpoint to /zedyolo. This will output a bounding box over the zed image. It's not very accurate.
As stated before, stereoDNN is not used specifically with Trailnet, but allows the integration of depth for other parts of the project. To test the stereoDNN, run the following
Assuming the tests pass, you can do a test run of each of the different DNNs on a set of sample images. The output of each run can be found at ./bin/disp.bin
To see the output from the ZED2, run the following
The output can be seen in RVIZ by adding the stereo_dnn_ros's image topic
mtbsteve created a node that displays the left and right images as well as the stereoDNN's output and a color coded version of the stereoDNN's output based on the KITTI color scheme. This can be seen in RVIZ by running the following
To run everything, minus the Autonomous Controller, you can run
Looking in this .launch file, you will see that it is running lots of different nodes: ZED2, Trailnet, YOLO, stereoDNN, MAVROS, ROS to RTSP, and redtail_debugger. If you want to run Darknet-YOLO on top of this, open another terminal and run it as shown in the previous section.
For a more visual representation of what Trailnet is trying to do, an arrow overlay can be added to any of the streamed topics for RTSP, so it can be seen in QGC. All changes are in the folder ~/catkin_ws/packages/redtail_debug/. In CMakelists.txt, OpenCV and cv-bridge dependencies were added. In package.xml, the cv_bridge dependency was added. Most of the modification was in the /src/redtail_debug_node.cpp file and was commented to help with both identification and understanding.
In order to do this, we piggybacked off of the redtail_debugger. In simple, we subscribed to the /zed2/zed_node/left/image_rect_color topic. Once subscribed, we can use cv_bridge's toCvCopy() in the cameraCallback() function to bridge between a ROS message to an OpenCV image.
This allows us to use any of the OpenCV functions (in this case arrowedLine()) to modify the image.
Then we can convert the OpenCV image back to a ROS message with toImageMsg() and publish the topic.
In order to change the topic you want the arrow overlay to subscribe to, you can change the first argument for the subscription found on line 69. This will require the catkin_ws/ to be rebuilt each time it is changed. Another option (probably better) would be to add another nh.param like on line 53, which can be changed in the launch file that starts the redtail_debug node ('caffe_ros everything.launch' for example).
This arrow overlay is still a work in progress. The algorithm for both the angle and offset may need to be tweaked.
Once you have everything running (i.e. roslaunch caffe_ros everything.launch) and you want to start autonomous flight, you should run the following
THIS SHOULD BE TESTED WITHOUT PROPELLERS FIRST
This will take over the drone and put the drone into offboard mode (QGC should tell you this). The drone will then take off and hover, awaiting input to begin autonomous travel. This process is more thoroughly described in the Field Testing page.
