How to link Unity3D and Gazebo for robot control


How to link Unity3D and Gazebo for robot control

  • Creation: 03/23/2020
  • Update: 08/26/2020

This introductory video explains the purpose and the context of the tutorial.

This tutorial has two steps:

1.      Running through the steps required to set up both environments

2.      Examining the code that controls the Turtlebot in more detail.

As you can see, ROS# manages the interface between Unity (on Windows) and Gazebo (on Ubuntu). For simplicity, an existing Gazebo simulation is used to simulate the TurtleBot2 on the ROS side. The control signals are sent from Unity to ROS. Furthermore, the outcomes of ROS are captured by Unity for illustrative purposes.


# Unity

  1. Start Unity and follow on screen instructions to sign in/create an account and a license
  2. Create a new project
  3. Copy the RosSharp folder from the latest commit of this repository into the Assets folder of your Unity project:



# Ubuntu

  1. ROS installation (pour Kinetic, la documentation officielle permet de faire une installation facile, voyez ici)
  2. Gazebo installation (in the case of an Ubuntu16.04 VM: Gazebo7 is most compatible, but compatibilities are well explained here.
$ sudo sh -c 'echo "deb `lsb_release -cs` main" > /etc/apt/sources.list.d/gazebo-stable.list'
$ wget -O - | sudo apt-key add -
$ sudo apt-get update
$ sudo apt-get install gazebo7
$ sudo apt-get install libgazebo7-* //required packages


3. Get the Turtlebot2 files.

$ sudo apt-get install ros-kinetic-turtlebot ros-kinetic-turtlebot-apps ros-kinetic-turtlebot-interactions ros-kinetic-turtlebot-simulator ros-kinetic-kobuki-ftdi ros-kinetic-ar-track-alvar-msgs ros-kinetic-turtlebot-gazebo



4. Setup the rosbridge server.

$ sudo apt-get install ros-kinetic-rosbridge-server 
// instalosbridge-server [link]

Find the ROS files at this link:

Place the file_server package in the src folder of your Catkin workspace, as well as the gazebo_simulation_scene and the unity_simulation_scene packages.

Then build by running:

$ catkin_make

Make packages executable:

$ chmod +x // make the 
//file executable
$ sudo apt install python-xlib // required 
// by in unity_simulation_scene/scripts
$ chmod +x // make the 
// file executable


5. Transfer de l’URDF

roslaunch file_server publish_description_turtlebot2.launch

Verifie l’addresse IP de la machine sur le reseau wifi :


This command is used to check the network configuration and to verify your Ethernet connection to the Windows OS. The IP address (enp0s8) of the Ubuntu system will be used by rosbridge_suite and RosBridgeClient.


Go to RosBridgeClient > Transfer URDF from ROS...

Utilise l’adresse IP dans la barre d’addresse.

Initiate the transfer by clicking the button Read Robot Description.

Compose the Unity scene GazeboSimulationScene by following this video.



First set up the environment variables!

$ export SVGA_VGPU10=0
$ export TURTLEBOT_BASE=roomba
$ export TURTLEBOT_STACKS=circles
$ export export TURTLEBOT_3D_SENSOR=kinect

Now we are ready to launch the default simulation scene.

$ roslaunch gazebo_simulation_scene gazebo_simulation_scene.launch

This will launch rosbridge_websocketfile_serverjoy_to_twistrqt_graph and a Gazebo simulation of the TurtleBot2 in the default turtlebot_world

When the Play button in pressed, the ROS terminal will show that a client has connected and subscribes to some topics:

[INFO] [1520503573.897354, 57.560000]: Client connected. 1 clients total.
[INFO] [1520503574.689767, 58.330000]: [Client 0] Subscribed to /camera/rgb/image_raw/compressed
[INFO] [1520503574.697262, 58.340000]: [Client 0] Subscribed to /joint_states
[INFO] [1520503574.736412, 58.380000]: [Client 0] Subscribed to /odom

After clicking the refresh button in the rqt_graph, a network similar to the following figure appears:

Pressing the controller buttons or the arrow keys will move the robot in Gazebo and in Unity. Furthermore, the camera image of the TurtleBot2 in Gazebo is projected onto a plane in Unity for illustrative purposes.

This is what it looks like:

That's it! Your environment is entirely ready. We can finally examine the code used to control the Turtlebot in more detail.

#Changing the coordinate system:

	// Initialize Speed Parameter
  //   axes  [l.x   l.y   l.z   a.x     a.y    a.z]
	scalers = [, -3.14, -3.14]	

	// Start Mapping from Joy to Twist
	if len(data.axes) >= 6 :
		msg.angular.x = data.axes[5] * scalers[3]		

	if len(data.axes) >= 5 :
		msg.linear.z = data.axes[4] * scalers[2]	

	if len(data.axes) >= 4 :
		msg.angular.y = data.axes[3] * scalers[4]

	if len(data.axes) >= 3 :
		msg.linear.y = data.axes[2] * scalers[1]

	if len(data.axes) >= 2 :	
		msg.angular.z = data.axes[1] * scalers[5]

	if len(data.axes) >= 1 :
		msg.linear.x = data.axes[0] * scalers[0]


The file controls the movement of the Turtlebot using the /joy ROS node.

if __name__ == '__main__':

Giving rise to:

# initialize node
	rospy.init_node('JoyToTwist', anonymous=True)

	# setup joy topic subscription
	joy_subscriber = rospy.Subscriber("joy", Joy, handleJoyMsg, queue_size=10)

	# spin() simply keeps python from exiting until this node is stopped

This information is published.

// FILE:	

// Setup Twist Publisher 
   	twist_publisher = rospy.Publisher("cmd_vel_mux/input/navi", Twist)
	msg = Twist()

// Publish msg
	rate = rospy.Rate(100) # 10hz


The file controls

while not rospy.is_shutdown():
		#### Initialize joy msg every loop
		msg.axes = []
		msg.buttons = []
		pos_x = 0.0
		pos_y = 0.0
		#### Get Display Dependent Parameters
		d = display.Display()
		screen = d.screen()
		window = screen.root.create_window(0, 0, 1, 1, 1, screen.root_depth)
		res = randr.get_screen_resources(window)

		resolution_x = res.modes[0].width	# i.e. 1920
		resolution_y = res.modes[0].height	# i.e. 1080
		middlePoint_x = resolution_x / 2.0
		middlePoint_y = resolution_y / 2.0 
		#### Start Getting Postion of Mouse
		MouseData = display.Display().screen().root.query_pointer()._data
		pos_x = MouseData["root_x"]
		pos_y = MouseData["root_y"]

		#### Start Mapping from Mouse Position to Joy
		vel_linear  = (pos_y - middlePoint_y) / resolution_y * (2)
		vel_angular = (pos_x - middlePoint_x) / resolution_x * (2)