New movies on robot programming interface

Programming industrial robots to perform repetitive tasks is time-consuming. Based on our knowledge in manipulation planning, we have developed

  • a software that handles the manipulation constraints evoked above, and
  • a graphical user interface that enables a programmer to define the manipulation constraints relative to a problem.

The movies illustrate a robot programming interface to build a sequence of motions in order to perform an industrial task.

The input is:

– a model of the robot (UR5 robot arm with gripper)

– a model of the environment and objects, like trays, shaver parts,

– a position of frames on shaver parts, trays and gripper in order to specify the relative position of gripper/object to grasp objects (shaver part or tray). Future work should focus on building interactively those frames with external vision systems,

The user specifies through the graphical interface:

– initial position of each object (which part on which slot)

– final position of each object (first movie),

– the possible grasps (which gripper can grasp which object),

– the possible slots for each object.

The software then computes a collision-free motion between initial and goal configurations. Repeating this actions several times results in the movie below. All videos are on our YouTube Channel:

To ease motion planning, pre-grasp positions are automatically defined for the gripper above the shaver part to grasp.



This work was done by our partner LAAS:


Read More →

How Team Delft won the APC 2016 – new blog online

Team Delft – the winner of the Amazon Pickling Challenge 2016 – published a blog with technical details on

their robotic system.


Read More →

Interview with Dr. Hernandez Corbato on Team Delft’s victory

Team Delft won the Amazon Picking Challenge (APC) almost two weeks ago. Now that everybody had a bit of time to recover, we interviewed Dr. Carlos Hernandez Corbato,TU Delft Robotics Institute, one of the team leaders of Team Delft, on the technical details.

Congratulations to the great performance. What was the most difficult challenge from a technical point-of-view in the APC?

Joey Durham/Amazon Robotics (right) hands over the award of the Amazon Picking Challenge to Dr. Hernandez Corbato (middle) and Mr. van Deurzen from Team Delft. ©Amazon

Joey Durham/Amazon Robotics (right) hands over the award of the Amazon Picking Challenge to Dr. Hernandez Corbato (middle) and Mr. van Deurzen from Team Delft. ©Amazon

Dr. Hernandez Corbato: Dealing with objects with reflective surfaces, e.g. transparent plastic wrapping, was probably the hardest challenge for our vision system. Our main solution for grasp planning required to estimate the object position and orientation, and for that we relied on PointCloud data (depth information) from our 3D camera to match with a 3D model of the object. But due to reflections there was simply not enough Point Cloud data to determine the pose of some objects in certain situations.

How did you solve it?

We included application-specific heuristics on top of our pose-estimation algorithm (Super4PCS) to correct it: e.g. we snapped the estimated height of the object position to be lying in its bin, since objects could not be piled or ‘floating’. This did not solve the problem completely, but our approach was to have a fast and robust system, so that upon difficulties it puts the difficult object in a ‘to do’ list, that is addressed once the rest of the task is finished. After some trials and different viewpoints to get a better 3D image, the robot manages to pick also these difficult objects.

Which part that you thought to be easy in the beginning proved to be more difficult?

The robot of Team Delft.

The robot of Team Delft.

It happened the other way around. We though there was no easy part in the challenge, and were specially concerned about deformable objects, e.g. t-shirt, gloves. They could be hard to recognise, and moreover you do not have a static 3D model of these object for pose estimation. However, our robust approach to object recognition using Deep Learning performed excellent even with those objects, and the powerful suction in our gripper paved the way for grasping. We discovered that the robot could easily pick deformable products without complete 3D pose estimation, by simply aiming the suction cup to the centre of the region where it is detected.

Please give us a short description of the outstanding features of your robot.

Following the Factory-in-a-day approach, we analysed the picking application and chose the best technologies and components to build a robust and fast solution, as industry would require. We aimed for a really fast and robust system, capable of detecting failed picks and try them again. Instead of trying slow complex manoeuvres to reach occluded products, our robot simply removes blocking objects to other bins, keeping track of the location of all products. We used ROS-Industrial to quickly test and integrate the software components that we needed, and develop those not available (e.g. a generic grasp planner for a suction-based-gripper).

  • The Robot itself: After preliminary analysis, we concluded that maximum reachability in the workspace was a critical requirement for the competition. We chose a configuration with aSIA20F Motoman robot arm by Yaskawa mounted on a rail. The total 8 degrees of freedom allowed the system to reach all the bins with enough manoeuvrability to pick the target objects. Thanks to the ROS-Industrial drive supported by Motoman could integrate our complex robot configuration in our system, and we have contributed to improved the driver features.
  • Perception: We decided to avoid noise and calibration problems using industrial stereo camera’s with rgb overlay camera’s. One for detection of objects in the tote and one on the robot gripper to scan the bins. Detection of the target object went in two steps. For
    object recognition we used a deep learning neural network based on Faster RCNN. Next we used an implementation
    of super4PCS to estimate the pose of the object, refining it with ICP (Iterative Closest Point).
  • Grasping: Following the Factory-in-a-day approach of quick 3D prototyping, we developed a hybrid suction+pinch gripper customised to handle all the 39 products in the competition. We designed an algorithm that auto generates candidate grasp locations on the objects based on their estimated pose from the vision system, their geometry and application constraints. Object-specific heuristics
    were also included after intensive testing.
  • Motion: For motion planning and control we develop customised ROS-services on top of MoveIt!. To optimise motions we created a ‘motion-database’, with static free-collision trajectories between relevant locations, e.g. image-capture and approach locations in front of the shelf’s bins and over the tote. For the approach and retreat motions to pick objects we used dynamic cartesian planning using the depth information from the 3D camera for collision avoidance.

Factory-in-a-day aims at quick installation time: did you fulfill this and could you repeat the Picking challenge with another robot as successfully?

The development of the APC project took four months, from March to June. But we got the robot only five weeks before the competition. Thanks to ROS we could prepare the required components and test them in advance using Rviz and the ROS-Industrial simulator. But time was so short, we could only focus on testing for the Picking challenge with the real robot, and not for the new Stowing challenge. I think this is a good demonstration of the Factory-in-a-day approach. We arrived to the competition, installed the system and calibrated it in less than a day. Then we integrated and tested the stowing during a day and a half. Next day we won the Stowing Challenge with an almost perfect scoring.

Our solution for perception, grasp planning and task planning and coordination could be easily re-used. Only our robot motion and control subsystem would need to be adapted.

How long did it take to teach the robot grasping 40 objects? And is there some kind of “learning algorithm” that would make it faster in the future?

We used Deep learning techniques for the recognition of the products. Given the time constraint and our custom hybrid gripper with suction and pinch, we chose a customised solution to automatically generate the grasp strategy based on geometry information, item-specific heuristics and application constraints. This took 10 person week to develop. We are now considering using Deep Learning also to teach the system how to grasp objects.

I imagine the last months were quite stressful, what was the first thing you did after returning to Delft?

I went to have a nice dinner at a new restaurant in the cozy Delft’s city centre! Now we are working on consolidating the knowledge acquired during the project, and preparing the technology we developed to release it to the community.

Thank you!


Here are links to the real-time videos of Team Delft’s performance:



There are also a number of articles in our media section.

Read More →

Congratulations to Team Delft

Team Delft celebrates their victory in the Amazon Picking Challenge.

Team Delft celebrates their victory in the Amazon Picking Challenge. © RoboValley

Team Delft won the Amazon Picking Challenge 2016 in Leipzig yesterday! Congratulations to the team! The competition was part of the RoboCup 2016, an international robot competition with different categories.

The Amazon Picking Challenge consists of two parts: In the picking challenge a set of items from the Amazon product range needs to be picked from the shelf and placed in a bin. For the stowing challenge it was the other way around. Team Delft, which consists of members from TU Delft and the company Delft Robotics, won both challenges: the picking and stowing finals and convinced with their speed and precision. The team won 50,000€ as prize money.

Team Delft won the stow task finals by collecting 214 points. Second came NimbRo Picking (186 points) and the team from MIT ended third (164 points). In the picking finals, the competition was so close that the judges had to resort to the second tie-breaker using video replay to determine the winner. Both Team Delft and PFN finished with 105 points, but Delft achieved their first pick in a mere 30 seconds, beating PFN’s 1:07 time.

Team Delft built a flexible robot system based on industry standards. The system is equipped with an Yaskawa robot arm with seven degrees of freedom, high-quality 3D cameras and an in-house developed gripper. To control the robot, the team integrated advanced software components based on state of the art artificial intelligent techniques and robotics. The components are developed with the Robot Operating System for industry (ROS-Industrial), and will be released as open software.

Although the development of the system still took well over 1 day, we are still proud of the achievement!

Prof. Martijn Wisse, coordinator of Factory-in-day commented the success of his team: “As part of the Factory-in-a-Day project, partners TU Delft and Delft Robotics BV collaborated to participate in the Amazon Picking Challenge. And we won! The Picking Challenge represents the high degree of variability that is needed for robots in the quickly-changing SME environments. By participating, we proved that we could install the system within a few hours after arrival. We also demonstrated the integration of a number of techniques, such as deep learning, 3D perception, motion planning, and robust task execution, all thanks to the ROS-Industrial framework. Although the development of the system still took well over 1 day, we are still proud of the achievement: we had possession of the Yaskawa Motoman robot only less than two months before the date of the Challenge, yet we managed to outperform all the other teams in both of the (quite separate) contests. We can call ourselves the best bin-pickers and bin-stowers of the world!”

Here is a video (thanks to RoboValley for making them) of the final of the stowing tasks:

and the picking tasks:

There is also a nice article on the website of RoboValley about the winning team:


Read More →

Team Delft at Amazon Picking Challenge

This afternoon Team Delft is finally able to demonstrate their skills in the Amazon Picking Challenge. Today, the teams have to  are the  to pick up items from the bin and put them in the shelf, referred to as the stow task finals.

We’ll cross our fingers!

Here is a video from their test trials yesterday:





Read More →

Amazon Picking Challenge – the countdown has started

Team Delft ready for the Amazon Picking Challenge.

Team Delft packed their robot up and left for Leipzig today as the Amazon Picking Challenge is approaching. The competition starts on June 30. In the Amazon Picking Challenge the teams have to pick 12 specified target items from the shelf into a tote in 20 minutes. These items are all standard kinds of Amazon products, such as books, cubic boxes or clothing.

This is of course easy for every human being, but still a challenge for a robot. Team Delft, one of 16 competitors this year, consists of 16 people from TU Delft and the company Delft Robotics.

The team is using a Yaskawa robot arm mounted on a rail. The 8-degrees of freedom provide the required maneuverability to reach all locations in the shelf. To identify and locate the target products on the Amazon Shelf, the team uses a 3D camera with a Deep learning vision system.

The gripper was developed through quick prototyping and testing, sucking up most of the products. For objects which don’t working by pure suction, there are mechanisms for more complex grasping. The ROS-Industrial based software is being developed and integrated in simulation thanks to the available ROS tools.

We cross our fingers for the team and kep you posted!

More details on

Follow the team on Twitter:


Read More →

Pioneer of Collaborative Robots Launches Universal Robots+

Here is a press release by our partner Universal Robots on a new system, which was officially introduced today at Automatica trade fair in Munich:

Universal Robots+. The early prototype of this system was a result of Factory-in-a-day, which was then subsequently developed by Universal Robots.

Universal Robots, the Danish pioneer of human-robot collaboration, revolutionizes the automation industry with the launch of Universal Robots+ at AUTOMATICA in Munich. Today, the company presents:

  • Universal Robots+: A showroom of Plug & Play application solutions offering a new level of simplicity for companies that want
    copyright: Universal Robots

    copyright: Universal Robots

    to hit the ground running when installing their next UR robot application;  By choosing accessories, end-effectors, and software solutions from Universal Robots+, both distributors and end users get high security and predictability that applications will run well from the start, saving weeks and months in the integration process from concept to operation of the UR cobots.

  • +YOU: a unique, free-of-charge developer program, offering a powerful marketing and support platform for the flourishing eco-system of UR-robot application developers.

With Universal Robots+, the company creates its own eco-system enabling applications revolving around the collaborative robot arms UR3, UR5 and UR10 to be developed and showcased. The so-called URCaps, accessory components that extend the UR robots’ capabilities, can be customized hardware components, software plug-ins, or a combination of both. The aim of Universal Robots+ is to reduce implementation periods, increase user comfort and reduce costs for all parties involved.

“With Universal Robots+, we create an unmatched win-win-win situation benefitting the developer community, our distribution partners and our end customers,” explains Esben H. Østergaard, CTO and co-founder of Universal Robots. “The participants in the developer program +YOU will receive free support from Universal Robots when developing URCaps. By integrating the accessory components showcased at the Universal Robots+ showroom, our distribution partners and end users reduce spending on application development and testing when they deploy the URCaps as simple Plug & Play solutions. In short, Universal Robots+ is easy access to efficient, well-proven, and safe automation solutions that we elevate to the next level by collaborating with a crucial component: the developers of today and tomorrow.”

The full press release is here:

Read More →

Blog entry from ROS-Industrial about RoboBusiness Europe 2016

We would like to hint at an article from our Partner Fraunhofer IPA, who is also leading the ROS-Industrial Consortium Europe. The RIC-EU Program Manager, Dr. Mirco Bordignon, writes about RoboBusiness Europe:

ROS-Industrial is an open-source project that extends the advanced capabilities of ROS software to manufacturing.

Read More →

RoboBusiness 2016: last test for the Amazon picking challenge


The Factory-in-a-day booth at the RoboBusiness trade fair in Odense/Denmark.

Factory-in-a-day presented part of the project’s work at the RoboBusiness 2016. At the booth, Team Delft and Team Applied Robotics tested their robotic systems for the upcoming Amazon Picking Challenge. 

Especially Team Delft – from TU Delft the coordinator of Factory-in-a-day – uses many technologies developed in our  project. Since picking and placing objects is an important factor in many production steps, SMEs willing to robotize certain work tasks need a reliable system for this. The Amazon picking Challenge is a good way to improve the system further and test it in a naturalistic scenario.

A couple of weeks before the competition at the RoboCup (Leipzig, June 28 – July 1), the two teams used the event to test their systems for last bugs and necessary improvements.

Both teams – the only two from the Netherlands among the 16 finalists – need to pick up a number of objects from the shelves used in the Amazon warehouses.  At the end of June, this needs to be done in 20 minutes, at the RoboBusiness event, we performed a mini challenge with picking up of 6 items in less time.

The booth was not the only presence of Factory-in-a-day at the event. Prof. Martijn Wisse gave a presentation on “Increasing the Application of Robotics and automation in EU”, Dr. Mirco Bordignon (Faunhofer IPA) talked about “ROS-Industrial: bringing open-source robotics innovation to the factory floor” and Prof. Jouke Verlinden (TU Delft) on “How Robotics and 3D Printing Enable us to Think Outside the Box”.

Have a look at our pictures on Facebook.


Read More →

Impressions of RoboBusiness 2016


The Factory-in-a-day booth with the set-up for the mini picking Challenge – left team Delft, right Team Applied Robotics.


Kanter van Deurzen explains the rules of the Amazon Picking Challenge competition, which we carried out at the booth.


Team Delft watching their robot perform in the Mini Amazon Picking Challenge competition at RoboBusiness 2016.


The crowd watching the performance of the two teams at the booth of Factory-in-a-day.


Visitors at the booth of Factory-in-a-day.


Prof. M. Wisse, coordinator of Factory-in-a-day during his presentation on “Increasing the Application of Robotics and automation in EU”


Dr. Mirko Bordignon, Fraunhofer IPA, presenting on “ROS-Industrial: bringing open-source robotics innovation to the factory floor”.


The two teams competing in the Amazon Picking Challenge: Team Delft (in black shirts) and Team Apllied Robotics (on the right).

Read More →