After moving the automated robotic arm on the car assembly line to the department store, how to teach it to neatly handle the palletized goods like playing Tetris?

For a long time, the automated robotic arm industry has been known as the “Four Families”: Japan’s FANUC, Yaskawa Electric (YASKAWA), Switzerland’s ABB, and Germany’s KUKA, these four top industrial robot companies, in It occupies more than 50% of the global market share, and has its own strengths in key fields such as motion control, automation, and servo motors.

In the retail scene, the application of industrial robots is also in full swing. Technology companies such as Amazon, Walmart, and Meituan have launched unmanned delivery robots, which have begun to take shape in contactless scenarios.

However, it will be a huge task to move the industrial robotic arm originally used for precision processing into the warehouse of retail department stores, so that it has the ability to make independent decisions and execute in complex scenarios and help operators flexibly sort fresh department stores. challenge. Although many companies in the industry are working in this direction, solutions with real commercial potential have not yet emerged.

Starting this year, the Visual Perception and Intelligent Systems Laboratory of Shandong University and Meituan have jointly launched a scientific research project. Precise identification and sorting of the department store merchandise on display.

This breakthrough in applied scientific research oriented to practical scenarios will further meet the retail industry’s blueprint for future warehousing automation logistics.

01“Three years to sharpen a sword”, train a smarter robotic arm

In 2021, the output of China’s industrial robots will reach 366,000 units, an increase of 68% over the previous year; the output of service robots will be 9.214 million units, an increase of 47% over the previous year. China has become the world’s largest application market for robots.

Especially in industrial assembly lines such as automobile assembly and welding, industrial robots represented by automated robotic arms have been successfully applied and have a highly mature market. Most of these production environments are single scenes and fixed trajectories. It is only necessary to calculate the fixed trajectory of the robotic arm to move, and it can be put into use after deployment.

But in warehousing and logistics applications, robotic arm systems have to deal with complex scenarios that contain a variety of unpredictable objects—for example, when faced with a table full of messy goods, most robotic arm gripping systems may fail. Even if the grab is successful, in the next step of placement, there may be collisions between different categories and different specifications of goods, making it difficult to place them neatly.

For scientific researchers, the ideal way is to teach the robotic arm to “use both hands and eyes” by equipped with visual recognition technology and control decision-making algorithms, so that the robotic arm can accurately identify and classify goods without human intervention. The sorted goods are neatly stacked to maximize the packing rate.

It is not easy to train such a “smart” robotic arm. Faced with a pile of messy goods, how does the robotic arm know which one to grab? How to grab it? Where are they stacked?

Focusing on these key issues, the research team of the Laboratory of Visual Perception and Intelligent Systems of Shandong University proposed an autonomous perception and decision-making algorithm for the robotic arm, and completed the realistic version of the “Tetris” task – the robotic arm can recognize objects with only one picture. properties, and determine the grab and placement position. The robotic arm changed from “clumsy” to “smart” through self-learning, and finally achieved “accurate grasping and balancing” in the complex task of the real version of “Tetris”.

Through the algorithm upgrade, the researchers focused on the identification and efficient sorting of disorderly stacked commodities, as well as the optimization of three-dimensional online palletizing strategies

At present, the research results have been published in ICRA, the top international conference in the field of robotics, and have been authorized by the national invention patent. After the idea of ​​algorithmic innovation in perception and decision-making has been verified, the next challenge is to make breakthroughs in cutting-edge technologies truly land, accurately identify and sort a wide variety of goods in real warehousing scenarios, and use higher Packing rate for stacking.

Focusing on this goal, the Shandong University team has upgraded the algorithm, focusing on the identification and efficient sorting of disorderly stacked commodities and the optimization of three-dimensional online palletizing strategies.

The “evolutionary” robotic arm has become smarter: on the one hand, it can accurately identify various commodities such as milk and beverages in messy logistics boxes and efficiently sort them; ”, orderly stack the goods of different sizes and specifications together with the optimized packing strategy. This achievement was published in the top international conferences in the field of robotics such as IROS and CoRL, and received attention from academia and industry.

Under the autonomous perception and decision-making algorithm, the robotic arm can identify the attributes of objects and judge the grasping and placement positions with only one picture. 

It is understood that the Shandong University team has invested more than 3 years of research and development in this project, and its self-developed intelligent robotic arm is now capable of being put into actual production, and will undoubtedly play an important role in the construction of a fully automated logistics system for warehousing.

02 From scientific research to application, introduce technology into the warehouse site

Applying scientific research to specific business scenarios is the last link in the realization of science and technology, and it is also a daunting task.

After deploying the automated robotic arm in a real warehousing scenario, it is faced with thousands of SKUs of different categories and specifications. How to achieve higher applicability and plan a reasonable location for each commodity in a complex environment Realize high-precision palletizing and packing?

This is not only a challenge faced by researchers, but also a problem that the industry needs to solve urgently. The purpose of jointly launching a scientific research project in Meituan and Shandong University is to enable the intelligent robotic arm to be used in various scenarios such as medicine and fresh food, and to meet the requirements of related indicators such as accuracy and efficiency.

The demand for this school-enterprise joint project comes from the smart pharmacy. In May of this year, Meituan Maiyao cooperated with offline pharmacies such as Neptune Xingchen to launch the first 24-hour smart pharmacy in Beijing. Through visual recognition, automatic control and other technologies, it solves the problem of inconvenient consumers taking medication at night.

Specifically, after consumers place an order through the Meituan App, the automatic sorting equipment can quickly pick up the order according to the order, and the ordered medicine will be transported to the packaging module through the robotic arm, automatically packaged and printed, and placed in the self-service delivery cabinet. , the consumer can pick up the product by entering the pick-up code.

During the operation of the project, Meituan’s engineering team decided to take on a more difficult challenge-currently, drug sorting relies on XY-axis robotic arms to sort on a fixed trajectory, and uses visual recognition algorithms to identify goods. But going further, it requires a full-process automation solution that integrates sorting, replenishment, and inventory.

In the 24-hour smart pharmacy where Meituan Buying helped Neptune land, modular technologies such as automatic sorting, automatic packaging, and self-service handover have been applied

“We hope to jointly develop a set of overall solutions based on visual recognition + robotic arms. In addition to dispensing medicines, it also automates replenishment, inventory and other links.” Ma Lin, a senior researcher at the Visual Intelligence Department of Meituan, believes that if This technological breakthrough can be successfully verified in this scenario, and can also be applied in warehousing scenarios such as fresh food and department stores in the future.

Ma Lin said that Meituan and Shandong University hope to develop multi-axis and multi-tooth robotic arms that are more suitable for specific scenarios through joint research. In the fresh department store scene, there will be products with all kinds of strange shapes, which requires more advanced end fixtures to handle.”

In the vision of both parties, this softer and smarter robotic arm will not only be used in smart pharmacy scenarios, but also in warehousing scenarios of fresh food and department store retail in the future to further reduce manual sorting and stocking. The cost of replenishment.

This will also become the starting point of a fully automated logistics system: in warehouse sorting, the robotic arm accurately sorts the goods from the tray, and dismantles the zero according to the consumer order, and transports the order directly to the designated location through the AGV (automatic guided transport vehicle). Packing, and finally complete the “last mile” distribution at the end through drones and automatic distribution vehicles.

Professor Zhang Wei, head of the visual perception and intelligent system team, believes that this joint research and development project is of great value. Meituan can provide a wealth of verification scenarios for the laboratory’s cutting-edge technologies, and has made clearer proposals for robotic arm sorting, packing and palletizing. application requirements. This is also a process of exploratory research. It is hoped that through school-enterprise cooperation, a set of practical and effective solutions can be polished together and bring some inspiration to the industry.

03 From vision to hardware, ask the upstream supply chain for answers

In fact, for the leading visual intelligence department of Meituan, this interdisciplinary applied research, spanning from software to hardware, is somewhat “out of line”.

“We not only need to do the algorithm part, but also build a complete set of solutions that can be used to achieve fully automated operations from identification to sorting.” Marin said that smart sorting is an important part of the real-time retail scene. The first step is to achieve more efficient and accurate sorting of goods to meet consumers’ “faster” requirements.

Before cooperating with colleges and universities, this innovative team has done many rounds of research on mainstream industrial robots on the market.

It is understood that robots are “the jewel at the top of the manufacturing crown”. At present, China has become the largest application market for robots in the world. However, in the field of global industrial robots, Japan’s FANUC (FANUC), Yaskawa Electric (YASKAWA), Switzerland’s ABB, Germany’s KUKA (KUKA) “four families” still occupy more than 50% of the international market share. In motion control, automation, servo motors and other fields are in a leading position.

Taking an imported industrial robot as an example, a six-axis robotic arm with a load of 12KG, the repeat positioning accuracy can be controlled within 0.04mm, but the price is at least about 150,000 yuan. If this cost input is measured according to the standards of the retail industry, it is obviously Excessive.

In addition, a more realistic problem is that most of the automated robotic arm solutions provided by the international “Four Families” or the domestic industrial robot manufacturers that are catching up are mostly used in the industrial field, entering the retail and warehousing fields. Its versatility is not high, and it is difficult to customize it according to specific scenarios.

“In retail application scenarios such as medicines, fresh food, and department stores, the cost of customized solutions is very high.” Marin explained that if a standardized solution is used for transformation, it may not be suitable for retail department stores, “such as logistics sorting machinery. The arm needs to take up a lot of space and must be closely matched with the production line. Behind the adaptation, it also involves business logic adjustment, and it is difficult to have a universal solution.”

For example, in retail application scenarios, there are a wide variety of commodities with different attributes, which also puts forward higher requirements for the end clamp of the robotic arm. It is understood that the current common manipulator end clamp solutions in the industry are mainly divided into suction cup type and multi-tooth type. The former relies on vacuum suction cups or magnetic suction cups to “suction” objects, mainly for objects with regular and flat surfaces; the latter It simulates human fingers, and uses multi-tooth mechanical clamps to flexibly “clamp” objects, which can be applied to irregular objects.

“At present, the end clamps on the market are similar, and most of them are geared towards general grabbing. In retail application scenarios, a series of factors such as the geometry and physical properties of commodities often need to be considered.” Marin said, for example, if multi-tooth clamps are used, how to clamp From a narrow mouth water bottle? How to control the strength to avoid breaking when clipping fruit? These are all challenges in real-world applications.

To this end, in the process of cooperation with Shandong University, the two parties plan to “magically modify” the hardware design and decision-making algorithm of the robotic arm according to the needs of the actual scene, and use self-research capabilities to meet specific needs and ask the upstream supply chain for answers.

According to the preliminary estimates of both parties, by further transforming fixtures and sensors, and taking the lead in self-developed products that meet the accuracy requirements of the real production environment, the cost can be reduced by at least 60%. If the project is developed smoothly and the performance and accuracy errors are controlled within a reasonable range, the overall cost of a set of solutions is expected to be controlled within 100,000 yuan. If it can be formally put into mass production in the future, the cost will be further reduced.

It is understood that in the process of technical research, the Shandong University team fully considered the economic problems when the product was put into commercial use, strictly controlled the cost, and tried to use low-cost domestic sensors to replace foreign high-end sensors. To make up for the sensor accuracy gap, the Shandong University team is also exploring error analysis and compensation strategies. In simple terms, it is to add sufficient redundancy to the policy model in visual recognition and action decision-making, and to correct the accuracy during actual grasping.

At present, this technology has been verified in the international academic community. Using augmented reality (AR) technology, the average error in aligning the robotic arm is about 10.54 ± 4.32 mm, which is lower than manual alignment without AR assistance (up to 19.62 mm), according to a data from the REDS laboratory at Imperial College London. , indicating that AR visualization significantly reduces the error of manual debugging.

With the emergence of the wave of instant retail, consumers are increasingly demanding “faster delivery”, and technological innovation breakthroughs in the retail industry are extremely important. At present, many technology companies, including IBM, NVIDIA, and Microsoft, are also actively participating in technology research and development in the retail industry. Fusion products based on artificial intelligence, machine learning and other technologies have been put into use in many retail companies at home and abroad.

Under the habit of shopping in the shortest 30 minutes, global retail companies have an urgent need for technological innovation. By introducing new technology applications such as automatic distribution and robotic sorting, they do their best to improve supply efficiency. Take the American supermarket chain Kroger as an example. It has established a fully automated contract fulfillment system and a micro-fulfillment site called “Zoom” in Florida, in cooperation with the technology company Ocado. , the performance time limit can be compressed to 30 minutes during the trial operation.

In the foreign retail industry, warehousing robots, automatic sorting, etc. have become hot technical applications

The leap of the retail industry in the future will depend on the driving force of technology. Professor Zhang Wei believes that the current research on the intersection of artificial intelligence and robotics is turning to be driven by scenarios, “especially as applied research, we should actively seek changes. In the past, we customized scenarios for verifying algorithms, but now we are developing algorithms for scenarios. Only by doing research from an application perspective and being problem-solving oriented can we help companies and industries solve their pain points.”