Autonomous Fruit Picking Machine: A Robotic Apple Harvester

Summary. This paper describes the construction and functionality of an Autonomous Fruit Picking Machine (AFPM) for robotic apple harvesting. The key element for the success of the AFPM is the integrated approach which combines state of the art industrial components with the newly designed flexible gripper. The gripper consist of a silicone funnel with a camera mounted inside. The proposed concepts guarantee adequate control of the autonomous fruit harvesting operation globally and of the fruit picking cycle particularly. Extensive experiments in the field validate the functionality of the AFPM.

1 Introduction

The use of robots is no longer strictly limited to industrial environments. Also for outdoor activities, robotic systems are increasingly combined with new technologies to automate labour intensive work, such as e.g. apple harvesting . This paper describes the feasibility study for and the development of an Autonomous Fruit Picking Machine (AFPM)3 .

There are two main approaches in robotic apple harvesting being bulk or apple by apple harvesting .

Peterson et al. developed a mechanical bulk robotic harvester for apples grown on narrow, inclined trellises. This type of bulk harvesting requires, in addition to the canopy-like growth habit, uniform fruit ripeness at harvest, firm fruit, resistant to damage, and short/stiff limbs .

The use of an apple by apple picking system, although inherently slower, does not suffer from any of the above restrictions. Moreover, only apples of satisfactory size and maturity are selected for harvesting and can be sorted out immediately. An apple by apple picking system does, however, requirean adequate fruit gripper. The gripper is the key element in the success of automated apple by apple harvesting. A good gripper ought to preserve the quality of the apple and should not damage the tree (nor the apple) during the picking cycle. Setiawan et al. propose a low cost gripper with flexible inflatable parts. Our gripper consists of a flexible silicone funnel and uses suction to pick the apple.

An indispensable part of any autonomous apple harvesting machine is the vision system used to locate the apple. Bulanon et al. use a (RT) machine vision system to recognize the location of the fruit centre and the abscission layer of the peduncle. In contrast to most researchers, in our approach the camera is positioned in the centre of the gripper. This simplifies the calibration of the set-up and ensures adequate control.

According to the classification given by Hutchinson et al. , our image based control is closer to a look and move strategy than to visual servoing. There is, however, margin for improvement.

Despite previous developments towards a harvesting platform navigating autonomously between orchard rows, we chose not (yet) to implement an automated navigation, for reasons of both development time as well as safety approval.

The aim of this project was to prove the feasibility and demonstrate the functionality of an Autonomous Fruit Picking Machine (AFPM), by using existing state of the art (industrial) components. This resulted in the prototype AFPM, described in the following sections. First, section 2 describes the overall construction of the AFPM. Section 3 presents our new, patented gripper designed specifically for the apple harvesting task. Section 4 goes into the control details for one picking cycle. Finally, sections 5 and 6 summarize the results of the field experiments and conclude this paper.

2 Overall construction of AFPM

The AFPM is built on a platform mounted behind an agriculture tractor. Figure 1 shows the first version of the AFPM. Figure 2 illustrates schematically the functional layout and data flow. In order to reduce the development period of the AFPM, although overkill, an industrial robot (Panasonic VR006L) is chosen as manipulator. The AFPM further consists of a tractor-driven generator for power supply, a (2D) horizontal stabilization unit, a 7th external vertical axis to enlarge the operation range, a safety scanning device, a central control unit, a touch panel PC with Human Machine Interface, a canopy and all around curtain to even out light conditions and, finally, a fruit gripper designed specifically for this task, with a camera mounted in the centre of the gripper. The flexible gripper (described in section 3) guarantees a firm grip without damaging the fruit and serves in fact as the mouth of a vacuum cleaner.

Fig. 1. Construction of the AFPM

Fig. 2. Functional scheme of the AFPM

The 2D horizontal stabilization consists of two hydraulic feet and one turn over cylinder, configured as a 3-point suspension. Controlled by two level sensors, the system ensures a stable positioning of the robot platform during the picking cycle.

3 New flexible gripper

To pick the apple with as little energy as possible, a completely new fruit gripper had to be designed. Its patented design has two main functions: catching the apple and enclosing the camera. Figure 3 gives some examples.

The gripper assumes the shape of the apple and encloses it firmly. After several prototypes, the optimal funnel shape with respect to edge thickness, funnel angle and size with a trade off between flexibility and firmness was designed and tested. The current version has a maximum diameter of 10.5 cm. The gripping function is activated by vacuum suction. It is e.g. possible the pick up an apple by just pushing the gripper onto the apple and closing the vacuum port with your finger. Experiments show that even an elongated exposure of the apple to the (rather small) vacuum levels4 used does not damage the apple in any way.

Fig. 3. Left: two examples of silicone gripper; right: gripper mounted on robot with camera inside

Placing the camera in the centre of the gripper offers numerous advantages. First of all, the gripper is always in line with the camera and thus with the image, which simplifies the (necessary) coordinate transformation from image to robot. Furthermore, the position of the camera is fully controllable. The camera can always point its optical axis to the apple (see section 4 and simulation figure 7). This reduces image distortion and eliminates the necessity for thorough calibration.

Fig.4. 3D, top and front view of the camera model defining the camera frame and illustrating the computation of the rotation angles θx and θy, needed to centre the apple in the image

4 Approach for fruit picking cycle

The autonomous harvesting operation is hierarchically structured in three levels. Once the AFPM is stationed in front of the tree with active stabilization (first level), it scans the tree from 40 look-out positions or sectors (second level). For each sector, all ripe apples are listed and picked one by one in a looped task (third level). A picking operation consists of following steps:

1. The position of the apple in the image, possibly after declustering, is determined. Only ripe apples with qualified size are selected.

2. The camera rotates around x- and y-axes, by θx and θy respectively, in order to point the optical axis straight to the apple. Positioning the camera by only rotating the wrist results in a small and therefore fast robot movement. Figure 4 defines the used set-up. The rotation angles yield:

                                (1)

for the rotation around the x-axis and

                                 (2)

For the (simultaneous) rotation around the y-axis, with f the focal length [mm], µp the pixel-size [mm/pix] and xp,yp the measured centre of the apple in the image plane [pix]. As equations 1 and 2 show, the rotation angles do not depend on the distance to the apple zcam. Only the focal length f needs to be calibrated.

Even if the focal length is not exactly known, the final offset of the centre of the gripper with respect to the centre of the apple will lie within the margin of the gripper: e.g. a (rather large) error of 10% on the focal length causes an error of 1.5 o on θx (or θy) resulting in an offset of 1.9 cm for an apple at 1 m distance. Due to the funnel-like design of our gripper, this offset will not cause a faulty picking cycle.

Fig. 5. Illustration of the remaining distance calculation

Fig. 6. Schematic overview of the control flow between vision system, PLC and
robot during one apple-picking cycle

3. While approaching the apple, several images are processed to calculate by triangulation the remaining distance to the apple. Given the distance Z, travelled between two images, and xp1, xp2 the measured radii of the (corresponding) apple from those two images, the remaining distance ∆Z to the apple yields.

                                   (3)

Figure 5 illustrates equation 3. Since the apple remains centred in the image, the correlation of a given apple in subsequent images is trivial.

4. Once the apple is within a given range of the gripper, the vacuum suction is activated. The detection of (a small level of) vacuum triggers the next step.

5. The apple is picked by rotating it and tilting it softly and is then put aside. The actual wrist/robot movement differs from section to section. It is programmed and optimized in advance with a minimum tree intrusion in mind. A standard apple transporting and collecting system still needs to be added.

Figure 6 gives a schematic overview of these 5 steps. The indicated numbers correspond to the step list for the picking cycle. The image processing is conducted on an industrial pentium IV 2 GHz PC with 1 GB RAM under Windows. The actual image processing5 is programmed with Halcon software [8] and takes about 0.6 seconds (for step 1). Note that the complete image processing is only needed at the start of a picking cycle. Subsequent image processing should be limited to a small portion of the image and scaled down to less functional steps.

5 Field experiments

Field experiments conducted over a period of several weeks in a Jonagold orchard show that the overall performance of the AFPM is more than satisfactory. The stabilization experiences no difficulties with possible rapid robot motion. Thanks to the roof and all-around cover, as shown in figure 7, the AFPM works even under poor weather conditions, although also here improvements are still possible.

The experiments demonstrate that about 80% of the apples (with diameter range from 6 cm to 11 cm) are detected and harvested. However, stem pulls of 30% are still too high. However, on should take into account the fact that the apples were treated to prolong the harvest period, causing a more firm connection of the stems to the limbs. Nevertheless, fine tuning step 5 in section 4 may lower this fault margin.

Apart from stem pulls, the apples show no bruises or damage at all. Apples not harvested are either not detectable by the vision system or not reachable for the robot manipulator. This problem can be partially solved by trimming the orchard rows into smaller hedges.

Fig. 7. Left: simulation model; right: field experiments

In the current setup the overall cycle time to pick one apple is 8 to 10 sec. Eliminating the current communication bottleneck between the vision system and the robot through the central controller ((trigger) signals in figure 6) will improve the overall picking cycle time. Extra care should be taken in selecting the first apple in a cluster in order not to push off other apples. Additional vision information about nearby sharp limbs could also help to protect the gripper itself from damage.

6 Conclusion

The first results of the AFPM are very promising. All the necessary components are fitted together and operate as planned, hereby proving the feasibility and functionality of the AFPM. There is, however, still margin for improvement. The bottleneck in communication lies with the connection/communication between the vision-PC and the central control unit. Future work will focus on improving the bandwidth of this connection and on optimizing the image processing. The aim is to reduce the picking cycle period from an average of 9 to about 5 seconds (or less). In that case, the productivity of the AFPM will be close to the work load of about 6 workers, which makes the machine economically viable.

Special attention will go to improving the actual apple detach movement by taking into account the relative pose of the apple with respect to the branch. This should lower the stem pull percentage. Further research is also planned to automate the navigation trough the orchard and to investigate the harvesting of pears.