Teaching Robots to Harvest Tomatoes Without Touching a Tomato

Researchers in Japan are building virtual tomato farms that could dramatically accelerate agricultural robotics

One of the biggest challenges in agricultural robotics is surprisingly simple: collecting data.

Before a robot can learn how to identify, approach, and harvest a tomato, researchers must gather thousands of images from real greenhouses. Those images then need to be labeled by humans, identifying fruit locations, stems, leaves, and obstacles. The process is slow, expensive, and must often be repeated whenever growing conditions change.

Researchers at Osaka Metropolitan University believe they have found a way around that problem.

In a recently published study, the team developed a method for automatically generating synthetic training data by recreating tomato-growing environments inside a virtual world. Rather than spending months collecting and labeling images in greenhouses, researchers can generate large volumes of realistic training data automatically and use it to train AI models for harvesting robots.

The approach represents a growing trend in robotics: teaching machines in simulation before deploying them in the real world.

Process 1 (P1) involves data acquisition in a real environment, including image capture by a tomato-harvesting robot and the reconstruction of 3D models using 3DGS. Process 2 (P2) involves model editing and placement in a virtual environment. Process 3 (P3) involves automatic dataset generation and transfer learning using YOLO.

Why Tomatoes Are So Difficult

Tomatoes might seem like an ideal target for agricultural automation. They are high-value crops grown in controlled greenhouse environments.

In practice, however, tomato harvesting remains one of the most difficult robotic tasks in agriculture.

Plants continuously change shape as they grow. Fruit may be partially hidden behind leaves. Lighting conditions vary throughout the day. Multiple tomatoes often overlap within a cluster, making it difficult for vision systems to determine which fruit is ripe and how it can be safely harvested. These challenges have limited the commercial deployment of tomato harvesting robots despite years of research.

For robots, agriculture is often the opposite of manufacturing.

A factory robot may encounter the same object in the same location thousands of times. A harvesting robot encounters a slightly different plant every few seconds.

Placement of 3D models in a virtual environment: (a) placement of fruit and stem, (b) imported as separate models, (c) imported as a single model.

Building a Virtual Greenhouse

The Osaka team tackled the problem by constructing realistic digital tomato plants based on actual greenhouse imagery.

Their system can automatically generate synthetic scenes containing tomatoes, stems, leaves, and the visual complexity that agricultural robots face in real production environments. Because the environment is virtual, every object is already labeled. The AI training process can therefore be automated without requiring human annotation.

The result is essentially a training arena for harvesting robots.

Instead of waiting for plants to grow, researchers can create thousands of virtual tomatoes in minutes.

Instead of manually labeling images, the system generates perfect annotations automatically.

And instead of conducting every experiment inside a greenhouse, researchers can evaluate algorithms virtually before moving them into physical systems.

The Rise of Digital Agriculture

The work is part of a broader movement toward digital twins in agriculture.

Researchers around the world are increasingly creating virtual representations of crops, greenhouses, and robotic systems. China has developed digital-twin tomato harvesting systems to evaluate harvesting strategies. Wageningen University & Research is building simulated greenhouse environments to accelerate robot development. Osaka Metropolitan University has also explored digital twins for optimizing robot approach angles and teleoperating harvesting robots through virtual reality interfaces.

The concept mirrors trends already seen in manufacturing and autonomous vehicles.

Before a self-driving car hits the road, it may drive millions of simulated miles.

Agricultural robotics appears to be heading in the same direction.

The Bigger Opportunity

What makes this research particularly interesting is not simply that it helps robots recognize tomatoes.

It changes the economics of robotics development.

Collecting real-world agricultural data is expensive. Greenhouse access is limited. Growing seasons are finite. Researchers must often wait months to test new ideas.

Virtual environments compress those timelines dramatically.

A robot can experience thousands of harvesting scenarios in simulation before ever entering a greenhouse. Failures become cheaper. Experiments become faster. New AI models can be evaluated at a scale that would be difficult or impossible in physical environments alone.

For an industry facing labor shortages, aging workforces, and increasing pressure to improve productivity, that acceleration could prove significant.

The future tomato harvester may still be a physical robot navigating rows of plants.

But increasingly, its education may happen inside a virtual greenhouse first.