In recent years, AI leaders have urged machine learning experts to consider understanding the world’s oceans and tackling climate change grand challenges on par with building autonomous vehicles, beating a computer at a game of chess, or robotic grasping.

Combining computer vision, logistics, robotics, and the science of botany, indoor farming has the potential to change human lives. But innovation in this area requires considering dozens of variants and doing more with less. These challenges are likely among the reasons companies like Intel, Microsoft, and Tencent have participated in experiments to automate greenhouses.

Bowery Farming may be the largest vertical farming company operating in the U.S. today. Founded in 2015, the company has introduced a number of major changes in recent weeks. These are part of its largest expansion since raising more than $170 million from investors including GV (formerly Google Ventures) and individuals like Uber CEO Dara Khosrowshahi.

One of those changes was giving the CTO role to former Google VP Injong Rhee, who had worked on IoT platforms for Google like the edge TPU AI chip and software and services for Samsung. Last month, Bowery also opened what it calls a “center of excellence” in New Jersey. Called Farm X, the center will focus on raising not just leafy greens like the kind Bowery sells today, but also cucumbers, root vegetables, strawberries, and tomatoes. This initiative will focus on research and development, functioning as a sandbox for considering the possible blends of seeds and conditions required to grow produce indoors. A Bowery spokesperson declined to share how much square footage is devoted to the project, but a statement from the company describes Farm X as “one of the largest and most sophisticated vertical farming R&D facilities in the world.”

The first two Bowery farms are located outside Baltimore and New York City. An additional commercial farm is scheduled to open in late 2021 in Bethlehem, Pennsylvania, about 70 miles from Philadelphia. Bowery claims its facilities are currently 100 times more productive than traditional outdoor farming methods. By adding operations in the Pennsylvania area, the company plans to serve a population of nearly 50 million along the Eastern seaboard of the United States. The goal, Bowery said, is to build indoor farms near every major city in the United States and the world.

Above: Bowery grow room near Baltimore

A number of companies want to crack the code of supplying vegetables to people in urban environments, reducing the need to truck produce into cities. Growing Underground, for example, occupies a World War II-era bunker in London. In Singapore, a company is exploring how to create indoor farming operations that can fit inside a shipping container as the country seeks greater food independence in the face of accelerating climate change and reduced global food supplies.

Scientific progress in indoor plant cultivation could play a key role in addressing food deserts and food security as climate change intensifies. Such knowledge could also advance further exploration of the Moon and Mars. But Bowery chief science officer Henry Sztul says his company is focused on growing produce at scale in monoculture environments in larger facilities.

Several indoor farming startups in the U.S. are currently dedicated to providing premium organic lettuce for customers at Whole Foods and other high-end grocery stores. Bowery, for example, sells to nearly 1,000 stores on the East Coast of the United States, including Amazon Fresh, Walmart, and Whole Foods, as well as ecommerce vendors. Bowery didn’t share specifics when we asked how sales are currently split between Whole Foods and Walmart. But Rhee said the goal is for advances in efficiency to result in more affordable, high-quality produce.

“The AI is still in its infancy, and there’s a lot of human touch that’s still needed to get it mature. But I think with the help of AI, what we get out of this is so amazing that we can actually drive what perceivably in the past was not economically viable,” Rhee said. “Now we’re bringing it to be economically viable, and that’s really the power of the AI and machine learning that makes that happen.”

VentureBeat sat down with Rhee and Sztul to talk about what they consider the holy grail of machine learning challenges for indoor farming, the specific challenges smart indoor farming companies encounter, and the idea of polyculture gardens with multiple types of plants growing in the same place.

This interview has been edited for brevity and clarity.

VentureBeat: In 2019, while speaking to reporters, Amazon VP of devices David Limp called a particular advance in Alexa tech a “holy grail of voice science.” What’s the holy grail of indoor farming, in terms of the machine learning in this space?

Injong Rhee: I think the scale of doing this is one thing. Another thing is this whole process of growing crops from seed all the way to the harvest, packing them, and then delivering them to the store. That entire life cycle of crop and then supply chain presents so many opportunities for optimization.

And really making this indoor farming and vertical farming popular or economically viable is one optimization, but there are so many different dials and levers that we have to optimize, and AI is the best method to do this multi-variable optimization across the space. It’s really emulating what farmers do, and then what the trucking companies do, a combination of all of them, and then making the machine actually do all of this in a much more optimized way to make this really economically viable to provide it to people who need [food] and mass produce it at a low cost. So that’s what’s the holy grail of [indoor farming].

It’s not one thing. I’ve developed voice assistants before. You can actually say language understanding could be a holy grail, but in this case, everything that you know about IoT, cloud AI, machine learning, and robotics all comes into the picture, being orchestrated to find the economical way to produce vegetables on a large scale.

Henry Sztul: We have camera coverage of every crop that grows in a Bowery farm, and we’re constantly taking pictures. We do use computer vision algorithms, deep learning algorithms, to understand things like growth rates over time. To understand, not just when we see something like a stress response, which with something like arugula could be something like purpling, or something like butterhead [lettuce] could be like a yellowing at the edges. And we can observe those things. But what we can also start to do is predict what the conditions are that create that response. And we can be triggered — not just when does it happen, but also [what are] the leading indicators? And so what we’re doing now is we’re starting to look at not just [being] told when we see a stress response, but when we start to see the conditions that we predict to impact that, to create that stress response. And so all of these things are like parts of a puzzle, like the holy grail. But the holy grail is also solving the challenge of how to do this at an immense scale.

VentureBeat: Injong, could you talk a bit about your background and how your past experience informs this work in indoor farming?

Rhee: Yeah, so I did work, especially what I did at Google is building the cloud IoT platform for developers. I developed an end-to-end software and hardware stack of cloud IoT platform, including IoT Core, and then the edge TPU, which is a purpose-built AI chip that you can embed at the edge … so that you can make faster decisions to do control. And so while I’m working on this IoT problem, developing a platform for the IoT developers, I find this smart farming so interesting. It’s a full combination of all the things that I love, like my background in IoT or background in computer networks and background in distributed systems and building software and hardware and sensor networks and all of that and AI coming into good use. And so that’s how a whole thing actually plays. It’s not just one thing that’s going to contribute. It’s many, many things that I have worked on, they’ve become so much in use, and this is amazing. That’s why I was looking for a smart farming opportunity, and Bowery was just presenting itself to me as a perfect opportunity to use what I’ve learned in academia and industry.

VentureBeat: To what degree do people play a role in growing operations today? Is part of the goal for Bowery to create fully autonomous indoor farms?

Sztul: I think the goal of Bowery is to build farms that can deliver more, healthier produce to more people at the right price point. And so one of the ways to do that is with automation in areas. I think there might be people out there that will say “Yeah, our goal is to totally automate everything.” I think we come into the space with an open mind, which I think [is] a little different. And so that’s why I was saying we may get there, but we’re really more focused on how do we put out the best, the highest quality product consistently?

VentureBeat: What are some challenges associated with machine learning systems for indoor farming?

Sztul: There’s a basic machine learning example called the multi-armed bandit problem. And if you think about an octopus in a casino, sitting at a slot machine, the octopus is exploring, pulling different slot machine handles until it finds one that it can start to take advantage of, it can exploit. This is a classic problem of exploration versus exploitation.

A recipe [for growing produce] at Bowery includes things like light intensity, photo period, spectrum, different types of concentrations of nutrients, water temperature, air temperature, and humidity. There’s dozens of components that come into a Bowery recipe. If you were to try and tweak all of those combinations, all of those recipe components to make different combinations, that would take forever, and so we do the same thing with recipes. We have dozens and dozens of recipes in our farm; actually, I believe now we have over 50 recipes currently active across 10 products.

Rhee: Another area to add is the ability to forecast. Obviously, mass will be based on, you take a picture, and millions and millions of pictures, and then throughout the life cycle … until it gets to the harvest. And so, if I take a picture, we can actually figure out … growth rate, and then what the height of the plant is going to be and how much it’s going to produce in a future harvest. And that’s really driven by computer vision, as well as the sensing technology, and then adding all of that into a machine learning model to predict the mass. So that’s an interesting problem that is also fairly challenging because you know, different plants have different patterns, right? And the different colors and density. And so that’s quite a challenging problem.

Sztul: That’s actually a problem similar to some of the problems that self-driving cars have in detecting cars, distinguishing between cars when they’re overlapping. And so, as leaves grow, it’s a problem. It’s called occlusion. As leaves grow, how do you know that you know this is a leaf, and that’s a leaf in a different part of different plants? So it’s an incredibly challenging space. And the better we do there — Injong’s totally right — the better we can understand how much do we have in our farms? And how are we doing today versus yesterday?

Another one — and you would never think about this as a problem, well, simple but challenging — is where do you put things? How do you fill up your farm? If you have thousands of discrete locations to put your crops, how do you decide where things go? And that comes back to science at scale and recipe optimization.

One of the things we’ve done is used machine learning to optimize based on what a basil wants versus a butterhead, as an example — one wanting a cooler, drier climate and one wanting a warmer, more humid environment. We can set preferences and place these crops in different locations in the farm based off of those preferences. And so that’s an area that’s ripe for machine learning because [for] a person to make those decisions, I tell you firsthand, is impossible. And especially as you’re adding more complexity in terms of what the rules are. So that’s an area that I think seems straightforward but is complex and rewarding for us to spend time in.

VentureBeat: I spoke with Ken Goldberg at UC Berkeley last year about a project to create a fully autonomous polyculture garden and computer vision systems to monitor diverse groups of plants growing together. Is Bowery doing any experiments with polyculture growing?

Sztul: We’ve started growing some things that way, where we grow different types of — like our spring blend. So you can go buy our spring blend, and it’s got a whole bunch of different types of lettuces in it, and we used to grow it all together. And actually, we’ve gone away from polyculture over time. We’ve actually moved away from that, and we think that’s right now a better model because we can target what the individual crop needs versus another. But we could go back to that one day, I don’t know.