This drone has a camera that can tell it where to fly — no G…

There is a drone, just north of Toronto, that can fly on its own — no human pilot required. It was built by researchers at the University of Toronto’s Institute of Aerospace Studies (UTIAS), who have been perfecting the technology for the past few years.

But they have an even more ambitious goal: to create a self-flying vehicle that can find its way home based solely on what its cameras can see, in the event that it can no longer locate itself using GPS.

What might be easy for some humans is not so easy for a self-flying drone. Tech giants, traditional aerospace companies and startups are all working on drones that can fly farther, faster and take on more responsibilities — all on their own, and out of their minders’ sights.

The promise is that autonomous drones might be used to conduct detailed surveillance, provide temporary connectivity and communication, ferry passengers or even replace labour-intensive jobs such as window washing and building maintenance.

Package delivery is one of the most hyped applications on that list of new responsibilities. But one of the most critical tests is proving drones can make those deliveries safely and reliably in a wide range of environments and conditions — without being hijacked or disrupted.

It probably won’t come as a surprise that both Canadian and American militaries are interested, too.

In light of all this, University of Toronto researchers have been developing a drone that would be able to fly along a previously known route when faced with a range of location-related uncertainty, such as benign interference from tall urban buildings, hostile attempts at remote control or, worst case, outright failure of the drone’s GPS hardware or the satellite network.

“The vehicle would fly with GPS, and then if GPS gets lost, the vehicle could fly back completely based on vision,” said assistant professor Angela Schoellig, who is overseeing the research as head of the school’s Dynamic Systems Lab.

Following breadcrumbs home

University of Toronto researchers presented their initial efforts in a paper published in 2017. They believed they could take a technique called Visual Teach and Repeat (VTR), which had been used successfully on their ground-based robots, and adapt it for use in the air.

With VTR, the vehicle first follows a pre-determined route — either under the manual control of a human operator or using GPS — and collects data along the way, noting landmarks and other unique features it sees below.

If the vehicle loses GPS connectivity when flying on its own, it attempts to return home by following the path it was previously taught. In order to orient itself, the drone compares the visual input from its cameras to what it saw before — the same way a person might look for a familiar storefront to guide them out of a sprawling shopping mall.

All of this happens so quickly that you’d never know the drone was using its cameras rather than GPS to guide itself.

The team’s most recent paper was published in the January issue of IEEE journal Robotics and Automation Letters, and was led by postdoctoral fellow Michael Warren, PhD student Melissa Greeff and masters student Bhavit Patel.

They spent a week last spring collecting data at Defence Research and Development Canada’s (DRDC) Suffield Research Centre in southern Alberta. The results were encouraging: Warren said that during simulated GPS outages, their drone could reliably fly itself at about 10km/h at an altitude of 25 metres, and in calm winds as fast as 20km/h.

“That was the first time when we flew completely based on vision,” Schoellig said. Greeff is focused on getting the drone to fly even faster in high winds.

Challenges ahead include eliminating shadows from the drone’s visual input, which can confuse the image recognition algorithms. Another is handling seasonal change, or teaching the drone to recognize locations over time — for example, after a snowfall.

The drone isn’t self-aware enough yet to understand when its GPS signals can’t be trusted, prompting the fallback to kick in.

“In the next few months, we want to make it even more reliable and push the limits,” said Schoellig. For example, flying at lower altitudes where the ground moves too quickly for the current design to keep up, while potentially augmenting the drone’s own imagery with previously gathered satellite imagery.

“Getting it to work once is very different from making it work reliably,” she said.

Tunnels, sewers, caves, and ships

Schoellig’s lab isn’t the only group tackling this problem. Chipmaker NVIDIA demonstrated a drone in 2017 that was able to navigate through more than a kilometre of forest trail using vision alone.

The U.S. Defense Advanced Research Projects Agency (DARPA), the U.S. Army and the Royal Canadian Navy have all sought to fund research on how to operate drones in “GPS-denied environments,” such as tunnels, sewers, caves, bunkers, buildings and the inside of potentially hostile ships.

Defence Research and Development Canada has also been exploring the issue for years in partnership with Schoellig and other researchers at the University of Toronto. A commercial company, Drone Delivery Canada, is also a partner.

The firm has ambitions to use drones for deliveries over large swaths of land that would otherwise be difficult to cross using more traditional means of transportation — for example, in Canada’s north. Drone Delivery Canada funded Schoellig’s lab’s most recent paper, and the company recently struck a deal worth $2.5-million with the island community of Moose Factory, Ont., to offer drone-delivered goods by the spring.

Amazon and Google’s parent company, Alphabet, are perhaps the highest-profile companies working on making drone-assisted deliveries a reality. Amazon in particular is “keenly aware of the difficulty of facing spoofers and jammers,” said Todd Humphreys, an associate professor at the University of Texas at Austin, who has published cutting-edge research on GPS hijacking, with his students demonstrating that it is more than a theoretical threat.

Amazon declined to comment on its efforts to thwart hijacking, while Alphabet’s drone delivery subsidiary Wing did not respond to a request for comment.

“I think what Amazon is worried about is that people will go hunting for their drones,” Humphreys said, giving attackers “a little trophy drone” to put up in their garage.

It’s a reality that researchers are trying to thwart before it arrives.