Fetch.ai’s car park agents: How they work

Nov 11, 2019

This article is about the Fetch.ai car park agent I have been working on for the last couple of months. It is a proof of concept application and is made available for anyone to make one themselves and play around with.

The device
The software

Overview

The car park agent consists of a Raspberry Pi fitted with a camera looking at potential car parking spaces. Our software runs on the Raspberry Pi and can report on the number of free parking spaces available. This information is advertised on the Fetch.ai network and other agents (such as a GPS or digital assistant) could buy this data in exchange for Fetch.ai tokens (FET) and so direct their owner to the parking space near their destination.

Fetch.ai is not aspiring to be a computer vision or car park monitoring company. We have used off the shelf components to build the unit and perform the car detection algorithms. Fetch.ai provides an economic platform to any company or individual who does make such things to allow their devices to communicate with, and transact with, other devices in a mutually beneficial way. They can feel safe in the knowledge that the network is open source and decentralized, ensuring that overhead costs are kept to a minimum and it will continue to serve the interests of its users (rather than advertisers).

This is a simple demonstration of the potential power of the Fetch.ai network. An Autonomous Economic Agent (AEA) gathers valuable data from the real world and uses it to create revenue for its owner.

In future, the digital assistants could make even more intelligent decisions as more information becomes available on the network.

For example:

  • Restaurants could share information about current wait times or specials on the menu.
  • Commercial car parks could allow you to book a space in advance (which your digital assistant could do without you having to tell it)
  • Previous visitors to a restaurant could monetise their recommendations to help your digital assistant decide where you would like the most. At present, we give our reviews to sites like Trip Advisor who then monetize them. Why not sell them directly to people who want to know?

In the system I have made, the client is a simple agent which just searches for agents offering car parking information, purchases the information and displays the results in the terminal window.

Many such Internet of Things (IoT) devices like this already exist. The Fetch.ai platform provides an economic incentive to deploy them and expose the information they can gather.

Privacy

We recognise the importance of individual privacy and security. While the Raspberry Pi captures images of the car park/street, these are only ever available to the owner of the device. The images are never sent anywhere else or made available on the Fetch.ai network. Only the raw car parking spaces and the number of detections are broadcast.

How to make the car park agent

There are detailed instructions on how to make and set up the agent in our Github repository:

https://github.com/fetchai/carpark_agent.

The system utilizes Fetch.ai’s AEA Framework which can be found here: https://github.com/fetchai/agents-aea

How to use the car park agent

Once the car park agent is up and running, you can access the GUI interface which looks like this:

We can define a region of interest in the image — shown here in blue. Cars will only be detected in this area. We also specify the maximum number of cars which can park in this area. In this case, the maximum is 8.

Every couple of minutes the car park agent records an image from the camera and performs car detection on it. When using the software, you can see these captured images on the right of the screen (it just shows the four most recent images) and next to the image we show the cars which have been detected.

Here is one such image showing the cars that have been detected:

The panel on the far right shows the data the agent has extracted. This represents the number of cars detected and the number of free spaces available (which is just the maximum number of spaces minus the number of car detections).

How the data is advertised and purchased

The agent needs to expose some data about itself for free. This is so that potential buyers can know whether they are interested in the data. Then when an offer is accepted, the “paid-for” data can be released.

This diagram illustrates the communication that goes on to accomplish the transaction. The initials “AEA” stand for “Autonomous Economic Agent” which are the kind of agents that Fetch.ai supports.

Here is the above diagram in words:

  1. The Car Park Agent registers a service on the Search and Discovery system to say “I can report on car parking information)
  2. The Client Agent then searches for agents offering this service
  3. The search and discovery system returns a list of agents which support it
  4. The Client Agent sends a message to all such agents asking them for proposals
  5. The Car Park Agents respond saying “I am in location X and can report on Y spaces and I charge Z for the information”
  6. The Client Agent then accepts and sends a transaction request to the Ledger to transfer the funds
  7. Then when the transfer is complete the Car Park Agent sends the number of free spaces to the Client Agent.

Code architecture

The car park agent has a number of different modules, all running on the Raspberry Pi.

Most of the components do not communicate with each other directly but instead do so through the Operating System (OS) by reading and writing to a database and various files. The reason for this separation is that during development I encountered some difficult memory-related bugs (mostly on Raspberry Pi Version 3) and being able to run each of the modules in separate processes made it much easier to track them down. However as a consequence of this, each of the modules is nicely decoupled from the others which makes modifying the scripts much easier.

Image Capture

This simply takes ownership over the camera and continually makes the latest camera image available to other modules in a thread safe way. It also regularly (say, once every 2 mins) records an image to the file system.

GUI

This uses the Image Capture module — just so that it can display the live feed from the camera on the screen. The GUI is what you see and interact with when you run the agent. It lets you define the region of interest mask and set the maximum number of parking spaces

Car park detection

This waits until a new image is written to the file system (from the Image Recorder) and when it is, it applies the region of interest mask and then performs a car detection algorithm. For this we use an off-the-shelf piece of code called Mask_RCNN which utilised Google’s Tensor Flow library:

https://github.com/matterport/Mask_RCNN

Once it has detected the cars in the image, it counts them, subtracts it from the max number of spaces and writes the number of free spaces back to the database.

Car Park Autonomous Economic Agent (AEA)

This connects to the Fetch.ai network; registering itself on the Open Economic Framework (OEF). It waits for client agents to ask it for proposals, its sends out proposals, and, if they are accepted, sends out the data about free parking spaces.

It is only this module which actually communicates with the Fetch.ai network.

Areas for further work

This is a proof of concept and is meant to demonstrate the value Fetch.ai can offer. Here I will discuss some of the limitations and how the system might be improved.

Agent Trust

At present, the client agent has to trust that the car park agent will send the data once the transaction is complete (and it not just someone setting up a fake agent pretending to know about car park spaces). There is no simple solution to this problem.

The solution rests on a number of different technologies:

  • AI-powered smart contracts — ensuring the data and FET are submitted before either side receives their due (this capability is already on the Fetch.ai system and solves part of the problem — but doesn’t prevent fake agents just posting rubbish data)
  • Reputation — Agents should be able to write reviews of other agents which collectively define the reputation of the agent in terms of their reliability and trustworthiness. (Again, this solves part of the problem, but doesn’t prevent an agent simply disappearing after ripping another agent off and restarting with a new identity).
  • Identity — Being able to certify that an agent belongs to a specific real-world individual/company means that any reputation acquired by an agent can then be associated with that individual/company. Together with Outlier Ventures, Fetch.ai and Sovrin are bringing this functionality to reality, using software called Anvil.

Car Detection

We have used an off-the-shelf car detection algorithm called Mask-RCNN. This gives really impressive results, but still has problems when cars are obscured by other cars. One solution to this is to find a better place to put your camera with a clearer view.

However the Mask-RCNN training data was from dashboard camera footage, so is not ideally suited to seeing cars from above. I suspect a more bespoke training data set, trained specifically on car parks, could help. Ideally, users could mark up their own images or flag where the algorithm had got it wrong and in doing so, train a model which is bespoke to a specific parking location.

Other improvements in this area could come from tracking cars which enter and leave a car park and cross-referencing that with the number of cars detected. However, the whole detection algorithm would need to be much faster for this to work.

Modular detection algorithms

The detection model we use is a specific instance of Mask-RCNN which we made some minor modifications to (which is why we have a copy of it in the codebase).

https://github.com/matterport/Mask_RCNN

However, it would be more flexible to have a more plug-in based architecture — perhaps based on Tensor Flow Zoo so that if people want to extend this functionality or specialize it on other kinds of objects, it is easier to do:

https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/detection_model_zoo.md

Prediction

The current agents can only tell you whether there is a space available now. However, they could look at past data as well as data from other sources (such as the weather) and make predictions about the future. That could be more useful because, as a client, I typically want to know if there is likely to be a space when I am due to arrive (perhaps in an hour’s time).

Equally, it could provide historical data to allow clients to make their own predictions.

Larger areas

Some car parks cannot be covered by a single camera. One improvement to the system would be to allow multiple cameras to cover different areas of a car park (perhaps with some overlap).

Client app

The client side app could be made much more sophisticated with a map showing the agents car parking spaces it knows about.

Region of interest location

At the moment, you simply specify the location of your camera. However, it would be better to specify the real world location and shape of the area of interest so clients can see precisely where it is on a map.

Get involved

To learn more about how to build agents on the Fetch.ai network, join our Developer Slack channel or visit our website to read our documentation. You can also stay up to date with our progress by joining our Telegram channel.