FIRST SPACE BASED NEUROMORPHIC CAMERA
WHAT IS IT?
Falcon Neuro is an experiment flying on the International Space Station (ISS) designed and built by cadets and faculty at the United States Air Force Academy (USAFA). Falcon Neuro demonstrates for the first time the use of biologically-inspired event-based, or neuromorphic, cameras for use in space.
Falcon Neuro is small, but mighty! It contains two neuromorphic cameras which were modified, by Western Sydney University (WSU) in Sydney, Australia,
for use in space. One camera looks down—Nadir, and one camera looks forward—Ram. Both cameras are controlled by electronics developed in the Space Physics and Atmospheric Research Center (SPARC) in the Department of Physics and Meteorology at the USAFA.
Falcon Neuro got a ride to the ISS from the DoD Space Test Program (STP) and started operations in January 2022 and will continue operations until January 2024. Falcon Neuro is run from a dedicated ground station at USAFA and performs operations during most week days. The two cameras have captured more than 200 recordings to date.
Flight unit of Falcon Neuro showing the Nadir and Ram cameras (left) CAD (Right) Photo Credit: USAFA SPARC
FACTS ABOUT FALCON NEURO
HOW DOES IT WORK?
- First demonstration of neuromorphic cameras in space
- Hyper-temporal observations with low bandwidth
- Full motion video — day or night
- Watch what is important — Detect what changes in a scene, not the background that stays the same
Event-based, or neuromorphic, cameras work more like the human eye than a regular camera, similar to a cell phone. Using complex circuitry, each pixel in Neuro records an event when the brightness in the pixel changes. Event cameras are high-speed, tens to hundreds of times faster than a standard video camera. Their unique circuitry provides this speed with much smaller amount of data being transmitted to the ground.
An event is just a list of the position of the pixel (row and column), the time and whether the brightness increased or decreased (polarity). Operators can send commands that change 21 different biases in the circuitry to allow Neuro to record brighter or dimmer events at the expense of more data to bring to the ground.
The data from Falcon Neuro is a long list of events that are post-processed on the ground by cadets and faculty at USAFA. The data is rendered at a frame rate chosen on the ground, allowing maximum flexibility in visualizing the data.
WHY IS IT IMPORTANT?
Neuromorphic cameras are new tools for space-based imagery. Since they read out much less data more quickly, they allow hyper-temporal (faster than real
time) recording capability. This is important for detecting challenging threats such as hypersonic re-entry vehicles, missiles or fast-moving aircraft. Future Air and Space Force leaders training at the USAFA are learning about space through first-hand experience. Falcon Neuro has allowed cadets from different backgrounds and departments to participate in this new cutting-edge experiment. Cadets from the Departments of Physics and Meterology, Astronautics, Electrical, Mechanical, Computer, Aero and Military Strategic Studies have all learned about space by helping build, test and fly Neuro.
Comparison of NASA ISS HD camera and Neuro motion compensated image of Honduras Photo Credit: United States Air Force Academy and Western Sydney University
ADDITIONAL FACTS
Falcon Neuro has proven so successful that a follow-on experiment called Falcon ODIN (Optical Defense and Intelligence through Neuromorpics) is planned. Falcon ODIN will contain greatly improved optics provided by AFRL’s Space Vehicles Directorate and a new, more sensitive focal plane array. Falcon ODIN is under construction in SPARC, and testing will start on the flight unit in the fall of 2023.
Captain Hayden Richards and Cadet Madison Yates Mechanical Engineering with the newly finished Neuro Photo Credit: United States Air Force Academy’s Space Physics and Atmospheric Research Center.
Falcon ODIN will be delivered to the DoD Space Test Program in late 2023 and fly to the ISS to continue research in neuromorphic technology in 2025.
Data from both experiments are freely available to government agencies and their contractors. These data are recorded in a wide variety of background lighting conditions and provide a unique “space-truth” that will be important for the validation of sensor modeling currently underway at USAFA, AFRL, WSU and the Air Force Institute of Technology.
FACT SHEET
THE AIR FORCE RESEARCH LABORATORY
An event is just a list of the position of the pixel (row and column), the time and whether the brightness increased or decreased (polarity). Operators can send commands that change 21 different biases in the circuitry to allow Neuro to record brighter or dimmer events at the expense of more data to bring to the ground.
On-board processing using conventional CNN software would use too much power ...
We have seen comments from WSU that they would be interested in Akida, and PvdM was keen for a ride to the space station.
RO)
But of course we're there:
.
