This talk investigates possible machine learning architectures to support digitization of voice commands for maneuvers commonly conducted by radio between pilots and air traffic controllers. The work is motivated by both high altitude traffic growth and the significant growth of low altitude traffic planned with the rapid rise of mixed autonomy airspace including both e-VTOLs and drones.
We investigate the process of learning a bijective mapping between instructions spoken over radio by air traffic control and aircraft trajectory maneuvers. In this multimodal approach, we leveraged the OpenSky Network and LiveATC.net APIs to collect data on a shared temporal basis, which was subsequently joined using regex-based methods and filtering by timestamps and callsigns. Our first architecture employs two separate transformer-based encoders using masked auto-encoding to project each modality into a latent feature space with high semantic expressiveness. The two representations are then aligned using a contrastive learning objective, and a bijective, pair-wise mapping is learned via a normalizing flow network called RealNVP. The model yields promising predictive results and appears to effectively mitigate dimensionality collapse.
Building on this, our second architecture moves from a pair-wise bijective formulation toward a generative approach based on diffusion. With the rapid progress in image generation and motion planning in robotics, Diffusion Transformer architectures enable diffusion and flow matching algorithms to operate with multiple modality conditioning. Our architecture is designed to leverage ATC instructions, geographical aeronautical data and weather to guide ADS-B trajectory diffusion and flow matching. This work includes trajectory prediction and multi-horizon positional density estimation for collision avoidance and can be generalised to advisory tools for aerial motion planning.
The results are illustrated for the San Francisco Bay Area, leveraging several months of ADS-B data collected and the corresponding radio recordings for this airspace, across multiple towers, airports and sectors.
Alexandre Bayen is director of CITRIS and the Banatao Institute, Liao-Cho Innovation Endowed Chair and professor of electrical engineering and computer sciences and of civil and environmental engineering at the University of California, Berkeley, and founding associate provost for the Berkeley Space Center. He is also a faculty scientist in mechanical engineering at the Lawrence Berkeley National Laboratory (LBNL). From 2014–21, he served as the director of the UC Berkeley Institute of Transportation Studies (ITS). He received an Engineering Degree in applied mathematics from the Ecole Polytechnique, France, in 1998, and an M.S. and Ph.D. in aeronautics and astronautics from Stanford University in 1999 and 2004, respectively. He was a visiting researcher at NASA Ames Research Center from 2000 to 2003. In 2004, he worked as the Research Director of the Autonomous Navigation Laboratory at the Laboratoire de Recherches Balistiques et Aérodynamiques, (Ministere de la Defense, Vernon, France), where he holds the rank of Major.
About Air Time by NAMS-2
Air Time is a series of seminars on advanced aviation hosted by Crown Innovations, Inc., in collaboration with the University of California’s CITRIS and the Banatao Institute. The seminars feature leading experts on cutting-edge research who share interesting ideas on pertinent topics and innovative methodologies. Air Time speakers include subject matter experts from UC Berkeley, Merced, Davis, and Santa Cruz. The seminars take place weekly.
Crown Innovations, Inc. is the prime contractor for the NASA Academic Mission Services 2 (NAMS-2) contract. Contact the program management at nams2pmo@crownci.com for more information or to arrange a collaboration in your field.