Ghost Autonomy, a pioneer in scalable autonomy software for consumer cars, today announced a $5 million investment from the OpenAI Startup Fund to bring large-scale, multi-modal large language models (MLLMs) to autonomous driving. The funds will be used to accelerate ongoing research and development of LLM-based complex scene understanding required for urban autonomy. The new investment brings the company’s total funding to $220 million to date.
“Multi-modal models have the potential to expand the applicability of LLMs to many new use cases including autonomy and automotive. With the ability to understand and draw conclusions by combining video, images, and sounds, multi-modal models may create a new way to understand scenes and navigate complex or unusual environments,” said Brad Lightcap, OpenAI’s COO and manager of the OpenAI Startup Fund.
MLLMs potentially represent a new architecture for self-driving software, capable of handling the long tail of rare and complex driving scenarios. Where existing single-task networks are limited to their narrow scope and training, LLMs allow autonomous driving systems to reason about driving scenes holistically, utilizing broad-based world knowledge to navigate complex and unusual situations, even those never seen before.
“Solving complex urban driving scenarios in a scalable way has long been the holy grail for this industry – LLMs provide a breakthrough that will finally enable everyday consumer vehicles to reason about and navigate through the toughest scenarios,” stated John Hayes, founder and CEO, Ghost Autonomy. “While LLMs have already proven valuable for offline tasks like data labeling and simulation, we are excited to apply these powerful models directly to the driving task to realize their full potential.”
Ghost’s platform allows leading automakers to bring artificial intelligence and advanced autonomous driving software into the next generation of vehicles, now expanding capabilities and use cases with MLLMs. Ghost is actively testing these capabilities via its development fleet today, and is partnering with automakers to jointly validate and integrate new large models into the autonomy stack.
