“This approach has the potential to radically change the way code is maintained in large enterprises,” a group of authors from Google Core and Google Ads wrote in a new “experience report” describing their approach. “Not only can it accelerate the work of engineers, but make possible efforts that were previously infeasible due to the huge investment needed.”
Ultimately, Google’s goal was to identify opportunities for LLMs to provide additional value and support scale without requiring difficult-to-maintain abstract syntax trees (AST). These are widely used to represent the structure of a program or snippet of code, but they are deterministic — that is, outcomes are already identified — and code migration use cases span much more complex constructs that are difficult to represent with ASTs, the authors explained.
“Achieving success in LLM-based code migration is not straightforward,” the authors noted. “The use of LLMs alone through simple prompting is not sufficient for anything but the simplest of migrations. Instead, as we found through our journeys, and as described in the case studies in this paper, a combination of AST-based techniques, heuristics, and LLMs are needed to achieve success. Moreover, rolling out the changes in a safe way to avoid costly regressions is also important.”
