Respect the Difference: Reinforcement Learning for Heterogeneous FPGA Placement

Simulated Annealing (SA) is widely used for FPGA placement due to its ability to adapt to different architectures and optimization goals. However, as FPGA capacity has grown and the devices have become more heterogeneous, the solution space has become very large, making efficient approaches to explore the space critical. We present RLPlace 2.0, an FPGA placer based on simulated annealing that utilizes Reinforcement Learning (RL) and better-than-random initial placement for efficient solution space exploration. The RL agent decides not only on the move types to use at each point in the anneal but also the type of block to move. The RL agent that controls the process is trained entirely online, enabling automatic adaptation to the architecture and design. For traditional FPGA benchmarks, RLPlace 2.0 outperforms both VPR 8 and RLPlace 1.0 in wirelength (WL) and critical path delay (CPD) at all runtime points. For example, on average over the Koios benchmark suite, RLPlace 2.0 reduces CPD by 3% and WL by 1.8% while cutting placement time by 54% at the default run time point vs. VPR 8. On more complex and heterogeneous FPGAs that incorporate a hard Network on Chip (NoC), RLPlace 2.0 enables more efficient co-optimization of NoC and programmable routing metrics by learning when and how to move the NoC routers throughout the anneal. On average for benchmarks containing NoC routers, RLPlace 2.0 achieves the same WL and CPD, 41% reduced NoC latency, and 6% improved NoC bandwidth usage in 50% less time compared to RLPlace 1.0.