Comprehensive and accurate energy audits are essential to maximize energy savings and improvements in buildings realized from the design and implementation of deep retrofits for building envelopes. This paper presents a methodology for employing drones to conduct rapid building envelope performance diagnostics and perform aerial mapping of energy flows. The presented framework is tested on the Syracuse University campus to showcase: (1) visually identifying areas of thermal anomalies using an Unmanned Aerial System (UAS) equipped with thermal cameras and (2) detailed inspection applied to areas of high interest to quantify envelope heat flow using computer-vision techniques. Overall precision and recall rates of 76% and 74% were achieved, respectively. A discussion of the findings suggests refining procedure accuracy as a step toward automated envelope inspection.