Inspectors may clear Patten Towers as early as Thursday