Hi Oysterowl83---Last fall when I brought my potensai brush cherry inside, I soon noticed a signicant amount of leaves dying and the dieback on my lower branches was substantial. I believe that Scale was the culprit (assisted from the stress caused from the move indoors). Some diligence with insecticidal soap and my tweezers helped save mine(though it's a long way from even potensai now
). Have a look under the leaves and see if you have any.
Best of luck
B