Not a Dr. here, but:
Brown lumps on the elm might be
scale it's really hard to tell without a picture. Aphids suck juice from leaves and twigs, they wouldn't be stripping the bark. The chinese elm may be semi-deciduous in your area, but it will replace leaves periodically.
Regards,
Matt