What am I doing wrong?

You probably should increase your calories a little, although calorie restriction has many more benefits than we once thought. I know this because I am 6'1",177 pounds and live on 1600 calories a day with plenty of energy to run 6 miles a day. You have to experiment and find what your body wants. Eliminating the refined carbs was an awesome step. Now just broaden a little more and eliminate ALL processed foods. Eat only real foods and drink nothing but water.