It's unlikely you actually lost followers because you didn't post daily. Medium periodically purges followers that aren't real from your count. That happened again not long ago. My own follower count went down too, and I have kept the same posting schedule for years.

I'm not upset when my account goes down like that, because I realize that Medium is simply making sure that my follower count is meaningful.