That depends on where your traffic originates from. Back when I tracked people on my site, I found I got very little from search results. Most of it (> 95%) came from links from social media and Github. On a blog that's heavily about privacy I wouldn't expect much to come from Google.

Also, so what if the numbers go down? If your reason from writing a blog is to see a number on a screen then what does that actually give you?