Well, it seems like you already made your decision and thankfully you didn't express any great blow out so do your best to part as friends in case individuals meet again (drummers can be hard to find). We all know bands are essentially a dysfunctional family, with shared equipment like the PA being the kids.
If you are not loosing your livelihood and get no enjoyment from it there is little to loose. It doesn't even sound like you have to quit.