I once had a car who rearview mirror fell off the inside of the windshield. Actually, this has happened to me in multiple cars I've had. The rearview mirror is only held onto the glass by glue or a sticker, and enough years of sitting in the sun will do that.
The most recent time it happened, I didn't even reattach the damn thing. I drove around for months with no rearview mirror in my car.
And I drove faster than I'd ever driven. Even when I changed vehicles, it was now a habit and I always drive faster than the signs suggest. And -- crossing my fingers -- I haven't been pulled over in years.
Looking back won't do much for you. It will either make you nostalgic about the past, slowing you down in the present, or have you feeling good for something already done -- which isn't always a bad thing, but it won't help you much with what you need to get done now.
Looking back also lets you see what everyone else is doing behind you, which is a complete waste of resources. They're behind you for a reason. And that's where they'll stay, regardless of what they do-- as long as you keep your eyes focused forward and continue doing what you do.
So, yes -- rip the rearview mirror out of you car and drive without it. Unless you have OnStar, then leave the mirror. You do need directions for where you're going.