In Luke 5, verse 16 talks about how Jesus would often slip away to pray.
It's important to get away from all things that distract us from God - the "hustle and bustle" of life. In the quiet, it's easier to focus on God and pray. Sure, our minds can wander during prayer (and I know mine often does), but it's important to refocus, ask for forgiveness for our minds wandering, and resume talking to God.
In the quiet, it's easier to hear from God as well. If the TV, radio, and computer is off, if people aren't around us talking and making noise, we should be able to focus on God better and hear Him if and when He speaks to us.
Sometimes I find it difficult to pray in the quietness. I think that is because I am afraid of being alone with God. I think I fear because I'm afraid He might ask me to do something I don't want to do, or He might say, "No" to a specific prayer request. I'm also afraid that it will be at that time when I let my anger, etc., towards God (which makes me feel guilty even admitting that I sometimes get angry at Him) show. I know it's OK to be angry (God gave us emotions), but when that happens, I need to remember to repent of it and ask forgiveness afterward.
Other than what I've mentioned, I don't know any specific reason that would cause me to "fear" being alone with God. I know He loves me and he knows what's best for me - yet I still "fear" it. I have to wonder if it's because during that time God may reveal something about myself that I don't like, or want to hear. In other words, He will examine my heart and see just how much of a sinner I am (and make me face it), and then I would need to repent (sometimes it seems easier to ignore a sin than to admit it and repent). Or maybe He would tell me that my deepest desire/dream will never become a reality. and I don't know if I could handle that.
It might be that during this quiet time, that God humbles me and brings me to my knees. I like to think that I am humble, but in reality, I am more proud than I would like to admit. And it's that pride that needs to be removed...which, even though I know is a good thing, scares me.