On my nerves

After not seeing Ray since last Sunday, we finally connected today. We spoke earlier today and planned on going to the pool later in the afternoon and going to see "Super Bad". He didn't finish up what he was doing until after 6 though, and didn't get up here until 7:30 so we just went out to eat. He lost his ATM card last week so he had no money to pay for his dinner. That's fine. Whatever, I'll pay. At the restaurant though it seemed like all he could do was make snide remarks about this or that, and just had a negative attitude in general about things. The waiter, the food, things I was talking about, the music. It wasn't that fun. I felt like I had to keep my niceness under control. I was just getting increasingly irritated with him, it was much like having dinner with a child who hasn't had thier afternoon nap. I felt as though I was having dinner with a child in many aspects anyways. I made sure to not be a bitch to him though, and tried not to be passive aggressive about things either. I kept trying to talk myself into being open to him. I don't think it's working.
After I paid we came home and I turned on the TV. I put on a "Rescue Me" I had on DVR and watched that. That was fine, because it was quiet time. When the show was over he started to kiss me and I stopped him and said I needed to talk to him. I told him that I didn't know what I was so scared of, but I was trying to figure it out and I think I need to step back a little. He said okay, and that he was starting to wonder about what was going on with me. I also told him that with school my free time is limited, and while I like spending time with him, or whomever my significant other is, but I also require a lot of personal time, and time with my friends, so I couldn't promise him a given amount of time. He also asked me how long it had been since I broke up with my ex, I told him May. He then asked how long we had been together. I told hime my standard "a year, this time" response. He didn't want details, so I didn't offer them. He only asked what this time meant, so I told him. He then said that 3 months isn't really very long, and he agreed that things had kind of gone fast. He said he understood and told me to keep "mullin' it over" and that he'd be here. I guess I could always use the unresolved issues angle if need be. It's not like I can say "Be less annoying and I'll want to hang out with you more." or "Why don't you give me a call when you get your shit together?" Can I?
Why does he get on my nerves so badly? Am I just looking for reasons? Sometimes, some of the stuff he says to me is everything I could possibly ever want from a man, so why doesn't it fit? I guess it's just me?