Are we borderline friends?

I like a guy and he seems to flirt but I'm absolutely blind when it comes to someone flirting, and other times I can read into things. We see each other every so often and aren't incredibly close but neither have asked for phone numbers before, and we haven't met in a social environment before. He often calls me endearing names like 'Darling' and is always friendly. The last time we saw each other he fiddled lightly with something on my shirt near my collar. Is he flirting or are we just friends?