I schedule appts and give driving directions on the phone at work, too. A request to move the appt 'back to' seems to imply it was that day originally, then they changed it, now they want that first choice again. I don't find myself using or hearing that terminology much at all. Maybe it's a regional thing?
Driving up or down the street however - makes me crazy when my co-worker says 'go down' a certain street when I know the driver will be aiming north on the trip. To me - that's UP.

Pinker also had an fascinating exploration of relative geography. Do you travel from point A up to point B, or down? Is it uptown or downtown? Is one choosing terms based on north-south-east-west ideas or elevation or what? Personally, I tend to think of up-down by north-south, but not always.

Chicago is neither beneath nor below Lake Michigan, although it appears that way on the pull-down classroom wall maps. Being a geography/map-oriented (no pun intended) person, I use direction. In fact, I use direction to the extent that when someone giving me directions over the phone tells me to "go right at the signal" I mentally translate that to and write it as the direction in terms of compass. I could never understand people who had to turn a map to orient it in the direction they were facing, but many do, and get totally lost, trying to read a map that is not turned.

In terms of time, I always think in terms of a timeline, with past on the left and future on the right. One moves forward toward the future, and backward to the past. So a meeting that is moved from Thursday to Friday is "moved forward" in my mind. I think the confusion arises because often people think in terms of lists or waiting in line. If you made a list of meetings, the soonest one would be on top, or first in line. So sooner becomes moving up on trhe list or forward in line, and later becomes moving down or backward.