Total no-brainer. Challenge is, as I read in an article a few years back, the fruit, vegetable, and meat vendors. They're sending and receiving lots of deliveries throughout the day and impacting them might not be the best thing and may result in increased gentrification (fish market moves out, Starbucks takes its place) and before long Kensington Mkt goes through the same transition Yorkville went through many years back.
That said, perhaps there's a balance to be struck. I've volunteered many times for Pedestrian Sundays and we still let business-owners and residents drive in (very slowly and carefully and often escorted). Deliveries could also be scheduled for off-hours as well.
It's definitely do-able - lots of places have already done just this sort of thing before. It's just a matter of logistics.