I remember sidewalk "windows" in New York City from when I was a child. They were usually near the access port doors, in the sidewalk, used to load things into the basement. They were some sort of glass brick, about 4 inches square. I'm not sure how they were supported but folks would walk on them without a care. So it is possible.
As a sidenote I also remember wooden escalators, they still show up in my bad dreams sometimes. Scary stuff.