According to (unattributed?) notes in the Soncino edition, "closed" doesn't necessarily mean entirely sealed. Here's what they've got:
First the text:
Abaye said to Rabbah, Something which supports you was taught: A closed house has four cubits; if one had broken open its door-frame, it does not receive four cubits.7 A closed house [room] does not defile all around it; if he had broken through the door-frame, it defiles all around it.8
7) If a number of houses open into a common courtyard and their owners wish to divide it, each to have his own privately, each receives four cubits along the breadth of the courtyard for every door to his house that gives upon it, and the rest is shared equally. Now, if one of the doors had been walled up, but without its frame being broken through, its owner can still claim the four cubits for it; but if the frame was first broken through and then it was closed up, it ceases to count as a door, and the four cubits are lost. V. B.B. 12a.
This seems to be saying "closed (sealed up) on that side", rather than "entirely sealed". "Closed" is achieved by walling off an intact doorway (but if you break through the door-frame first it's not "closed").
The cross-reference to Bava Batra 12a doesn't appear to add anything to our understanding.
The second note also says that "closed" means walled up in this way:
(8) If a room containing a corpse is closed, i.e., the door is walled up, the defilement of the corpse does not extend beyond it. But if the door-frame was first broken and then walled up, so that no aperture at all is visible, the house is regarded as a grave and defiles everything around it to a distance of four cubits. — Thus an opening must be absolutely closed before it ceases to count as such, and the same applies to the cask.