Sigil: City of Doors

Considering I have been gone from the game almost 7 years I was looking through all the new mechanics and forgot some of the different factions on the skull so I was using it to read through the factions. It went into the shadowguard faction and did say if you joined that you would be a member and could not swap. Considering lore wise my character is neutral in all the Sigil factions for almost 6 years I hit escape to leave the text menu since there was no option to back out but it still officiated me into the shadowguard faction. Is there a way to have it removed and go back to being factionless please?

You can click on the Faction badge and use its text menu to leave the faction.