A structure-based virtual screening of over 400,000 small molecules against the constitutive proteasome activity followed by in vitro assays led to the discovery of a family of proteasome inhibitors with a sulfonyl piperazine scaffold. Some members of this family of small non-peptidic inhibitors were found to act selectively on the β2 trypsin-like catalytic site with a preference for the immunoproteasome β2i over the constitutive proteasome β2c, while some act on the β5 site and post-acid site β1 of both, the immunoproteasome and the constitutive proteasome. Anti-proliferative and anti-invasive effects on tumor cells were investigated and observed for two compounds. We report novel chemical inhibitors able to interfere with the three types of active centers of both, the immuno- and constitutive proteasomes. Identifying and analyzing a novel scaffold with decorations able to shift the binders’ active site selectivity is essential to design a future generation of proteasome inhibitors able to distinguish the immunoproteasome from the constitutive proteasome.