Currently, if we use any interaction not including LEFT and RIGHT click, we can essentially get items from the inventory. This probably isn't an intended behavior, but it would be good to add that as a boolean nonetheless.

Than it's probably just a case of:

if(clickedGui.areUnrealisticInteractionsBlocked()){
    event.setCancelled(true);
}

You can already achieve the same behavior without that change, but it would be helpful nonetheless:

...
).withListener((InventoryClickEvent e) -> {
    if(e.getClick() == ClickType.LEFT){
        //left click stuff
    } else if (e.getClick() == ClickType.RIGHT) {
        //right click stuff
    } else {
        e.setCancelled(true);
    }
})

Would've made a PR but I would rather give that someone who actually knows what they're doing lmao