Adding support for conditional match blocks into sshd_config would provide more granular access control.