Can I restrict a Call Center Supervisor to manage only his departments?

Yes

Call Center Supervisors can be configured to manage only the departments assigned to them.
See Screenshot

Screenshot_20230117_123026