You can use an Nvidia GPU for hardware trancoding. Although a 1080ti is overkill for that job (and it has a high power consumption).
You can also use a GPU with passthrough in a VM you create or use it with Tdarr to transcode H264 to H265 for example.
I'm not a 100% sure a 1080ti will work but it's easy enough to test...