Usually, you want to mitigate all possible vulnerabilities unless we are talking about Meltdown and Spectre which are a class or family of dozens of vulnerabilities. But what sysadmins hate more than these vulnerabilities are mitigations offered to these vulnerabilities. Some of these mitigations have a massive impact on performance, while not offering any significant protection.
Gauging the pros and cons, sysadmins have gone as far as asking the Linux kernel community to give them an option to disable these mitigations. The Linux kernel community always listens.
Linux Kernel 4.15 added the ability for sysadmins to disable the kernel’s built-in mitigations for the Spectre v2 vulnerability, then Linux Kernel 4.17 offered the option to disable all mitigations for Spectre v4 and now Linux Kernel 4.19 allows admins to disable mitigations for Spectre v1.
You may or may not trust NSA, but they have a very decent guide on GitHub to help keep up with all Spectre related vulnerabilities.