I discovered a setting in a the service file that splunk docs recommend for systemd setup. Funny thing we ran into was that splunk is having problem running from systemd. If I run it the orginal way /opt/splunk/bin/splunk start it runs better or as it should. Something in Systemd and splunk doesn't seem to be working quite right. If I run splunk through the systemd service file splunk keeps running into thread limits, and errors stating the OS thread limit has been reached when it clearly hasn't. We'll see what splunk comes back with.

This is on a physical server by the way not a VM.