There are a number of sites that say to edit /etc/security/limits.conf.
Ubuntu Studio 14.04.2 automatically makes the entries for Jack in
The debian.tryphon.eu Rivendell package automatically sets them up in
These files get read in by an include statement in /etc/security/limits.conf so all is good, don’t edit limits.conf.
The general philosophy with jack is to protect application’s priority by protecting the transport. If you have a busy audio processor or something else that MUST run in realtime you might want to add a file for it in /etc/security/limits.d/ reserving its necessary memory and a negative nice value closer to zero than that of Jack and Rivendell. This is unnecessary unless you have overruns or gaps or frequency or time shifts that are otherwise unexplained.
Another source of problems can come from hard drive partitions that are filled up. Beware of backup programs, log files, etc. in a partition that is pretty full. Ubuntu studio hangs up waiting for disk writes to complete.