How to find memory "leaks" in clojure + jvm 1.8

Question

My clojure application has exhibited weird behaviour on linux systems running java 8. Specifically, when using httpkit to communicate to clients via websockets, the memory footprint grows apparently without bounds - depending on the system the process is then killed by the OS.

I looked at the memory usage in jconsole, and it's apparently the (new) metaspace that keeps growing. I managed to suppress the problem by passing -XX:MaxMetaspaceSize=128m to jar: In this case, the metaspace memory graph takes a dip whenever it exceeds the 128m, and the program doesn't end up eating all the memory. But it's a workaround - I'd like to understand why this growth is happening, but I'm not sure how to proceed. In C++, I'd track down leaks with valgrind, but since Java/Clojure is garbage-collected, I'm not sure what to look for.

I'm not 100% sure that it's only the httpkit code portion, but from my tests it seems like it - here's a code excerpt, where I'm listening for messages and depending on the message I send back a json object containing some data; coreparams-atom can be expected to have around 10 float entries at any time.

(httpkit/on-receive channel (fn [data]
  (let [data-map (json/read-str data)
        param (first (get data-map "data"))
        value (second (get data-map "data"))]
    (case (get data-map "type")
      ; ...
      "curparams" (let [tosend (json/write-str 
                                 {:type "curparams"
                                  :data (-> @state/coreparams-atom
                                            (assoc :timestamp (db/timestamp))
                                            (util/keyword-replace-char ":" "_"))})]
                    (httpkit/send! channel tosend))
      ; ...
    ))))

System:

• Ubuntu 14.04
• OpenJDK Runtime Environment (build 1.8.0_40-internal-b27)
• 6GB RAM
• Clojure 1.6.0
• http-kit 2.1.6

Similar behaviour on the same JVM on a CentOS to which I currently don't have access, so can't give detailed specs.

Answer 1

Memory sampling (using VisualVM or another tool like for example YourKit) will usually point you in the general direction of the problem. It can for example tell you which packages the classes belong to that are causing memory leaks. You can then do a more detailed profiling after that.

I have also run into problems in the past with VisualVM freezing. One thing you can try is to use a JMX connection, i.e. you launch your application with a JMX agent and connect to it from VisualVM as if it would be remote. See the docs for more information.

That said, YourKit is a much superior piece of profiling software, in my opinion.

Answer 2

As far as I know the best java memory analyzer tool is Memory Analyzer. It is free, fast. It can even used to analyze large heap, e.g. 30g, if we give it enough memory.