EntityManagerFactory is closed - only on particular machine

Question

My code constantly throws the "EntityManagerFactory is closed" exception.

The strange thing is - it only does so on our production machines. When I run the code with exact same configuration on my local machine, while connecting to exact same database - the errors don't appear. Another strange thing is - even though the exceptions keep getting thrown, it looks like Hibernate is somehow able to ignore them and successfully completes transactions. After a while though, it isn't able to complete any transactions and I am forced to restart application to make it work again (exceptions persist after a restart too).

I am injecting EntityManager into my DAO classes as follows:

 @PersistenceContext
 private EntityManager em;

I use these DAOs mostly from service methods annotated with @Transactional.

Stack trace:

2018-08-02 14:27:29,695 ERROR [stderr] (PG-JDBC EventLoop (6)) org.springframework.transaction.CannotCreateTransactionException: Could not open JPA EntityManager for transaction; nested exception is java.lang.IllegalStateException: EntityManagerFactory is closed
2018-08-02 14:27:29,695 ERROR [stderr] (PG-JDBC EventLoop (6))  at org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:450)
2018-08-02 14:27:29,695 ERROR [stderr] (PG-JDBC EventLoop (6))  at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:378)
2018-08-02 14:27:29,695 ERROR [stderr] (PG-JDBC EventLoop (6))  at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:474)
2018-08-02 14:27:29,695 ERROR [stderr] (PG-JDBC EventLoop (6))  at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:289)
2018-08-02 14:27:29,695 ERROR [stderr] (PG-JDBC EventLoop (6))  at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:98)
2018-08-02 14:27:29,695 ERROR [stderr] (PG-JDBC EventLoop (6))  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
2018-08-02 14:27:29,695 ERROR [stderr] (PG-JDBC EventLoop (6))  at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:688)
2018-08-02 14:27:29,695 ERROR [stderr] (PG-JDBC EventLoop (6))  at xxx.services.examination.service.ExaminationDocumentService$$EnhancerBySpringCGLIB$$ba40e12e.markDocumentError(<generated>)
2018-08-02 14:27:29,695 ERROR [stderr] (PG-JDBC EventLoop (6))  at xxx.services.examination.listener.ExaminationListener.notification(ExaminationListener.java:68)
2018-08-02 14:27:29,696 ERROR [stderr] (PG-JDBC EventLoop (6))  at com.impossibl.postgres.system.BasicContext.reportNotification(BasicContext.java:745)
2018-08-02 14:27:29,696 ERROR [stderr] (PG-JDBC EventLoop (6))  at com.impossibl.postgres.protocol.v30.ProtocolImpl.receiveNotification(ProtocolImpl.java:1143)
2018-08-02 14:27:29,696 ERROR [stderr] (PG-JDBC EventLoop (6))  at com.impossibl.postgres.protocol.v30.ProtocolImpl.dispatch(ProtocolImpl.java:781)
2018-08-02 14:27:29,696 ERROR [stderr] (PG-JDBC EventLoop (6))  at com.impossibl.postgres.protocol.v30.MessageHandler.channelRead(MessageHandler.java:45)
2018-08-02 14:27:29,696 ERROR [stderr] (PG-JDBC EventLoop (6))  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
2018-08-02 14:27:29,696 ERROR [stderr] (PG-JDBC EventLoop (6))  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349)
2018-08-02 14:27:29,696 ERROR [stderr] (PG-JDBC EventLoop (6))  at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341)
2018-08-02 14:27:29,696 ERROR [stderr] (PG-JDBC EventLoop (6))  at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293)
2018-08-02 14:27:29,696 ERROR [stderr] (PG-JDBC EventLoop (6))  at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267)
2018-08-02 14:27:29,696 ERROR [stderr] (PG-JDBC EventLoop (6))  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
2018-08-02 14:27:29,696 ERROR [stderr] (PG-JDBC EventLoop (6))  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349)
2018-08-02 14:27:29,696 ERROR [stderr] (PG-JDBC EventLoop (6))  at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341)
2018-08-02 14:27:29,696 ERROR [stderr] (PG-JDBC EventLoop (6))  at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334)
2018-08-02 14:27:29,696 ERROR [stderr] (PG-JDBC EventLoop (6))  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
2018-08-02 14:27:29,696 ERROR [stderr] (PG-JDBC EventLoop (6))  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349)
2018-08-02 14:27:29,696 ERROR [stderr] (PG-JDBC EventLoop (6))  at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926)
2018-08-02 14:27:29,696 ERROR [stderr] (PG-JDBC EventLoop (6))  at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:129)
2018-08-02 14:27:29,697 ERROR [stderr] (PG-JDBC EventLoop (6))  at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:642)
2018-08-02 14:27:29,697 ERROR [stderr] (PG-JDBC EventLoop (6))  at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:527)
2018-08-02 14:27:29,697 ERROR [stderr] (PG-JDBC EventLoop (6))  at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:481)
2018-08-02 14:27:29,697 ERROR [stderr] (PG-JDBC EventLoop (6))  at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:441)
2018-08-02 14:27:29,697 ERROR [stderr] (PG-JDBC EventLoop (6))  at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
2018-08-02 14:27:29,697 ERROR [stderr] (PG-JDBC EventLoop (6))  at java.lang.Thread.run(Thread.java:748)
2018-08-02 14:27:29,697 ERROR [stderr] (PG-JDBC EventLoop (6)) Caused by: java.lang.IllegalStateException: EntityManagerFactory is closed
2018-08-02 14:27:29,697 ERROR [stderr] (PG-JDBC EventLoop (6))  at org.hibernate.internal.SessionFactoryImpl.validateNotClosed(SessionFactoryImpl.java:531)
2018-08-02 14:27:29,697 ERROR [stderr] (PG-JDBC EventLoop (6))  at org.hibernate.internal.SessionFactoryImpl.createEntityManager(SessionFactoryImpl.java:613)
2018-08-02 14:27:29,697 ERROR [stderr] (PG-JDBC EventLoop (6))  at org.hibernate.internal.SessionFactoryImpl.createEntityManager(SessionFactoryImpl.java:154)
2018-08-02 14:27:29,697 ERROR [stderr] (PG-JDBC EventLoop (6))  at org.springframework.orm.jpa.JpaTransactionManager.createEntityManagerForTransaction(JpaTransactionManager.java:468)
2018-08-02 14:27:29,697 ERROR [stderr] (PG-JDBC EventLoop (6))  at org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:389)
2018-08-02 14:27:29,698 ERROR [stderr] (PG-JDBC EventLoop (6))  ... 31 more
(END)

My Spring configuration (config.xml) for hibernate-related beans looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xmlns:context="http://www.springframework.org/schema/context" xmlns:tx="http://www.springframework.org/schema/tx"
       xmlns:jpa="http://www.springframework.org/schema/data/jpa"
       xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context.xsd http://www.springframework.org/schema/tx http://www.springframework.org/schema/tx/spring-tx.xsd http://www.springframework.org/schema/data/jpa http://www.springframework.org/schema/data/jpa/spring-jpa.xsd">
    <jpa:repositories base-package="xxx.services.examination.dao"/>
    <context:annotation-config/>
    <tx:annotation-driven/>

    <!-- POSTGRES -->
    <bean id="entityManagerFactory" class="org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean">
        <property name="dataSource" ref="listenerDataSource"/>
        <property name="packagesToScan">
            <array>
                <value>xxx.services.examination</value>
            </array>
        </property>
        <property name="jpaVendorAdapter">
            <bean class="org.springframework.orm.jpa.vendor.HibernateJpaVendorAdapter">
                <!--<property name="showSql" value="true" />-->
                <property name="databasePlatform" value="org.hibernate.dialect.PostgreSQL95Dialect"/>
            </bean>
        </property>

    </bean>

    <bean id="transactionManager"
          class="org.springframework.orm.jpa.JpaTransactionManager">
    </bean>

    <bean id="hikariConfig" class="com.zaxxer.hikari.HikariConfig">
        <property name="poolName" value="springHikariCP" />
        <property name="dataSource" ref="innerDataSource"/>
        <property name="connectionTimeout" value="60000"/>
        <property name="maximumPoolSize" value="${postgres.max.connection.pool}"/>
    </bean>

    <bean name="innerDataSource" class="com.impossibl.postgres.jdbc.PGDataSource">
        <property name="host" value="${postgres.host}"/>
        <property name="port" value="${postgres.port}"/>
        <property name="database" value="${postgres.database}"/>
        <property name="user" value="${postgres.user}"/>
        <property name="password" value="${postgres.password}"/>
    </bean>

    <bean id="listenerDataSource" class="com.zaxxer.hikari.HikariDataSource" destroy-method="close">
        <constructor-arg ref="hikariConfig" />
    </bean>

    <!-- END POSTGRES -->


</beans>

My web.xml looks like this:

<!DOCTYPE web-app PUBLIC
        "-//Sun Microsystems, Inc.//DTD Web Application 2.3//EN"
        "http://java.sun.com/dtd/web-app_2_3.dtd" >

<web-app xmlns="http://java.sun.com/xml/ns/javaee"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_3_0.xsd"
         version="3.0">

    <display-name>xxx</display-name>

      <context-param>
        <param-name>contextConfigLocation</param-name>
        <param-value>classpath*:config.xml</param-value>
      </context-param>

      <listener>
        <listener-class>
          org.springframework.web.context.ContextLoaderListener
        </listener-class>
      </listener>

</web-app>

Hibernate version: 5.3.1.Final Spring version: 5.0.6.RELEASE HikariCP version: 3.1.0

Answer 1

In the end, it turned out to be caused by configuration of the machine. The ulimits were too low and there were too many "heavy" process ran by single user, which led to quick saturation of the limits. Moving the limits up solved the problem.