Re: [sakai-dev] Sakai 11.4 issues under heavy load

classic Classic list List threaded Threaded
2 messages Options
Sam Ottenhoff Sam Ottenhoff
Reply | Threaded
Open this post in threaded view
|

Re: [sakai-dev] Sakai 11.4 issues under heavy load

MaxMetaspace is unrelated. See the changes in https://jira.sakaiproject.org/browse/SAK-33296.



On Mon, Jan 8, 2018 at 4:24 PM, Austin <[hidden email]> wrote:
Hello Sakai Devs,

We recently upgraded to Sakai 11.4 from 10.7.  Our servers seemed to be ok during testing and over the winter break, but today is the first day of the new semester and under heavy load, we are seeing lots of, I believe, database connection errors once a server gets to around 2000 sessions.

java.sql.SQLTransientConnectionException: sakai - Connection is not available, request timed out after 30000ms.

Also, our apache max processes seem to be hitting the max 256 connections on all servers.
And one java memory setting I wasn't sure of when upgrading to java 8 was the 

-XX:MaxMetaspaceSize=1024m

I'm not sure if that would have anything to do with the error we're seeing above?

We're running:

Sakai 11.4
Tomcat 8.0.47
Java 8
RedHat 7

with java config:

-Djava.util.logging.config.file=/home/sakai/tomcat/conf/logging.properties 
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager 
-Djdk.tls.ephemeralDHKeySize=2048 
-Djava.protocol.handler.pkgs=org.apache.catalina.webresources 
-server 
-Xmx10280m 
-Xms10280m
-XX:MaxMetaspaceSize=1024m 
-Xmn2500m 
-XX:+UseConcMarkSweepGC
-XX:+UseParNewGC 
-XX:CMSInitiatingOccupancyFraction=80 
-XX:+DisableExplicitGC 
-XX:+DoEscapeAnalysis 
-Dhttp.agent=Sakai 
-Dorg.apache.jasper.compiler.Parser.STRICT_QUOTE_ESCAPING=false
-Djava.awt.headless=true -Dcom.sun.management.jmxremote 
-Dsun.lang.ClassLoader.allowArraySyntax=true 
-Duser.timezone=Pacific/Honolulu 
-Dsakai.security=/home/sakai/security/ 
-Denable.jms.or.rabbitmq.components=rabbitmq 
-Dgradebook.update.request.message.sender=RabbitMqUpdateRequestMessageSender -server -verbose:gc 
-XX:+PrintGCDetails 
-XX:+PrintGCTimeStamps 
-Dcom.sun.management.jmxremote.port=8802 
-Dcom.sun.management.jmxremote.ssl=false 
-Dcom.sun.management.jmxremote.authenticate=true 
-Dcom.sun.management.jmxremote.access.file=/home/sakai/jmxremote.access 
-Dcom.sun.management.jmxremote.password.file=/home/sakai/jmxremote.password
 -javaagent:/home/sakai/common-1.17-socketStatisticsAgent.jar 
-Djava.endorsed.dirs=/usr/local/pkg/tomcat/endorsed 
-classpath /usr/local/pkg/tomcat/bin/bootstrap.jar:/usr/local/pkg/tomcat/bin/tomcat-juli.jar 
-Dcatalina.base=/home/sakai/tomcat 
-Dcatalina.home=/usr/local/pkg/tomcat
-Djava.io.tmpdir=/home/sakai/tomcat/temp org.apache.catalina.startup.Bootstrap 
-config /home/sakai/tomcat/conf/server.xml


java.sql.SQLTransientConnectionException: sakai - Connection is not available, request timed out after 30000ms.
at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:602)
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:195)
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:145)
at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:112)
at org.sakaiproject.db.impl.BasicSqlService.borrowConnection(BasicSqlService.java:260)
at org.sakaiproject.db.impl.BasicSqlService.dbRead(BasicSqlService.java:540)
at org.sakaiproject.util.BaseDbFlatStorage.getResource(BaseDbFlatStorage.java:341)
at org.sakaiproject.util.BaseDbFlatStorage.getResource(BaseDbFlatStorage.java:321)
at org.sakaiproject.site.impl.DbSiteService$DbStorage.get(DbSiteService.java:248)
at org.sakaiproject.site.impl.BaseSiteService.getDefinedSite(BaseSiteService.java:672)
at org.sakaiproject.site.impl.BaseSiteService.getSite(BaseSiteService.java:753)
at org.sakaiproject.site.impl.BaseSiteService.getSiteVisit(BaseSiteService.java:831)
at org.sakaiproject.site.cover.SiteService.getSiteVisit(SiteService.java:157)
at org.sakaiproject.portal.charon.site.PortalSiteHelperImpl.getSiteVisit(PortalSiteHelperImpl.java:1087)
at org.sakaiproject.portal.charon.handlers.SiteHandler.doSite(SiteHandler.java:317)
at org.sakaiproject.portal.charon.handlers.SiteHandler.doGet(SiteHandler.java:265)
at org.sakaiproject.portal.charon.handlers.SiteHandler.doGet(SiteHandler.java:153)
at org.sakaiproject.portal.charon.SkinnableCharonPortal.doGet(SkinnableCharonPortal.java:856)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:622)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:292)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
at org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:460)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:212)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:94)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:504)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:141)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:620)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:502)
at org.apache.coyote.ajp.AbstractAjpProcessor.process(AbstractAjpProcessor.java:877)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:684)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:283)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)

--
You received this message because you are subscribed to the Google Groups "Sakai Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/a/apereo.org/group/sakai-dev/.

--
You received this message because you are subscribed to the Google Groups "Sakai Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/a/apereo.org/group/sakai-dev/.
Austin Nakasone Austin Nakasone
Reply | Threaded
Open this post in threaded view
|

Re: [sakai-dev] Sakai 11.4 issues under heavy load

that error we're seeing is related to HikariCP version?  we update that in kernel pom.xml, right?

On Mon, Jan 8, 2018 at 11:28 AM, Sam Ottenhoff <[hidden email]> wrote:
MaxMetaspace is unrelated. See the changes in https://jira.sakaiproject.org/browse/SAK-33296.



On Mon, Jan 8, 2018 at 4:24 PM, Austin <[hidden email]> wrote:
Hello Sakai Devs,

We recently upgraded to Sakai 11.4 from 10.7.  Our servers seemed to be ok during testing and over the winter break, but today is the first day of the new semester and under heavy load, we are seeing lots of, I believe, database connection errors once a server gets to around 2000 sessions.

java.sql.SQLTransientConnectionException: sakai - Connection is not available, request timed out after 30000ms.

Also, our apache max processes seem to be hitting the max 256 connections on all servers.
And one java memory setting I wasn't sure of when upgrading to java 8 was the 

-XX:MaxMetaspaceSize=1024m

I'm not sure if that would have anything to do with the error we're seeing above?

We're running:

Sakai 11.4
Tomcat 8.0.47
Java 8
RedHat 7

with java config:

-Djava.util.logging.config.file=/home/sakai/tomcat/conf/logging.properties 
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager 
-Djdk.tls.ephemeralDHKeySize=2048 
-Djava.protocol.handler.pkgs=org.apache.catalina.webresource
-server 
-Xmx10280m 
-Xms10280m
-XX:MaxMetaspaceSize=1024m 
-Xmn2500m 
-XX:+UseConcMarkSweepGC
-XX:+UseParNewGC 
-XX:CMSInitiatingOccupancyFraction=80 
-XX:+DisableExplicitGC 
-XX:+DoEscapeAnalysis 
-Dhttp.agent=Sakai 
-Dorg.apache.jasper.compiler.Parser.STRICT_QUOTE_ESCAPING=false
-Djava.awt.headless=true -Dcom.sun.management.jmxremote 
-Dsun.lang.ClassLoader.allowArraySyntax=true 
-Duser.timezone=Pacific/Honolulu 
-Dsakai.security=/home/sakai/security/ 
-Denable.jms.or.rabbitmq.components=rabbitmq 
-Dgradebook.update.request.message.sender=RabbitMqUpdateRequestMessageSender -server -verbose:gc 
-XX:+PrintGCDetails 
-XX:+PrintGCTimeStamps 
-Dcom.sun.management.jmxremote.port=8802 
-Dcom.sun.management.jmxremote.ssl=false 
-Dcom.sun.management.jmxremote.authenticate=true 
-Dcom.sun.management.jmxremote.access.file=/home/sakai/jmxremote.access 
-Dcom.sun.management.jmxremote.password.file=/home/sakai/jmxremote.password
 -javaagent:/home/sakai/common-1.17-socketStatisticsAgent.jar 
-Djava.endorsed.dirs=/usr/local/pkg/tomcat/endorsed 
-classpath /usr/local/pkg/tomcat/bin/bootstrap.jar:/usr/local/pkg/tomcat/bin/tomcat-juli.jar 
-Dcatalina.base=/home/sakai/tomcat 
-Dcatalina.home=/usr/local/pkg/tomcat
-Djava.io.tmpdir=/home/sakai/tomcat/temp org.apache.catalina.startup.Bootstrap 
-config /home/sakai/tomcat/conf/server.xml


java.sql.SQLTransientConnectionException: sakai - Connection is not available, request timed out after 30000ms.
at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:602)
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:195)
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:145)
at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:112)
at org.sakaiproject.db.impl.BasicSqlService.borrowConnection(BasicSqlService.java:260)
at org.sakaiproject.db.impl.BasicSqlService.dbRead(BasicSqlService.java:540)
at org.sakaiproject.util.BaseDbFlatStorage.getResource(BaseDbFlatStorage.java:341)
at org.sakaiproject.util.BaseDbFlatStorage.getResource(BaseDbFlatStorage.java:321)
at org.sakaiproject.site.impl.DbSiteService$DbStorage.get(DbSiteService.java:248)
at org.sakaiproject.site.impl.BaseSiteService.getDefinedSite(BaseSiteService.java:672)
at org.sakaiproject.site.impl.BaseSiteService.getSite(BaseSiteService.java:753)
at org.sakaiproject.site.impl.BaseSiteService.getSiteVisit(BaseSiteService.java:831)
at org.sakaiproject.site.cover.SiteService.getSiteVisit(SiteService.java:157)
at org.sakaiproject.portal.charon.site.PortalSiteHelperImpl.getSiteVisit(PortalSiteHelperImpl.java:1087)
at org.sakaiproject.portal.charon.handlers.SiteHandler.doSite(SiteHandler.java:317)
at org.sakaiproject.portal.charon.handlers.SiteHandler.doGet(SiteHandler.java:265)
at org.sakaiproject.portal.charon.handlers.SiteHandler.doGet(SiteHandler.java:153)
at org.sakaiproject.portal.charon.SkinnableCharonPortal.doGet(SkinnableCharonPortal.java:856)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:622)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:292)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
at org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:460)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:212)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:94)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:504)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:141)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:620)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:502)
at org.apache.coyote.ajp.AbstractAjpProcessor.process(AbstractAjpProcessor.java:877)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:684)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:283)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)

--
You received this message because you are subscribed to the Google Groups "Sakai Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/a/apereo.org/group/sakai-dev/.


--
You received this message because you are subscribed to the Google Groups "Sakai Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/a/apereo.org/group/sakai-dev/.