Re: [sakai-dev] Too Many Open Files system errors

classic Classic list List threaded Threaded
4 messages Options
Austin Nakasone Austin Nakasone
Reply | Threaded
Open this post in threaded view
|

Re: [sakai-dev] Too Many Open Files system errors

Hello Matthew,

Thanks for your quick reply!  Currently, using that lsof -p <pid> command we have between 3500 - 4000 open files (looks like it counts TCP connections a a file).  Next time we hit the error, I'll be sure to check the open files then.

Also, I can't view KNL-1485, even though I'm logged in to jira.

Thanks,

Austin

On Thu, Jan 18, 2018 at 12:05 PM, Matthew Jones <[hidden email]> wrote:
How many open files does it have? What are the files?

Use something like lsof -p and check https://www.cyberciti.biz/faq/howto-linux-get-list-of-open-files/ on a running process.

Either your process limit isn't being used correctly by that process (there's lots of articles out there about that) or there's a bug in Sakai that's keeping too many processes open. 

I think there was this issue fixed in 11.5 https://jira.sakaiproject.org/browse/KNL-1485

Nightly has ~2000 files open for the 11.x process.



On Thu, Jan 18, 2018 at 5:00 PM Austin <[hidden email]> wrote:
Hello Sakai Devs,

I think I mentioned this in another thread involving some other issues, but we hit this problem again on one of our servers today.

The tomcat logs will start showing errors on nearly everything with "Too many open files"

When we first encountered this error we increased the OS's ulmit to 32K.  But since we hit it again today, we're thinking of increasing it to 65K.  But before we do that, has anyone else encountered errors like this?  What do other folks have ulimit set to if at all?  would setting it to 'unlimited' be too high?

I found a couple of older threads that mention it:


but they're pretty old, so I don't know if it would still apply to 11.4.  We're running:

Sakai 11.4
Tomcat 8.0.47
java 8
RHEL7

(also, we never ran into this issue with Sakai 10.7 / RHEL6)

Thanks,

Austin




java.io.FileNotFoundException: /content01/store/content71/2016/227/20/0b09a6cf-4fde-4d73-b941-7f4fe664cecf (Too many open files)

java.net.SocketException: Too many open files

org.apache.wicket.pageStore.DiskDataStore.getFileChannel /home/sakai/tomcat/work/Catalina/localhost/profile2-tool/sakai.profile2-filestore/3167/6024/e157f028-6d67-44ce-bb77-f711437228fa/data (Too many open files)

org.sakaiproject.util.StorageUtils.readDocumentFromString readDocumentFromString: java.lang.IllegalStateException: java.io.FileNotFoundException: /home/sakai/tomcat/webapps/etudes-mneme-tool/WEB-INF/lib/junit-4.11.jar (Too many open files)



--
You received this message because you are subscribed to the Google Groups "Sakai Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/a/apereo.org/group/sakai-dev/.

--
You received this message because you are subscribed to the Google Groups "Sakai Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/a/apereo.org/group/sakai-dev/.
Jonas Eriksson-2 Jonas Eriksson-2
Reply | Threaded
Open this post in threaded view
|

SV: [sakai-dev] Too Many Open Files system errors

We are on 11.4 and on one of our machines “lsof” gives 2800 lines.

 

Our machines restarts once a week for OS patches.

ulimit is set to unlimited

 

tomcat 8.0.41, java 8, rhel7

 

Regards

Jonas Eriksson

Umeå University

 

Från: 'Matthew Jones' via Sakai Development [mailto:[hidden email]]
Skickat: den 18 januari 2018 23:06
Till: Austin <[hidden email]>
Kopia: sakai-dev <[hidden email]>
Ämne: Re: [sakai-dev] Too Many Open Files system errors

 

How many open files does it have? What are the files?

 

Use something like lsof -p and check https://www.cyberciti.biz/faq/howto-linux-get-list-of-open-files/ on a running process.

 

Either your process limit isn't being used correctly by that process (there's lots of articles out there about that) or there's a bug in Sakai that's keeping too many processes open. 

 

I think there was this issue fixed in 11.5 https://jira.sakaiproject.org/browse/KNL-1485

 

Nightly has ~2000 files open for the 11.x process.

 

 

 

On Thu, Jan 18, 2018 at 5:00 PM Austin <[hidden email]> wrote:

Hello Sakai Devs,

 

I think I mentioned this in another thread involving some other issues, but we hit this problem again on one of our servers today.

 

The tomcat logs will start showing errors on nearly everything with "Too many open files"

 

When we first encountered this error we increased the OS's ulmit to 32K.  But since we hit it again today, we're thinking of increasing it to 65K.  But before we do that, has anyone else encountered errors like this?  What do other folks have ulimit set to if at all?  would setting it to 'unlimited' be too high?

 

I found a couple of older threads that mention it:

 

 

but they're pretty old, so I don't know if it would still apply to 11.4.  We're running:

 

Sakai 11.4

Tomcat 8.0.47

java 8

RHEL7

 

(also, we never ran into this issue with Sakai 10.7 / RHEL6)

 

Thanks,

 

Austin

 

 

 

 

java.io.FileNotFoundException: /content01/store/content71/2016/227/20/0b09a6cf-4fde-4d73-b941-7f4fe664cecf (Too many open files)

 

java.net.SocketException: Too many open files

 

org.apache.wicket.pageStore.DiskDataStore.getFileChannel /home/sakai/tomcat/work/Catalina/localhost/profile2-tool/sakai.profile2-filestore/3167/6024/e157f028-6d67-44ce-bb77-f711437228fa/data (Too many open files)

 

org.sakaiproject.util.StorageUtils.readDocumentFromString readDocumentFromString: java.lang.IllegalStateException: java.io.FileNotFoundException: /home/sakai/tomcat/webapps/etudes-mneme-tool/WEB-INF/lib/junit-4.11.jar (Too many open files)

 

 

 

--
You received this message because you are subscribed to the Google Groups "Sakai Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/a/apereo.org/group/sakai-dev/.

--
You received this message because you are subscribed to the Google Groups "Sakai Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/a/apereo.org/group/sakai-dev/.

--
You received this message because you are subscribed to the Google Groups "Sakai Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/a/apereo.org/group/sakai-dev/.
Austin Nakasone Austin Nakasone
Reply | Threaded
Open this post in threaded view
|

Re: [sakai-dev] Too Many Open Files system errors

Hello All,

We hit this problem again, so I ran 

$ lsof -p <pid>

and it indicated that there are 34403 files open.  Our ulimit is set to 35K.  However the strange thing is that of the open files 30K of them are

/tomcat/components/pasystem-pack/WEB-INF/lib/pasystem-impl-11.4.jar

Does anyone know why pasystem would open so many instances of that file?

Thanks,

Austin

On Thu, Jan 18, 2018 at 9:38 PM, Jonas Eriksson <[hidden email]> wrote:

We are on 11.4 and on one of our machines “lsof” gives 2800 lines.

 

Our machines restarts once a week for OS patches.

ulimit is set to unlimited

 

tomcat 8.0.41, java 8, rhel7

 

Regards

Jonas Eriksson

Umeå University

 

Från: 'Matthew Jones' via Sakai Development [mailto:[hidden email]]
Skickat: den 18 januari 2018 23:06
Till: Austin <[hidden email]>
Kopia: sakai-dev <[hidden email]>
Ämne: Re: [sakai-dev] Too Many Open Files system errors

 

How many open files does it have? What are the files?

 

Use something like lsof -p and check https://www.cyberciti.biz/faq/howto-linux-get-list-of-open-files/ on a running process.

 

Either your process limit isn't being used correctly by that process (there's lots of articles out there about that) or there's a bug in Sakai that's keeping too many processes open. 

 

I think there was this issue fixed in 11.5 https://jira.sakaiproject.org/browse/KNL-1485

 

Nightly has ~2000 files open for the 11.x process.

 

 

 

On Thu, Jan 18, 2018 at 5:00 PM Austin <[hidden email]> wrote:

Hello Sakai Devs,

 

I think I mentioned this in another thread involving some other issues, but we hit this problem again on one of our servers today.

 

The tomcat logs will start showing errors on nearly everything with "Too many open files"

 

When we first encountered this error we increased the OS's ulmit to 32K.  But since we hit it again today, we're thinking of increasing it to 65K.  But before we do that, has anyone else encountered errors like this?  What do other folks have ulimit set to if at all?  would setting it to 'unlimited' be too high?

 

I found a couple of older threads that mention it:

 

 

but they're pretty old, so I don't know if it would still apply to 11.4.  We're running:

 

Sakai 11.4

Tomcat 8.0.47

java 8

RHEL7

 

(also, we never ran into this issue with Sakai 10.7 / RHEL6)

 

Thanks,

 

Austin

 

 

 

 

java.io.FileNotFoundException: /content01/store/content71/2016/227/20/0b09a6cf-4fde-4d73-b941-7f4fe664cecf (Too many open files)

 

java.net.SocketException: Too many open files

 

org.apache.wicket.pageStore.DiskDataStore.getFileChannel /home/sakai/tomcat/work/Catalina/localhost/profile2-tool/sakai.profile2-filestore/3167/6024/e157f028-6d67-44ce-bb77-f711437228fa/data (Too many open files)

 

org.sakaiproject.util.StorageUtils.readDocumentFromString readDocumentFromString: java.lang.IllegalStateException: java.io.FileNotFoundException: /home/sakai/tomcat/webapps/etudes-mneme-tool/WEB-INF/lib/junit-4.11.jar (Too many open files)

 

 

 

--
You received this message because you are subscribed to the Google Groups "Sakai Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/a/apereo.org/group/sakai-dev/.

--
You received this message because you are subscribed to the Google Groups "Sakai Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/a/apereo.org/group/sakai-dev/.


--
You received this message because you are subscribed to the Google Groups "Sakai Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/a/apereo.org/group/sakai-dev/.
Austin Nakasone Austin Nakasone
Reply | Threaded
Open this post in threaded view
|

Re: [sakai-dev] Too Many Open Files system errors

I added the patch in SAK-33866 Update handlebars version in PASystem to version 4
and I'm still seeing extra instances of

/lib/pasystem-api-11.4.jar

being created.

On Wed, Jan 24, 2018 at 9:04 AM, Austin <[hidden email]> wrote:
e.g. 

$ sudo lsof -p 7549 | grep -c "pasystem-impl"
2
$ sudo lsof -p 7549 | grep -c "pasystem-api"
22

Is the SAK-33866 Update handlebars version in PASystem to version 4
patch also required for this fix?

On Wed, Jan 24, 2018 at 8:59 AM, Austin <[hidden email]> wrote:
Hello Matthew,

I tried your patch on my laptop with Sakai 11.4 and while 

/components/pasystem-pack/WEB-INF/lib/pasystem-impl-11.4.jar

doesn't increase after a user logs in, I am now seeing multiple instances of

/lib/pasystem-api-11.4.jar

get added each time I login.




On Wed, Jan 24, 2018 at 4:16 AM, Matthew Jones <[hidden email]> wrote:
I think my fix is good, but I'd wait at least until it's merged to master. Really if the tool still works and it doesn't load additional jars (which I'm not seeing anymore) it's good. The fix on SAK-33866 to update handlebars might be useful too. 

The timezone footer was also in this impl so yeah, having that on could also end up causing this same problem as adding a popup message.

On Tue, Jan 23, 2018 at 5:12 PM Austin <[hidden email]> wrote:
Hello Matthew,

Thanks for taking a look at this.  

Yes, we have 1500+ sessions per server.  However, we do not have any PA Banners set up at the moment, although the PA system tool does say that "timezone checking" is active.

Would the patch you included in the JIRA be safe to apply to our system right away?  Or should we wait a bit for it to be tested.

Thanks,

Austin

On Tue, Jan 23, 2018 at 11:55 AM, Matthew Jones <[hidden email]> wrote:
Hi Austin,

I'm seeing this problem locally too with master. It looks like if you set a PA Banner and then hit the site with a student it opens this impl file about 4 times. I guess if you had 1000+ sessions it could open this file quite a lot of times. It doesn't leave them open for more than a few seconds for me locally, but on an active site with a lot of sessions and PA Banner, this can certainly be a problem.

I have a possible fix, moving the templates into the api (which is in shared) that looks to fix this and still have this work.

On Tue, Jan 23, 2018 at 3:20 PM Austin <[hidden email]> wrote:
Hello All,

We hit this problem again, so I ran 

$ lsof -p <pid>

and it indicated that there are 34403 files open.  Our ulimit is set to 35K.  However the strange thing is that of the open files 30K of them are

/tomcat/components/pasystem-pack/WEB-INF/lib/pasystem-impl-11.4.jar

Does anyone know why pasystem would open so many instances of that file?

Thanks,

Austin

On Thu, Jan 18, 2018 at 9:38 PM, Jonas Eriksson <[hidden email]> wrote:

We are on 11.4 and on one of our machines “lsof” gives 2800 lines.

 

Our machines restarts once a week for OS patches.

ulimit is set to unlimited

 

tomcat 8.0.41, java 8, rhel7

 

Regards

Jonas Eriksson

Umeå University

 

Från: 'Matthew Jones' via Sakai Development [mailto:[hidden email]]
Skickat: den 18 januari 2018 23:06
Till: Austin <[hidden email]>
Kopia: sakai-dev <[hidden email]>
Ämne: Re: [sakai-dev] Too Many Open Files system errors

 

How many open files does it have? What are the files?

 

Use something like lsof -p and check https://www.cyberciti.biz/faq/howto-linux-get-list-of-open-files/ on a running process.

 

Either your process limit isn't being used correctly by that process (there's lots of articles out there about that) or there's a bug in Sakai that's keeping too many processes open. 

 

I think there was this issue fixed in 11.5 https://jira.sakaiproject.org/browse/KNL-1485

 

Nightly has ~2000 files open for the 11.x process.

 

 

 

On Thu, Jan 18, 2018 at 5:00 PM Austin <[hidden email]> wrote:

Hello Sakai Devs,

 

I think I mentioned this in another thread involving some other issues, but we hit this problem again on one of our servers today.

 

The tomcat logs will start showing errors on nearly everything with "Too many open files"

 

When we first encountered this error we increased the OS's ulmit to 32K.  But since we hit it again today, we're thinking of increasing it to 65K.  But before we do that, has anyone else encountered errors like this?  What do other folks have ulimit set to if at all?  would setting it to 'unlimited' be too high?

 

I found a couple of older threads that mention it:

 

 

but they're pretty old, so I don't know if it would still apply to 11.4.  We're running:

 

Sakai 11.4

Tomcat 8.0.47

java 8

RHEL7

 

(also, we never ran into this issue with Sakai 10.7 / RHEL6)

 

Thanks,

 

Austin

 

 

 

 

java.io.FileNotFoundException: /content01/store/content71/2016/227/20/0b09a6cf-4fde-4d73-b941-7f4fe664cecf (Too many open files)

 

java.net.SocketException: Too many open files

 

org.apache.wicket.pageStore.DiskDataStore.getFileChannel /home/sakai/tomcat/work/Catalina/localhost/profile2-tool/sakai.profile2-filestore/3167/6024/e157f028-6d67-44ce-bb77-f711437228fa/data (Too many open files)

 

org.sakaiproject.util.StorageUtils.readDocumentFromString readDocumentFromString: java.lang.IllegalStateException: java.io.FileNotFoundException: /home/sakai/tomcat/webapps/etudes-mneme-tool/WEB-INF/lib/junit-4.11.jar (Too many open files)

 

 

 

--
You received this message because you are subscribed to the Google Groups "Sakai Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/a/apereo.org/group/sakai-dev/.

--
You received this message because you are subscribed to the Google Groups "Sakai Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/a/apereo.org/group/sakai-dev/.


--
You received this message because you are subscribed to the Google Groups "Sakai Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/a/apereo.org/group/sakai-dev/.




--
You received this message because you are subscribed to the Google Groups "Sakai Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/a/apereo.org/group/sakai-dev/.