System hangs

The forum for help and support with FreeNATS as well as any useful hints and tips
Post Reply
BodgeIT
Posts: 6
Joined: Mon Jun 14, 2010 10:27 am

System hangs

Post by BodgeIT » Mon Jun 14, 2010 10:40 am

Hi Dave,
Thanks for all your efforts on FreeNATS, I came accross it on a recommendation from an Automated Home article about how to convert a print server into a networked sensor. Very nifty.
Anyway I have the system setup up on my ubuntu server 9.10 running on a VIA Epia cn1000. The hard disks is a Samsung mini drive, so not very good perfromance wise however the whole system has also been running my Squeezebox server without any issues for several months.
Since installing FreeNATS, the system no longer seems to make it through the night. When checking the system itself, all the lights are on but it doesn't respoind at all on any level.

After restarting the system and checking syslog, I can see these:
Jun 14 09:35:57 stargate /etc/mysql/debian-start[1500]: Triggering myisam-recover for all MyISAM tables
Jun 14 09:35:58 stargate mysqld[1338]: 100614 9:35:58 [ERROR] /usr/sbin/mysqld: Table './freenats/fnalert' is marked as crashed and should be repaired
Jun 14 09:35:58 stargate mysqld[1338]: 100614 9:35:58 [Warning] Checking table: './freenats/fnalert'
Jun 14 09:35:58 stargate mysqld[1338]: 100614 9:35:58 [ERROR] /usr/sbin/mysqld: Table './freenats/fnalertlog' is marked as crashed and should be repaired
Jun 14 09:35:58 stargate mysqld[1338]: 100614 9:35:58 [Warning] Checking table: './freenats/fnalertlog'
Jun 14 09:35:58 stargate mysqld[1338]: 100614 9:35:58 [ERROR] /usr/sbin/mysqld: Table './freenats/fneval' is marked as crashed and should be repaired
Jun 14 09:35:58 stargate mysqld[1338]: 100614 9:35:58 [Warning] Checking table: './freenats/fneval'
Jun 14 09:35:58 stargate mysqld[1338]: 100614 9:35:58 [ERROR] /usr/sbin/mysqld: Table './freenats/fngroup' is marked as crashed and should be repaired
Jun 14 09:35:58 stargate mysqld[1338]: 100614 9:35:58 [Warning] Checking table: './freenats/fngroup'
Jun 14 09:35:58 stargate mysqld[1338]: 100614 9:35:58 [ERROR] /usr/sbin/mysqld: Table './freenats/fngrouplink' is marked as crashed and should be repaired
Jun 14 09:35:58 stargate mysqld[1338]: 100614 9:35:58 [Warning] Checking table: './freenats/fngrouplink'
Jun 14 09:35:58 stargate mysqld[1338]: 100614 9:35:58 [ERROR] /usr/sbin/mysqld: Table './freenats/fnlocaltest' is marked as crashed and should be repaired
Jun 14 09:35:58 stargate mysqld[1338]: 100614 9:35:58 [Warning] Checking table: './freenats/fnlocaltest'
Jun 14 09:35:58 stargate mysqld[1338]: 100614 9:35:58 [ERROR] /usr/sbin/mysqld: Table './freenats/fnlog' is marked as crashed and should be repaired
Jun 14 09:35:58 stargate mysqld[1338]: 100614 9:35:58 [Warning] Checking table: './freenats/fnlog'
Jun 14 09:35:58 stargate mysqld[1338]: 100614 9:35:58 [ERROR] /usr/sbin/mysqld: Table './freenats/fnnode' is marked as crashed and should be repaired
Jun 14 09:35:58 stargate mysqld[1338]: 100614 9:35:58 [Warning] Checking table: './freenats/fnnode'
Jun 14 09:35:58 stargate mysqld[1338]: 100614 9:35:58 [ERROR] /usr/sbin/mysqld: Table './freenats/fnnstest' is marked as crashed and should be repaired
Jun 14 09:35:58 stargate mysqld[1338]: 100614 9:35:58 [Warning] Checking table: './freenats/fnnstest'
Jun 14 09:35:59 stargate mysqld[1338]: 100614 9:35:59 [ERROR] /usr/sbin/mysqld: Table './freenats/fnrecord' is marked as crashed and should be repaired
Jun 14 09:35:59 stargate mysqld[1338]: 100614 9:35:59 [Warning] Checking table: './freenats/fnrecord'
Jun 14 09:35:59 stargate mysqld[1338]: 100614 9:35:59 [ERROR] /usr/sbin/mysqld: Table './freenats/fnsession' is marked as crashed and should be repaired
Jun 14 09:35:59 stargate mysqld[1338]: 100614 9:35:59 [Warning] Checking table: './freenats/fnsession'
Jun 14 09:35:59 stargate mysqld[1338]: 100614 9:35:59 [ERROR] /usr/sbin/mysqld: Table './freenats/fntestrun' is marked as crashed and should be repaired
Jun 14 09:35:59 stargate mysqld[1338]: 100614 9:35:59 [Warning] Checking table: './freenats/fntestrun'
Jun 14 09:35:59 stargate mysqld[1338]: 100614 9:35:59 [ERROR] /usr/sbin/mysqld: Table './freenats/fnview' is marked as crashed and should be repaired
Jun 14 09:35:59 stargate mysqld[1338]: 100614 9:35:59 [Warning] Checking table: './freenats/fnview'
Jun 14 09:35:59 stargate mysqld[1338]: 100614 9:35:59 [ERROR] /usr/sbin/mysqld: Table './freenats/fnviewitem' is marked as crashed and should be repaired
Jun 14 09:35:59 stargate mysqld[1338]: 100614 9:35:59 [Warning] Checking table: './freenats/fnviewitem'
Jun 14 09:36:58 stargate kernel: [ 168.736080] Clocksource tsc unstable (delta = -100162645 ns)
Jun 14 09:39:02 stargate CRON[1566]: (root) CMD ( [ -x /usr/lib/php5/maxlifetime ] && [ -d /var/lib/php5 ] && find /var/lib/php5/ -type f -cmin +$(/usr/lib/php5/maxlifetime) -print0 | xargs -n 200 -r -0 rm)
Jun 14 10:00:01 stargate CRON[1716]: (root) CMD (/etc/cron.min/freenats-tester)
Jun 14 10:09:01 stargate CRON[1805]: (root) CMD ( [ -x /usr/lib/php5/maxlifetime ] && [ -d /var/lib/php5 ] && find /var/lib/php5/ -type f -cmin +$(/usr/lib/php5/maxlifetime) -print0 | xargs -n 200 -r -0 rm)
Jun 14 10:17:01 stargate CRON[1823]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)

Not being very experienced with Linux, I need support in trying to find what is causing the system to hang and what do I need to do to repair the database if anything?

Thanks again for the product, very useful, as I start to dig deeper into what it can do, I will provide some ideas as to how it can be improved further.

Cheers
Gary

dave
Site Admin
Posts: 260
Joined: Fri May 30, 2008 9:09 pm
Location: UK
Contact:

Re: System hangs

Post by dave » Mon Jun 14, 2010 11:21 am

Hi Gary,

Glad you're finding the system useful but sorry for the hanging.

The error messages obviously imply that there is a problem or corruption with the MySQL tables. Sadly what I can't tell you is if this is a cause of a problem or a symptom!

What I would certainly suggest is repairing the tables - on a source install like yours I would probably recommend using MyRep which is able to analyse and repair the tables.

Download and explode it then...

* Stop CRON - (/etc/init.d/crond stop on Ubuntu I think)
* Wait a few minutes to make sure no freenats stuff is still running
* Use myrep as follows: php myrep.php -r -w -s [server_name_or_ip] -d [database] -u [username_prob_root] -p [password]

See what that generates.

Now if it looks like freenats is causing the system hang then maybe what I would suggest is lowering the frequency that the tester is fired from the recommended 1 minute to five or even ten minutes to see what effect that has.

Cheers,

Dave.

BodgeIT
Posts: 6
Joined: Mon Jun 14, 2010 10:27 am

Re: System hangs

Post by BodgeIT » Mon Jun 14, 2010 12:28 pm

Thanks for speedy response.
I did the myrep check and wierdly all tables are status OK.
I then checked my cron and the tester had been misconfigured to only fire ever hour, I have now set this to every 5 mins.
Anything else I can check or any other logging I could dig into that may shine light?

Cheers
Gary

dave
Site Admin
Posts: 260
Joined: Fri May 30, 2008 9:09 pm
Location: UK
Contact:

Re: System hangs

Post by dave » Mon Jun 14, 2010 12:38 pm

Hi,

Well re-looking at your logs this was mysql reporting (and fixing) the crashed errors on startup but it's always worth checking again.

Hmm... Obviously I know the timing supports freenats being somehow behind your system hanging but, especially if it was firing every hour then it seems unlikely (I hope).

What were the last logs you had on the server for the night it froze? Can you see what the last ten or twenty minutes of activity was?

Oh and also how many nodes do you have configured and are you running tester.sh or test-threaded.sh from cron?

Cheers,

Dave.

BodgeIT
Posts: 6
Joined: Mon Jun 14, 2010 10:27 am

Re: System hangs

Post by BodgeIT » Mon Jun 14, 2010 2:47 pm

Hi Dave,
Only got 5 nodes so far and 5 or 6 tests on each, some local, some node-side on windows and nix.
Having to use Webmin to view logs so haven't tracked back to the time it hung yet but I know it's happened twice, so defo seems repeatable.
Funny thing, looking at the system logs in FreeNATs, it looks like tester is being fired every second?
I'm firing tester-threaded from cron and it was set for hourly, but is now every 5 mins, not sure why the logs are showing every second.

Cheers
Gary

BodgeIT
Posts: 6
Joined: Mon Jun 14, 2010 10:27 am

Re: System hangs

Post by BodgeIT » Mon Jun 14, 2010 3:13 pm

Hi Dave,
I've found this in my Apache logs:
PHP Warning: PHP Startup: Unable to load dynamic library '/usr/lib/php5/20060613+lfs/msql.so' - /usr/lib/php5/20060613+lfs/msql.so: cannot open shared object file: No such file or directory in Unknown on line 0
[Mon Jun 14 12:34:10 2010] [notice] Apache/2.2.12 (Ubuntu) PHP/5.2.10-2ubuntu6.4 with Suhosin-Patch configured -- resuming normal operations

Any ideas?
Cheers
Gary

dave
Site Admin
Posts: 260
Joined: Fri May 30, 2008 9:09 pm
Location: UK
Contact:

Re: System hangs

Post by dave » Mon Jun 14, 2010 3:48 pm

Hi,

The msql error just refers to a shared object that can't be loaded - will just mean msql functions don't work. Shouldn't affect anything.

As for it firing every second - hmm... not sure.

Could you try the following:

cd into the server/bin directory
php logwatch.php -c 100

You can either cut+paste the output from this or if it's easier do:

php logwatch.php -c 100 > /tmp/logwatch

And then open and cut+paste the logwatch data.

This will give us the last 100 freenats log entries and may make a clue.

Likewise if you want to just email me your entire syslog as well (rather than posting it publically obviously) I'm happy to go through and see if I can see anything from around the time of the crash. I'll PM you my email in case you do.

Cheers,

Dave.

BodgeIT
Posts: 6
Joined: Mon Jun 14, 2010 10:27 am

Re: System hangs

Post by BodgeIT » Sun Jun 20, 2010 9:57 pm

Hi Dave,
Changing the script tester cron script from the threaded one to the series one seems to have made the system more stable.
Running for a couple days now and now more hanging.

If you want to try and look at what causes the issue (with a view to system improvement), please let me know.
Am happy to provide time and system to test.

Cheers
Gary

** Edit **
I spoke too soon. The system is down again.
I think I'll need to move over to a proper hard disk.

Post Reply