Re: /var/log/messages doesn't have crash info when kernel gets panic/oops/crash

[Dharmosoth Seetharam <dseetharam@xxxxxxxxx>]

Thanks a lot.Seetharam
> -----Original Message-----> From: nhorman@xxxxxxxxxx> Sent: Tue, 16 Jun 2009 07:24:25 -0400> To: dseetharam@xxxxxxxxx> Subject: Re: /var/log/messages doesn't have crash info when kernel gets> panic/oops/crash> > On Mon, Jun 15, 2009 at 08:43:02PM -0800, Dharmosoth Seetharam wrote:>> Hi,>> >> I have experimented few things and results are below.>> >> Case 1) echo 0 > /proc/sys/kernel/panic_on_oops>>         When BUG/oops occurred it is able to log the diagnostics to>> /var/log/messages>>         and then continue(not switching to dump-saving kernel).>> >>         What I am looking is - I want both the things to be happen.>>         that is - it should log the appropriate messages in>> /var/log/messsages and>>         should be able to take kernel dump.>> >> case 2) In general, When panic has occurred we won't log any type of>> messages in>>         /var/log/messages ... am I right ?>>         If KEXEC is enabled and dump-saving kernel is loaded then>>         will switch to it and save the kernel dump.>> >>         I have added the dump_stack() function call in the panic()>> function>>         before calling crash_kexec with few seconds of delay. It is>> behaving>>         like randomly (ie. some time stack traces are logged in>> /var/log/messages and>>         not for rest of the time)>> >>         What I am looking is - I want to log the  appropriate messages>> in /var/log/messages>>         and then switch to dump-saving kernel.>> > Then what you want to do is, enable kdump, and set your kernel up to> panic_on_oops.  Then, configure kdump to extract the dmesg log from> /proc/vmcore, and write that to a file (either a file you create, or to> the end> of /var/log/messages).  Most distributions kdump configurations allow for> something simmilar to this.  The new config method I've got in rawhide> should> make this very easy.> > The problem is that when you panic the box (which is what has to happen> for> kdump to start), the syslog daemon doesn't get a chance to run first, so> theres> no way to write the oops backtrace to var/log/messages.  You'll need to> tell> kdump to do that for you after the fact.> >> case 3) In general, When die/nmi_die called we used to log the>> appropriate messages in /var/log/message>>         and then saving kernel dump will be depends on KEXEC enable and>> loading dump-saving kernel.>>         --- am I right ?>> >>         I have added the delay in die() function with few seconds before>> calling crash_kexec()>>         But I never seen messages logged in /var/log/messages, I can see>> those on serial console.>> > See above, when we printk those messages, klogd/syslogd needs to run to> get them> from the console buffer and write them to /var/log/messages.  Theres no> guarantee that will ever happen.> >>         Here also my intention is want both, ie. log the appropriate>> messages in /var/log/messages>>         and then take the action as per KEXEC enabled.>> >> >> >> Please advice me, What we can do to achieve those things.>> > See above, use makedumpfile to extract the dmesg log from /proc/vmcore in> kdump,> then use a custom script to append the log to /var/log/messages.  In> fedora that> will be equivalent to using the core_collector and kdump_post options in> /etc/kdump.conf.> > Neil> >> Thanks for helping me here.>> >> >> regards,>> Seetharam>> >> >> >> >> >>> -----Original Message----->>> From: nhorman@xxxxxxxxxx>>> Sent: Sat, 13 Jun 2009 14:07:36 -0400>>> To: dseetharam@xxxxxxxxx>>> Subject: Re: /var/log/messages doesn't have crash info when kernel gets>>> panic/oops/crash>>> >>> On Fri, Jun 12, 2009 at 11:01:35PM -0800, Dharmosoth Seetharam wrote:>>>> >>>> Hi Neil,>>>> >>>> Thanks for your quick reply.>>>> >>>> What you have suggested is nice when we got/save crash dump (i.e>>>> vmcore)>>>> What if we failed to save/capture vmcore info or vmcore is corrupted>>>> etc. then we dont get any information>>>> regarding the panic/crash/oops at anywhere?>>>> >>> If the vmcore is corrupted to the point where makedumpfile can't>>> extract>>> the>>> dmesg log, then its certain that syslog wouldn't have been able to>>> write>>> anything to /var/log/messages either.  As a last resort you can try run>>> strings>>> on the corrupted vmcore and see if the dmesg log is still in there>>> somewhere.>>> >>> If you forgot to capture the vmcore, theres not much more help I can>>> give>>> you,>>> Im afraid.  You'll just have to make sure you capture it next time.  If>>> you only>>> need the backtrace, and don't want the rest of the vmcore, you can make>>> kdump>>> just run makedumpfile --dump-dmesg /proc/vmcore /var/log/oops.txt>>> >>> Neil>>> >> >> >>>> >>>> >>>> >>>> Thanks and Regards>>>> >>>>  Seetharam>>>> >>>> >>>>> -----Original Message----->>>>> From: nhorman@xxxxxxxxxx>>>>> Sent: Fri, 12 Jun 2009 16:05:13 -0400>>>>> To: dseetharam@xxxxxxxxx>>>>> Subject: Re: /var/log/messages doesn't have crash info when kernel>>>>> gets>>>>> panic/oops/crash>>>>> >>>>> On Fri, Jun 12, 2009 at 01:25:49AM -0800, Dharmosoth Seetharam wrote:>>>>>> Hi,>>>>>> >>>>>> I have encounter a problem called /var/log/messages does not contain>>>>>> any>>>>>> information>>>>>> on kernel crash dump.>>>>>> >>>>>> Prior to enabling KEXEC we saw stack traces and bug info etc in>>>>>> /var/log/messages.>>>>>> After enabling KEXEC feature those info was missed.>>>>>> >>>>>> How can we get those missed info in /var/log/messages even when>>>>>> KEXEC>>>>>> feature is enabled.>>>>>> >>>>>> Any help will appreciate.>>>>>> >>>>> run crash on the resultant vmcore file that kexec can produce/save>>>>> and>>>>> get the>>>>> dmesg log out of it.>>>>> alternatively you can use makedumpfile with the --dump-dmesg log>>>>> >>>>> Neil>>>>> >>>>>> >>>>>> Thanks and Regards>>>>>> Seetharam>>>>>> >>>>>> ____________________________________________________________>>>>>> Receive Notifications of Incoming Messages>>>>>> Easily monitor multiple email accounts & access them with a click.>>>>>> Visit http://www.inbox.com/notifier and check it out!>>>>>> >>>>>> _______________________________________________>>>>>> kexec mailing list>>>>>> kexec@xxxxxxxxxxxxxxxxxxx>>>>>> http://lists.infradead.org/mailman/listinfo/kexec>>>> >>>> ____________________________________________________________>>>> GET FREE SMILEYS FOR YOUR IM & EMAIL - Learn more at>>>> http://www.inbox.com/smileys>>>> Works with AIM®, MSN® Messenger, Yahoo!® Messenger, ICQ®, Google Talk™>>>> and most webmails_______________________________________________fastboot mailing listfastboot@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx://lists.linux-foundation.org/mailman/listinfo/fastboot