A couple of OSD-crashes after serious network trouble

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi *,

around midnight yesterday we faced some layer-2 network problems. OSD's started to lose heartbeats and so on. Slow requests... you name it. So, after all OSD's doing their work, we had in sum around 6 of them crashed, 2 had to be restarted after first start. Should be 8 crashes in total.

Typical output:


=== 8-< ===
--- begin dump of recent events ---
-10> 2012-12-04 23:35:26.623091 7f1db7895700 5 filestore(/data/osd6-1) _do_op 0x21035870 seq 111010292 osr(65.72 0x9e13570)/0x9e13570 start -9> 2012-12-04 23:35:26.623995 7f1db7895700 5 filestore(/data/osd6-1) _do_op 0x21035500 seq 111010294 osr(10.3 0x5b5c170)/0x5b5c170 start -8> 2012-12-04 23:35:26.624013 7f1db6893700 5 --OSD::tracker-- reqid: client.290626.0:798537, seq: 151093878, time: 2012-12-04 23:35:26.624012, event: sub_op_applied, request: osd_sub_op(client.290626.0:798537 65.72 c9612472/rb.0.2d5e5.39bd39.000000000652/head//65 [] v 8084'770407 snapset=0=[]:[] snapc=0=[]) v7 -7> 2012-12-04 23:35:26.624047 7f1db8096700 5 filestore(/data/osd6-1) _do_op 0x21035c80 seq 111010293 osr(65.72 0x9e13570)/0x9e13570 start -6> 2012-12-04 23:35:26.624119 7f1db6893700 5 --OSD::tracker-- reqid: client.290626.0:798537, seq: 151093878, time: 2012-12-04 23:35:26.624119, event: done, request: osd_sub_op(client.290626.0:798537 65.72 c9612472/rb.0.2d5e5.39bd39.000000000652/head//65 [] v 8084'770407 snapset=0=[]:[] snapc=0=[]) v7 -5> 2012-12-04 23:35:26.624953 7f1db6893700 5 --OSD::tracker-- reqid: client.290626.0:798549, seq: 151093879, time: 2012-12-04 23:35:26.624953, event: sub_op_applied, request: osd_sub_op(client.290626.0:798549 65.72 c9612472/rb.0.2d5e5.39bd39.000000000652/head//65 [] v 8084'770408 snapset=0=[]:[] snapc=0=[]) v7 -4> 2012-12-04 23:35:26.625017 7f1db6893700 5 --OSD::tracker-- reqid: client.290626.0:798549, seq: 151093879, time: 2012-12-04 23:35:26.625017, event: done, request: osd_sub_op(client.290626.0:798549 65.72 c9612472/rb.0.2d5e5.39bd39.000000000652/head//65 [] v 8084'770408 snapset=0=[]:[] snapc=0=[]) v7 -3> 2012-12-04 23:35:26.626220 7f1db7895700 5 filestore(/data/osd6-1) _do_op 0x21035f00 seq 111010296 osr(6.7 0x5ca4570)/0x5ca4570 start -2> 2012-12-04 23:35:26.626218 7f1db8096700 5 filestore(/data/osd6-1) _do_op 0x21035e10 seq 111010295 osr(10.3 0x5b5c170)/0x5b5c170 start -1> 2012-12-04 23:35:26.652283 7f1daed81700 5 throttle(msgr_dispatch_throttler-cluster 0x2791560) get 1049621 (0 -> 1049621) 0> 2012-12-04 23:35:26.654669 7f1db1f89700 -1 *** Caught signal (Aborted) **
 in thread 7f1db1f89700

ceph version 0.48.2argonaut (commit:3e02b2fad88c2a95d9c0c86878f10d1beb780bfe)
 1: /usr/bin/ceph-osd() [0x6edaba]
 2: (()+0xfcb0) [0x7f1dc34c7cb0]
 3: (gsignal()+0x35) [0x7f1dc208e425]
 4: (abort()+0x17b) [0x7f1dc2091b8b]
 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f1dc29e769d]
 6: (()+0xb5846) [0x7f1dc29e5846]
 7: (()+0xb5873) [0x7f1dc29e5873]
 8: (()+0xb596e) [0x7f1dc29e596e]
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1de) [0x7a82fe]
 10: (ReplicatedPG::recover_got(hobject_t, eversion_t)+0x4ae) [0x52b5ee]
11: (ReplicatedPG::submit_push_complete(ObjectRecoveryInfo&, ObjectStore::Transaction*)+0x470) [0x52ddd0] 12: (ReplicatedPG::handle_pull_response(std::tr1::shared_ptr<OpRequest>)+0x4d4) [0x54b124] 13: (ReplicatedPG::sub_op_push(std::tr1::shared_ptr<OpRequest>)+0x98) [0x54bef8] 14: (ReplicatedPG::do_sub_op(std::tr1::shared_ptr<OpRequest>)+0x3f7) [0x54c3a7]
 15: (PG::do_request(std::tr1::shared_ptr<OpRequest>)+0x9f) [0x60073f]
 16: (OSD::dequeue_op(PG*)+0x238) [0x5bfaf8]
 17: (ThreadPool::worker()+0x4d5) [0x79f835]
 18: (ThreadPool::WorkThread::entry()+0xd) [0x5d87cd]
 19: (()+0x7e9a) [0x7f1dc34bfe9a]
 20: (clone()+0x6d) [0x7f1dc214bcbd]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- end dump of recent events ---

=== 8-< ===

A - not very scientific, but useful - aggregation of all OSD-outputs as follows. My hope is, that someone says:
"Uhm, OK, tha's fixed in ..." ;)

( count of occurences and corresponding string)

=== 8-< ===

      4 (boost::statechart::simple_state<PG::RecoveryState::Stray,
4 (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine,
     18 (ceph::__ceph_assert_fail(char
     36 (clone()+0x6d)
     18 (gsignal()+0x35)
     16 (OSD::dequeue_op(PG*)+0x238)
     16 (OSD::dequeue_op(PG*)+0x39a)
      4 (OSD::_dispatch(Message*)+0x173)
      4 (OSD::dispatch_op(std::tr1::shared_ptr<OpRequest>)+0x11b)
      4 (OSD::handle_pg_log(std::tr1::shared_ptr<OpRequest>)+0x666)
      4 (OSD::ms_dispatch(Message*)+0x184)
     16 (PG::do_request(std::tr1::shared_ptr<OpRequest>)+0x9f)
     16 (PG::do_request(std::tr1::shared_ptr<OpRequest>)+0xab)
      4 (PG::merge_log(ObjectStore::Transaction&,
      4 (PG::RecoveryState::handle_log(int,
      4 (PG::RecoveryState::Stray::react(PG::RecoveryState::MLogRec
     16 (ReplicatedPG::do_sub_op(std::tr1::shared_ptr<OpRequest>)+0x32e)
     16 (ReplicatedPG::do_sub_op(std::tr1::shared_ptr<OpRequest>)+0x3f7)
12 (ReplicatedPG::handle_pull_response(std::tr1::shared_ptr<OpRequest>)+0x4d4) 16 (ReplicatedPG::handle_pull_response(std::tr1::shared_ptr<OpRequest>)+0xb24)
      4 (ReplicatedPG::handle_push(std::tr1::shared_ptr<OpRequest>)+0x263)
     32 (ReplicatedPG::recover_got(hobject_t,
     32 (ReplicatedPG::submit_push_complete(ObjectRecoveryInfo&,
     12 (ReplicatedPG::sub_op_push(std::tr1::shared_ptr<OpRequest>)+0x98)
     16 (ReplicatedPG::sub_op_push(std::tr1::shared_ptr<OpRequest>)+0xa2)
      4 (ReplicatedPG::sub_op_push(std::tr1::shared_ptr<OpRequest>)+0xf3)
      4 (SimpleMessenger::dispatch_entry()+0x15)
      4 (SimpleMessenger::DispatchQueue::entry()+0x5e9)
      4 (SimpleMessenger::DispatchThread::entry()+0xd)
     16 (ThreadPool::worker()+0x4d5)
     16 (ThreadPool::worker()+0x76f)
     32 (ThreadPool::WorkThread::entry()+0xd)

=== 8-< ===

Everything has cleared up so far, so that's some good news ;)

Comments welcome,

Oliver.

--

Oliver Francke

filoo GmbH
Moltkestraße 25a
33330 Gütersloh
HRB4355 AG Gütersloh

Geschäftsführer: S.Grewing | J.Rehpöhler | C.Kunz

Folgen Sie uns auf Twitter: http://twitter.com/filoogmbh

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html


[Index of Archives]     [CEPH Users]     [Ceph Large]     [Information on CEPH]     [Linux BTRFS]     [Linux USB Devel]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]
  Powered by Linux