Hbase regionserver 挂掉,求分析原因

2018-11-05 20:42:14,818 INFO  [StoreOpener-e8bc4d7f29ef73a3a2916f1fcf312ca3-1] hfile.CacheConfig: Created cacheConfig for 0: blockCache=LruBlockCache{blockCount=72843, currentSize=2434868688, freeSize=128165680, maxSize=2563034368, heapSize=2434868688, minSize=2434882560, minFactor=0.95, multiSize=1217441280, multiFactor=0.5, singleSize=608720640, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false

2018-11-05 20:42:14,818 INFO  [StoreOpener-e8bc4d7f29ef73a3a2916f1fcf312ca3-1] compactions.CompactionConfiguration: size [1073741824, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 21474836480; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory

2018-11-05 20:42:15,063 INFO  [RS_OPEN_PRIORITY_REGION-st3:16020-0] regionserver.HRegion: Onlined e8bc4d7f29ef73a3a2916f1fcf312ca3; next sequenceid=5711

2018-11-05 20:42:15,171 INFO  [PostOpenDeployTasks:e8bc4d7f29ef73a3a2916f1fcf312ca3] regionserver.HRegionServer: Post open deploy tasks for IDX_TEST_TCODE,tbi_1540880625568_31523,1540884702865.e8bc4d7f29ef73a3a2916f1fcf312ca3.

2018-11-05 20:42:15,229 INFO  [PostOpenDeployTasks:e8bc4d7f29ef73a3a2916f1fcf312ca3] hbase.MetaTableAccessor: Updated row IDX_TEST_TCODE,tbi_1540880625568_31523,1540884702865.e8bc4d7f29ef73a3a2916f1fcf312ca3. with server=st3,16020,1541120487262

2018-11-05 20:42:15,259 INFO  [RpcServer.priority.FPBQ.Fifo.handler=15,queue=1,port=16020] regionserver.RSRpcServices: Close ab6eab869436e11122d9a9fe6dbd4a7d, moving to st2,16020,1541120522054

2018-11-05 20:42:15,302 INFO  [StoreCloserThread-TEST,tbi_1541415616417_20763,1541417148921.ab6eab869436e11122d9a9fe6dbd4a7d.-1] regionserver.HStore: Closed CODES

2018-11-05 20:42:15,396 INFO  [RS_CLOSE_REGION-st3:16020-2] write.TrackingParallelWriterIndexCommitter: Shutting down TrackingParallelWriterIndexCommitter

2018-11-05 20:42:15,396 INFO  [RS_CLOSE_REGION-st3:16020-2] parallel.BaseTaskRunner: Shutting down task runner because Indexer is being stopped

2018-11-05 20:42:15,407 INFO  [RS_CLOSE_REGION-st3:16020-2] client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x166d1ed8fd70341

2018-11-05 20:42:15,427 INFO  [RS_CLOSE_REGION-st3:16020-2] zookeeper.ZooKeeper: Session: 0x166d1ed8fd70341 closed

2018-11-05 20:42:15,427 INFO  [regionserver/st3/192.168.0.137:16020-longCompactions-1541120489985-EventThread] zookeeper.ClientCnxn: EventThread shut down for session: 0x166d1ed8fd70341

2018-11-05 20:42:15,497 INFO  [RS_CLOSE_REGION-st3:16020-2] client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x166d1ed8fd70342

2018-11-05 20:42:15,506 INFO  [RS_CLOSE_REGION-st3:16020-2] zookeeper.ZooKeeper: Session: 0x166d1ed8fd70342 closed

2018-11-05 20:42:15,506 INFO  [RS_CLOSE_REGION-st3:16020-2] client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x166d1ed8fd70343

2018-11-05 20:42:15,509 INFO  [st3,16020,1541120487262-index-writer--pool6-t641-EventThread] zookeeper.ClientCnxn: EventThread shut down for session: 0x166d1ed8fd70342

2018-11-05 20:42:15,541 INFO  [RS_CLOSE_REGION-st3:16020-2] zookeeper.ZooKeeper: Session: 0x166d1ed8fd70343 closed

2018-11-05 20:42:15,541 INFO  [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16020-EventThread] zookeeper.ClientCnxn: EventThread shut down for session: 0x166d1ed8fd70343

2018-11-05 20:42:15,551 INFO  [RS_CLOSE_REGION-st3:16020-2] write.TrackingParallelWriterIndexCommitter: Shutting down TrackingParallelWriterIndexCommitter

2018-11-05 20:42:15,551 INFO  [RS_CLOSE_REGION-st3:16020-2] parallel.BaseTaskRunner: Shutting down task runner because Indexer is being stopped

2018-11-05 20:42:15,557 INFO  [RS_CLOSE_REGION-st3:16020-2] regionserver.HRegion: Closed TEST,tbi_1541415616417_20763,1541417148921.ab6eab869436e11122d9a9fe6dbd4a7d.

2018-11-05 20:42:15,557 INFO  [RS_CLOSE_REGION-st3:16020-2] regionserver.HRegionServer: Adding moved region record: ab6eab869436e11122d9a9fe6dbd4a7d to st2,16020,1541120522054 as of 113873

2018-11-05 20:42:15,769 INFO  [RpcServer.priority.FPBQ.Fifo.handler=15,queue=1,port=16020] regionserver.RSRpcServices: Close ffc98e82285b336862e8f97c299f1b7b, moving to st2,16020,1541120522054

2018-11-05 20:42:15,835 INFO  [StoreCloserThread-TEST,tbi_1541419618808_45444,1541420624005.ffc98e82285b336862e8f97c299f1b7b.-1] regionserver.HStore: Closed CODES

2018-11-05 20:42:15,917 INFO  [RS_CLOSE_REGION-st3:16020-0] write.TrackingParallelWriterIndexCommitter: Shutting down TrackingParallelWriterIndexCommitter

2018-11-05 20:42:15,917 INFO  [RS_CLOSE_REGION-st3:16020-0] parallel.BaseTaskRunner: Shutting down task runner because Indexer is being stopped

2018-11-05 20:42:15,917 INFO  [RS_CLOSE_REGION-st3:16020-0] write.TrackingParallelWriterIndexCommitter: Shutting down TrackingParallelWriterIndexCommitter

2018-11-05 20:42:15,917 INFO  [RS_CLOSE_REGION-st3:16020-0] parallel.BaseTaskRunner: Shutting down task runner because Indexer is being stopped

2018-11-05 20:42:15,924 INFO  [RS_CLOSE_REGION-st3:16020-0] regionserver.HRegion: Closed TEST,tbi_1541419618808_45444,1541420624005.ffc98e82285b336862e8f97c299f1b7b.

2018-11-05 20:42:15,924 INFO  [RS_CLOSE_REGION-st3:16020-0] regionserver.HRegionServer: Adding moved region record: ffc98e82285b336862e8f97c299f1b7b to st2,16020,1541120522054 as of 126668

2018-11-05 20:42:16,180 INFO  [RpcServer.priority.FPBQ.Fifo.handler=15,queue=1,port=16020] regionserver.RSRpcServices: Close a1733a51b0c2784c06a5c15252310e42, moving to st4,16020,1541120487004

2018-11-05 20:42:16,246 INFO  [StoreCloserThread-TEST,tbi_1541148896544_1024,1541150876080.a1733a51b0c2784c06a5c15252310e42.-1] regionserver.HStore: Closed CODES

2018-11-05 20:42:16,311 INFO  [RS_CLOSE_REGION-st3:16020-1] write.TrackingParallelWriterIndexCommitter: Shutting down TrackingParallelWriterIndexCommitter

2018-11-05 20:42:16,311 INFO  [RS_CLOSE_REGION-st3:16020-1] parallel.BaseTaskRunner: Shutting down task runner because Indexer is being stopped

2018-11-05 20:42:16,311 INFO  [RS_CLOSE_REGION-st3:16020-1] write.TrackingParallelWriterIndexCommitter: Shutting down TrackingParallelWriterIndexCommitter

2018-11-05 20:42:16,311 INFO  [RS_CLOSE_REGION-st3:16020-1] parallel.BaseTaskRunner: Shutting down task runner because Indexer is being stopped

2018-11-05 20:42:16,314 INFO  [RS_CLOSE_REGION-st3:16020-1] regionserver.HRegion: Closed TEST,tbi_1541148896544_1024,1541150876080.a1733a51b0c2784c06a5c15252310e42.

2018-11-05 20:42:16,314 INFO  [RS_CLOSE_REGION-st3:16020-1] regionserver.HRegionServer: Adding moved region record: a1733a51b0c2784c06a5c15252310e42 to st4,16020,1541120487004 as of 63032

2018-11-05 20:42:16,495 INFO  [RpcServer.priority.FPBQ.Fifo.handler=15,queue=1,port=16020] regionserver.RSRpcServices: Close 95b0f7bc11570ccf2e093e42d21dae21, moving to st4,16020,1541120487004

2018-11-05 20:42:16,616 INFO  [StoreCloserThread-TEST,tbi_1541062433112_45689,1541064896360.95b0f7bc11570ccf2e093e42d21dae21.-1] regionserver.HStore: Closed CODES

2018-11-05 20:42:16,720 INFO  [RS_CLOSE_REGION-st3:16020-2] write.TrackingParallelWriterIndexCommitter: Shutting down TrackingParallelWriterIndexCommitter

2018-11-05 20:42:16,720 INFO  [RS_CLOSE_REGION-st3:16020-2] parallel.BaseTaskRunner: Shutting down task runner because Indexer is being stopped

2018-11-05 20:42:16,720 INFO  [RS_CLOSE_REGION-st3:16020-2] write.TrackingParallelWriterIndexCommitter: Shutting down TrackingParallelWriterIndexCommitter
已邀请:

hmaster - HBase 从业者

赞同来自:

看起来是协处理器之类的导致的,TrackingParallelWriterIndexCommitter/BaseTaskRunner 这个应该不是HBase 自己的

xybetter

赞同来自:

phoenix的。还有这段错误。应该是堆内存不够导致的
[RpcServer.default.FPBQ.Fifo.handler=10,queue=1,port=16020] regionserver.HRegionServer: Dump of metrics as JSON on abort: {

  "beans" : [ {

    "name" : "java.lang:type=Memory",

    "modelerType" : "sun.management.MemoryImpl",

    "ObjectPendingFinalizationCount" : 0,

    "NonHeapMemoryUsage" : {

      "committed" : 125038592,

      "init" : 2555904,

      "max" : -1,

      "used" : 123297616

    },

    "Verbose" : false,

    "HeapMemoryUsage" : {

      "committed" : 6407585792,

      "init" : 262144000,

      "max" : 6407585792,

      "used" : 5532393264

    },

    "ObjectName" : "java.lang:type=Memory"

  } ],

stony

赞同来自:

调一下jvm参数吧

xybetter

赞同来自:

谢谢

codermonkey

赞同来自:

请问这个问题解决了吗?
并没有看出是堆内存不足啊。

要回复问题请先登录注册


中国HBase技术社区微信公众号:
hbasegroup

欢迎加入HBase生态+Spark社区钉钉大群