hbase regionserver陆续节点都挂掉

2020-04-27 14:05:48,264 INFO  [main] zookeeper.ZooKeeper: Client environment:java.library.path=:/usr/hdp/3.1.4.0-315/hadoop/lib/native/Linux-amd64-64:/usr/hdp/3.1.4.0-315/hadoop/lib/native/Linux-amd64-64:/usr/hdp/3.1.4.0-315/hadoop/lib/native
2020-04-27 14:05:48,264 INFO  [main] zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
2020-04-27 14:05:48,264 INFO  [main] zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
2020-04-27 14:05:48,264 INFO  [main] zookeeper.ZooKeeper: Client environment:os.name=Linux
2020-04-27 14:05:48,264 INFO  [main] zookeeper.ZooKeeper: Client environment:os.arch=amd64
2020-04-27 14:05:48,264 INFO  [main] zookeeper.ZooKeeper: Client environment:os.version=3.10.0-1062.el7.x86_64
2020-04-27 14:05:48,264 INFO  [main] zookeeper.ZooKeeper: Client environment:user.name=hbase
2020-04-27 14:05:48,264 INFO  [main] zookeeper.ZooKeeper: Client environment:user.home=/var/run/hbase
2020-04-27 14:05:48,264 INFO  [main] zookeeper.ZooKeeper: Client environment:user.dir=/run/hbase
2020-04-27 14:05:48,265 INFO  [main] zookeeper.ZooKeeper: Initiating client connection, connectString=node01:2181 sessionTimeout=150000 watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@27a2a089
2020-04-27 14:05:48,427 INFO  [main-SendThread(node01:2181)] zookeeper.ClientCnxn: Opening socket connection to server node01/192.168.72.111:2181. Will not attempt to authenticate using SASL (unknown error)
2020-04-27 14:05:48,441 INFO  [main-SendThread(node01:2181)] zookeeper.ClientCnxn: Socket connection established, initiating session, client: /192.168.72.112:52934, server: node01/192.168.72.111:2181
2020-04-27 14:05:48,473 INFO  [main-SendThread(node01:2181)] zookeeper.ClientCnxn: Session establishment complete on server node01/192.168.72.111:2181, sessionid = 0x171b9e774b50099, negotiated timeout = 60000
2020-04-27 14:05:49,093 INFO  [main] util.log: Logging initialized @11675ms
2020-04-27 14:05:49,444 INFO  [main] http.HttpRequestLog: Http request log for http.requests.regionserver is not defined
2020-04-27 14:05:49,530 INFO  [main] http.HttpServer: Added global filter 'safety' (class=org.apache.hadoop.hbase.http.HttpServer$QuotingInputFilter)
2020-04-27 14:05:49,530 INFO  [main] http.HttpServer: Added global filter 'clickjackingprevention' (class=org.apache.hadoop.hbase.http.ClickjackingPreventionFilter)
2020-04-27 14:05:49,533 INFO  [main] http.HttpServer: Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context regionserver
2020-04-27 14:05:49,533 INFO  [main] http.HttpServer: Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context static
2020-04-27 14:05:49,533 INFO  [main] http.HttpServer: Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs
2020-04-27 14:05:49,653 INFO  [main] http.HttpServer: Jetty bound to port 16030
2020-04-27 14:05:49,655 INFO  [main] server.Server: jetty-9.3.27.v20190418, build timestamp: 2019-04-19T02:11:38+08:00, git hash: d3e249f86955d04bc646bb620905b7c1bc596a8d
2020-04-27 14:05:50,118 INFO  [main] handler.ContextHandler: Started o.e.j.s.ServletContextHandler@38792286{/logs,file:///var/log/hbase/,AVAILABLE}
2020-04-27 14:05:50,118 INFO  [main] handler.ContextHandler: Started o.e.j.s.ServletContextHandler@665522c2{/static,file:///usr/hdp/3.1.4.0-315/hbase/hbase-webapps/static/,AVAILABLE}
2020-04-27 14:05:50,872 INFO  [main] handler.ContextHandler: Started o.e.j.w.WebAppContext@5e519ad3{/,file:///usr/hdp/3.1.4.0-315/hbase/hbase-webapps/regionserver/,AVAILABLE}{file:/usr/hdp/3.1.4.0-315/hbase/hbase-webapps/regionserver}
2020-04-27 14:05:50,912 INFO  [main] server.AbstractConnector: Started ServerConnector@6cff61fc{HTTP/1.1,[http/1.1]}{0.0.0.0:16030}
2020-04-27 14:05:50,913 INFO  [main] server.Server: Started @13495ms
2020-04-27 14:05:51,090 INFO  [regionserver/node02:16020] regionserver.HRegionServer: ClusterId : dfd6c47c-d70e-41ef-99b9-b8db6aa7d871
2020-04-27 14:05:51,408 INFO  [regionserver/node02:16020] zookeeper.ReadOnlyZKClient: Connect 0x7cb91f43 to node01:2181 with session timeout=150000ms, retries 6, retry interval 1000ms, keepAlive=60000ms
2020-04-27 14:05:51,459 INFO  [ReadOnlyZKClient-node01:2181@0x7cb91f43] zookeeper.ZooKeeper: Initiating client connection, connectString=node01:2181 sessionTimeout=150000 watcher=org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$40/1956729550@31a3c23e
2020-04-27 14:05:51,613 INFO  [ReadOnlyZKClient-node01:2181@0x7cb91f43-SendThread(node01:2181)] zookeeper.ClientCnxn: Opening socket connection to server node01/192.168.72.111:2181. Will not attempt to authenticate using SASL (unknown error)
2020-04-27 14:05:51,636 INFO  [ReadOnlyZKClient-node01:2181@0x7cb91f43-SendThread(node01:2181)] zookeeper.ClientCnxn: Socket connection established, initiating session, client: /192.168.72.112:52936, server: node01/192.168.72.111:2181
2020-04-27 14:05:51,674 INFO  [ReadOnlyZKClient-node01:2181@0x7cb91f43-SendThread(node01:2181)] zookeeper.ClientCnxn: Session establishment complete on server node01/192.168.72.111:2181, sessionid = 0x171b9e774b5009b, negotiated timeout = 60000
2020-04-27 14:05:51,945 INFO  [regionserver/node02:16020] regionserver.RegionServerCoprocessorHost: System coprocessor loading is enabled
2020-04-27 14:05:51,945 INFO  [regionserver/node02:16020] regionserver.RegionServerCoprocessorHost: Table coprocessor loading is enabled
2020-04-27 14:05:52,001 INFO  [regionserver/node02:16020] regionserver.HRegionServer: reportForDuty to master=node02,16000,1587967267739 with port=16020, startcode=1587967540214
2020-04-27 14:05:53,834 INFO  [regionserver/node02:16020] wal.WALFactory: Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider
2020-04-27 14:05:54,597 INFO  [regionserver/node02:16020] regionserver.MetricsRegionServerWrapperImpl: Computing regionserver metrics every 5000 milliseconds
2020-04-27 14:05:54,844 INFO  [regionserver/node02:16020] regionserver.MemStoreFlusher: globalMemStoreLimit=409.6 M, globalMemStoreLimitLowMark=389.1 M, Offheap=false
2020-04-27 14:05:54,876 INFO  [regionserver/node02:16020] throttle.PressureAwareCompactionThroughputController: Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms
2020-04-27 14:05:54,894 INFO  [regionserver/node02:16020] regionserver.HRegionServer: CompactionChecker runs every PT10S
2020-04-27 14:05:55,192 WARN  [regionserver/node02:16020] regionserver.SecureBulkLoadManager: hbase.bulkload.staging.dir  is deprecated. Bulkload staging directory is hdfs://node01:8020/apps/hbase/data/staging
2020-04-27 14:05:55,628 INFO  [regionserver/node02:16020] regionserver.HeapMemoryManager: Starting, tuneOn=false
2020-04-27 14:05:55,633 INFO  [regionserver/node02:16020] regionserver.ChunkCreator: Allocating data MemStoreChunkPool with chunk size 2 MB, max count 184, initial count 0
2020-04-27 14:05:55,637 INFO  [SplitLogWorker-node02:16020] regionserver.SplitLogWorker: SplitLogWorker node02,16020,1587967540214 starting
2020-04-27 14:05:55,660 INFO  [regionserver/node02:16020] regionserver.ChunkCreator: Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 204, initial count 0
2020-04-27 14:05:55,740 INFO  [SplitLogWorker-node02:16020] coordination.ZkSplitLogWorkerCoordination: worker node02,16020,1587967540214 acquired task /hbase-unsecure/splitWAL/WALs%2Fnode01%2C16020%2C1587874273263-splitting%2Fnode01%2C16020%2C1587874273263
2020-04-27 14:05:55,938 INFO  [regionserver/node02:16020] regionserver.HRegionServer: Serving as node02,16020,1587967540214, RpcServer on node02/192.168.72.112:16020, sessionid=0x171b9e774b50099
2020-04-27 14:05:55,944 INFO  [RS_LOG_REPLAY_OPS-regionserver/node02:16020-0] handler.WALSplitterHandler: Worker node02,16020,1587967540214 done with task org.apache.hadoop.hbase.coordination.ZkSplitLogWorkerCoordination$ZkSplitTaskDetails@9d70315 in 165ms. Status = null
2020-04-27 14:05:55,944 ERROR [RS_LOG_REPLAY_OPS-regionserver/node02:16020-0] executor.EventHandler: Caught throwable while processing event RS_LOG_REPLAY
java.lang.IllegalArgumentException: passed in file status is for something other than a regular file.
        at org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkArgument(Preconditions.java:142)
        at org.apache.hadoop.hbase.wal.WALSplitter.splitLogFile(WALSplitter.java:233)
        at org.apache.hadoop.hbase.wal.WALSplitter.splitLogFile(WALSplitter.java:196)
        at org.apache.hadoop.hbase.regionserver.SplitLogWorker$1.exec(SplitLogWorker.java:100)
        at org.apache.hadoop.hbase.regionserver.handler.WALSplitterHandler.process(WALSplitterHandler.java:70)
        at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
2020-04-27 14:05:55,947 ERROR [RS_LOG_REPLAY_OPS-regionserver/node02:16020-0] regionserver.HRegionServer: ***** ABORTING region server node02,16020,1587967540214: Caught throwable while processing event RS_LOG_REPLAY *****
java.lang.IllegalArgumentException: passed in file status is for something other than a regular file.
        at org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkArgument(Preconditions.java:142)
        at org.apache.hadoop.hbase.wal.WALSplitter.splitLogFile(WALSplitter.java:233)
        at org.apache.hadoop.hbase.wal.WALSplitter.splitLogFile(WALSplitter.java:196)
        at org.apache.hadoop.hbase.regionserver.SplitLogWorker$1.exec(SplitLogWorker.java:100)
        at org.apache.hadoop.hbase.regionserver.handler.WALSplitterHandler.process(WALSplitterHandler.java:70)
        at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
2020-04-27 14:05:55,947 ERROR [RS_LOG_REPLAY_OPS-regionserver/node02:16020-0] regionserver.HRegionServer: RegionServer abort: loaded coprocessors are: []
2020-04-27 14:05:55,962 INFO  [ReplicationExecutor-0] regionserver.ReplicationSourceManager: Current list of replicators: [node03,16020,1587906281947, node01,16020,1587906314664, node03,16020,1587912065482, node01,16020,1587904591616, node02,16020,1587906051398, node03,16020,1587913293344, node01,16020,1587965286333, node01,16020,1587874273263, node02,16020,1587902791456, node03,16020,1587967544193, node02,16020,1587907773841, node03,16020,1587905543846, node02,16020,1587912667857, node03,16020,1587913794820, node02,16020,1587909794879, node03,16020,1587966123311, node02,16020,1587965617849, node02,16020,1587904596728, node01,16020,1587913792836, node02,16020,1587913799055, node03,16020,1587912655424, node01,16020,1587913034584, node01,16020,1587906060230, node03,16020,1587906970452, node02,16020,1587910589679, node01,16020,1587908786981, node01,16020,1587966124758, node01,16020,1587907679390, node03,16020,1587910046160, node03,16020,1587910142445, node01,16020,1587912657185, node03,16020,1587902817997, node01,16020,1587912079051, node02,16020,1587906960473, node01,16020,1587909987733, node02,16020,1587967540214, node02,16020,1587962237879, node03,16020,1587965606095, node02,16020,1587907646153, node01,16020,1587910588895, node01,16020,1587906960785, node02,16020,1587966131290, node03,16020,1587962269670, node03,16020,1587874275962, node02,16020,1587874246606, node02,16020,1587906296723, node01,16020,1587907764058, node02,16020,1587905556519, node01,16020,1587965612548, node03,16020,1587904589652, node02,16020,1587912082406, node03,16020,1587906041299, node03,16020,1587910582205, node02,16020,1587913163747, node01,16020,1587905552876, node01,16020,1587902814353, node03,16020,1587907663899, node01,16020,1587962273630] other RSs: [node03,16020,1587967544193, node02,16020,1587967540214]
2020-04-27 14:05:55,997 INFO  [regionserver/node02:16020] quotas.RegionServerRpcQuotaManager: Quota support disabled
2020-04-27 14:05:55,997 INFO  [regionserver/node02:16020] quotas.RegionServerSpaceQuotaManager: Quota support disabled, not starting space quota manager.
2020-04-27 14:05:56,238 INFO  [RS_LOG_REPLAY_OPS-regionserver/node02:16020-0] regionserver.HRegionServer: Dump of metrics as JSON on abort: {
  "beans" : [ {
    "name" : "java.lang:type=Memory",
    "modelerType" : "sun.management.MemoryImpl",
    "Verbose" : true,
    "ObjectPendingFinalizationCount" : 0,
    "NonHeapMemoryUsage" : {
      "committed" : 62455808,
      "init" : 2555904,
      "max" : -1,
      "used" : 61058104
    },
    "HeapMemoryUsage" : {
      "committed" : 1073741824,
      "init" : 1073741824,
      "max" : 1073741824,
      "used" : 35086984
    },
    "ObjectName" : "java.lang:type=Memory"
  } ],
  "beans" : [ {
    "name" : "Hadoop:service=HBase,name=RegionServer,sub=IPC",
    "modelerType" : "RegionServer,sub=IPC",
    "tag.Context" : "regionserver",
    "tag.Hostname" : "node02",
    "queueSize" : 0,
    "numCallsInGeneralQueue" : 0,
    "numCallsInReplicationQueue" : 0,
    "numCallsInPriorityQueue" : 0,
    "numOpenConnections" : 0,
    "numActiveHandler" : 0,
    "numGeneralCallsDropped" : 0,
    "numLifoModeSwitches" : 0,
    "numCallsInWriteQueue" : 0,
    "numCallsInReadQueue" : 0,
    "numCallsInScanQueue" : 0,
    "numActiveWriteHandler" : 0,
    "numActiveReadHandler" : 0,
    "numActiveScanHandler" : 0,
    "receivedBytes" : 426,
    "exceptions.RegionMovedException" : 0,
    "exceptions.multiResponseTooLarge" : 0,
    "authenticationSuccesses" : 0,
    "authorizationFailures" : 0,
    "TotalCallTime_num_ops" : 0,
    "TotalCallTime_min" : 0,
    "TotalCallTime_max" : 0,
    "TotalCallTime_mean" : 0,
    "TotalCallTime_25th_percentile" : 0,
    "TotalCallTime_median" : 0,
    "TotalCallTime_75th_percentile" : 0,
    "TotalCallTime_90th_percentile" : 0,
    "TotalCallTime_95th_percentile" : 0,
    "TotalCallTime_98th_percentile" : 0,
    "TotalCallTime_99th_percentile" : 0,
    "TotalCallTime_99.9th_percentile" : 0,
    "exceptions.RegionTooBusyException" : 0,
    "exceptions.FailedSanityCheckException" : 0,
    "ResponseSize_num_ops" : 0,
    "ResponseSize_min" : 0,
    "ResponseSize_max" : 0,
    "ResponseSize_mean" : 0,
    "ResponseSize_25th_percentile" : 0,
    "ResponseSize_median" : 0,
    "ResponseSize_75th_percentile" : 0,
    "ResponseSize_90th_percentile" : 0,
    "ResponseSize_95th_percentile" : 0,
    "ResponseSize_98th_percentile" : 0,
    "ResponseSize_99th_percentile" : 0,
    "ResponseSize_99.9th_percentile" : 0,
    "exceptions.UnknownScannerException" : 0,
    "exceptions.OutOfOrderScannerNextException" : 0,
    "exceptions" : 1,
    "ProcessCallTime_num_ops" : 0,
    "ProcessCallTime_min" : 0,
    "ProcessCallTime_max" : 0,
    "ProcessCallTime_mean" : 0,
    "ProcessCallTime_25th_percentile" : 0,
    "ProcessCallTime_median" : 0,
    "ProcessCallTime_75th_percentile" : 0,
    "ProcessCallTime_90th_percentile" : 0,
    "ProcessCallTime_95th_percentile" : 0,
    "ProcessCallTime_98th_percentile" : 0,
    "ProcessCallTime_99th_percentile" : 0,
    "ProcessCallTime_99.9th_percentile" : 0,
    "authenticationFallbacks" : 0,
    "exceptions.NotServingRegionException" : 0,
    "exceptions.callQueueTooBig" : 0,
    "authorizationSuccesses" : 1,
    "exceptions.ScannerResetException" : 0,
    "RequestSize_num_ops" : 0,
    "RequestSize_min" : 0,
    "RequestSize_max" : 0,
    "RequestSize_mean" : 0,
    "RequestSize_25th_percentile" : 0,
    "RequestSize_median" : 0,
    "RequestSize_75th_percentile" : 0,
    "RequestSize_90th_percentile" : 0,
    "RequestSize_95th_percentile" : 0,
    "RequestSize_98th_percentile" : 0,
    "RequestSize_99th_percentile" : 0,
    "RequestSize_99.9th_percentile" : 0,
    "sentBytes" : 778,
    "QueueCallTime_num_ops" : 0,
    "QueueCallTime_min" : 0,
    "QueueCallTime_max" : 0,
    "QueueCallTime_mean" : 0,
    "QueueCallTime_25th_percentile" : 0,
    "QueueCallTime_median" : 0,
    "QueueCallTime_75th_percentile" : 0,
    "QueueCallTime_90th_percentile" : 0,
    "QueueCallTime_95th_percentile" : 0,
    "QueueCallTime_98th_percentile" : 0,
    "QueueCallTime_99th_percentile" : 0,
    "QueueCallTime_99.9th_percentile" : 0,
    "authenticationFailures" : 0
  } ],
  "beans" : [ {
    "name" : "Hadoop:service=HBase,name=RegionServer,sub=Replication",
    "modelerType" : "RegionServer,sub=Replication",
    "tag.Context" : "regionserver",
    "tag.Hostname" : "node02"
  } ],
  "beans" : [ {
    "name" : "Hadoop:service=HBase,name=RegionServer,sub=Server",
    "modelerType" : "RegionServer,sub=Server",
    "tag.Context" : "regionserver",
    "tag.Hostname" : "node02"
  } ]
}
2020-04-27 14:05:56,387 INFO  [RS_LOG_REPLAY_OPS-regionserver/node02:16020-0] regionserver.HRegionServer: ***** STOPPING region server 'node02,16020,1587967540214' *****
2020-04-27 14:05:56,387 INFO  [RS_LOG_REPLAY_OPS-regionserver/node02:16020-0] regionserver.HRegionServer: STOPPED: Caught throwable while processing event RS_LOG_REPLAY
2020-04-27 14:05:56,403 WARN  [regionserver/node02:16020] regionserver.HRegionServer: Initialize abort timeout task failed
java.lang.IllegalAccessException: Class org.apache.hadoop.hbase.regionserver.HRegionServer can not access a member of class org.apache.hadoop.hbase.regionserver.HRegionServer$SystemExitWhenAbortTimeout with modifiers "private"
        at sun.reflect.Reflection.ensureMemberAccess(Reflection.java:102)
        at java.lang.reflect.AccessibleObject.slowCheckMemberAccess(AccessibleObject.java:296)
        at java.lang.reflect.AccessibleObject.checkAccess(AccessibleObject.java:288)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:413)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1020)
        at java.lang.Thread.run(Thread.java:748)
2020-04-27 14:05:56,425 INFO  [regionserver/node02:16020] regionserver.SplitLogWorker: Sending interrupt to stop the worker thread
2020-04-27 14:05:56,426 INFO  [regionserver/node02:16020] regionserver.HRegionServer: Stopping infoServer
2020-04-27 14:05:56,426 INFO  [SplitLogWorker-node02:16020] regionserver.SplitLogWorker: SplitLogWorker interrupted. Exiting. 
2020-04-27 14:05:56,427 INFO  [SplitLogWorker-node02:16020] regionserver.SplitLogWorker: SplitLogWorker node02,16020,1587967540214 exiting
2020-04-27 14:05:56,447 INFO  [regionserver/node02:16020] handler.ContextHandler: Stopped o.e.j.w.WebAppContext@5e519ad3{/,null,UNAVAILABLE}{file:/usr/hdp/3.1.4.0-315/hbase/hbase-webapps/regionserver}
2020-04-27 14:05:56,454 INFO  [regionserver/node02:16020] server.AbstractConnector: Stopped ServerConnector@6cff61fc{HTTP/1.1,[http/1.1]}{0.0.0.0:16030}
2020-04-27 14:05:56,456 INFO  [regionserver/node02:16020] handler.ContextHandler: Stopped o.e.j.s.ServletContextHandler@665522c2{/static,file:///usr/hdp/3.1.4.0-315/hbase/hbase-webapps/static/,UNAVAILABLE}
2020-04-27 14:05:56,458 INFO  [regionserver/node02:16020] handler.ContextHandler: Stopped o.e.j.s.ServletContextHandler@38792286{/logs,file:///var/log/hbase/,UNAVAILABLE}
2020-04-27 14:05:56,473 INFO  [regionserver/node02:16020] regionserver.HeapMemoryManager: Stopping
2020-04-27 14:05:56,473 INFO  [regionserver/node02:16020] flush.RegionServerFlushTableProcedureManager: Stopping region server flush procedure manager abruptly.
2020-04-27 14:05:56,473 INFO  [regionserver/node02:16020] snapshot.RegionServerSnapshotManager: Stopping RegionServerSnapshotManager abruptly.
2020-04-27 14:05:56,473 INFO  [regionserver/node02:16020] regionserver.HRegionServer: aborting server node02,16020,1587967540214
2020-04-27 14:05:56,473 INFO  [regionserver/node02:16020] zookeeper.ReadOnlyZKClient: Close zookeeper connection 0x7cb91f43 to node01:2181
2020-04-27 14:05:56,476 INFO  [regionserver/node02:16020] regionserver.HRegionServer: stopping server node02,16020,1587967540214; all regions closed.
2020-04-27 14:05:56,477 INFO  [regionserver/node02:16020] regionserver.Leases: Closed leases
2020-04-27 14:05:56,477 INFO  [MemStoreFlusher.0] regionserver.MemStoreFlusher: MemStoreFlusher.0 exiting
2020-04-27 14:05:56,482 INFO  [MemStoreFlusher.1] regionserver.MemStoreFlusher: MemStoreFlusher.1 exiting
2020-04-27 14:05:56,495 INFO  [regionserver/node02:16020] hbase.ChoreService: Chore service for: regionserver/node02:16020 had [[ScheduledChore: Name: MovedRegionsCleaner for region node02,16020,1587967540214 Period: 120000 Unit: MILLISECONDS], [ScheduledChore: Name: CompactionThroughputTuner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: CompactedHFilesCleaner Period: 120000 Unit: MILLISECONDS], [ScheduledChore: Name: MemstoreFlusherChore Period: 10000 Unit: MILLISECONDS]] on shutdown
2020-04-27 14:05:56,495 INFO  [regionserver/node02:16020] regionserver.CompactSplit: Waiting for Split Thread to finish...
2020-04-27 14:05:56,495 INFO  [regionserver/node02:16020] regionserver.CompactSplit: Waiting for Large Compaction Thread to finish...
2020-04-27 14:05:56,495 INFO  [regionserver/node02:16020] regionserver.CompactSplit: Waiting for Small Compaction Thread to finish...
2020-04-27 14:05:56,495 INFO  [regionserver/node02:16020.logRoller] regionserver.LogRoller: LogRoller exiting.
2020-04-27 14:05:56,497 INFO  [regionserver/node02:16020] ipc.NettyRpcServer: Stopping server on /192.168.72.112:16020
2020-04-27 14:05:56,502 INFO  [ReadOnlyZKClient-node01:2181@0x7cb91f43] zookeeper.ZooKeeper: Session: 0x171b9e774b5009b closed
2020-04-27 14:05:56,504 INFO  [ReadOnlyZKClient-node01:2181@0x7cb91f43-EventThread] zookeeper.ClientCnxn: EventThread shut down
2020-04-27 14:05:56,566 INFO  [regionserver/node02:16020] zookeeper.ZooKeeper: Session: 0x171b9e774b50099 closed
2020-04-27 14:05:56,566 INFO  [regionserver/node02:16020] regionserver.HRegionServer: Exiting; stopping=node02,16020,1587967540214; zookeeper connection closed.
2020-04-27 14:05:56,567 INFO  [main-EventThread] zookeeper.ClientCnxn: EventThread shut down
2020-04-27 14:05:56,567 ERROR [main] regionserver.HRegionServerCommandLine: Region server exiting
java.lang.RuntimeException: HRegionServer Aborted
        at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.start(HRegionServerCommandLine.java:67)
        at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.run(HRegionServerCommandLine.java:87)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
        at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:149)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.main(HRegionServer.java:3021)
2020-04-27 14:05:56,684 INFO  [shutdown-hook-0] regionserver.ShutdownHook: Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@5b84f14
2020-04-27 14:05:56,685 INFO  [shutdown-hook-0] regionserver.ShutdownHook: Starting fs shutdown hook thread.
2020-04-27 14:05:56,724 INFO  [shutdown-hook-0] regionserver.ShutdownHook: Shutdown hook finished
已邀请:

justzyyan

赞同来自:

查看下gc日志有没有耗时异常的gc,同时查看下master上regionserver挂掉时间相近的日志

要回复问题请先登录注册


中国HBase技术社区微信公众号:
hbasegroup

欢迎加入HBase生态+Spark社区钉钉大群