In last post of this series we learnt about how to add nodes in a cassandra cluster. If you have missed earlier posts of this series
then you can read them from below links:
2: Understanding Cassandra Read/Write Mechanism
3: Installing Cassandra on RHEL6
4: Adding Node To Cassandra Cluster
In this post we will explore nodetool and will see how we can manage a cassandra cluster using nodetool utility.
What is nodetool?
The nodetool utility is a command line interface for managing a cluster. It provides a simple command line interface to expose operations and attributes available with cassandra.
There are hundreds of options available with nodetool utility and in this post I will cover only those which ae used more often.
Note: This post is suitable for those admins who are not sevrice owners of cassandra nodes/cluster but have to interact with cassandra day in day out in their infrastructure.
Let’s begin with exploring nodetool.
1: Nodetool version: This provides the version of Cassandra running on the specified node.
[root@cassdb01 ~]# nodetool version
ReleaseVersion: 3.9
2: Nodetool status: Tis is one of the most common command which you will be using in a cassandra cluster. It provide information about the cluster, such as the state, load, and IDs. It will aslo tell you the name of datacenter where your nodes are lying and what is their state.
State ‘UN’ referes to up and normal. When a new node is added to cluster, you might see the state of node as ‘UJ’ which means node is up and now in process of joining the cluster.
Nodetool status will give you IP address of all of your nodes and also how much percentage load each node is owning. It is not neccsary that each node will own exactly the same percentage of load. For e.g in a 4 node cluster, it is not neccessary that each node owns exactly 25% of total load on cluster. One node might be owning 30% and other may be at 22% or so. But there should not be much difference in % of load being owned by each node.
[code]
[root@cassdb04 ~]# nodetool status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
— Address Load Tokens Owns (effective) Host ID Rack
UN 192.168.109.72 175.92 KiB 256 48.8% 32da4275-0b20-4805-ab3e-2067f3b2b32b rack1
UN 192.168.109.73 124.63 KiB 256 50.0% c04ac5dd-02db-420c-9933-181b99848c4f rack1
UN 192.168.109.70 298.79 KiB 256 50.8% 14ba62c6-59e4-404b-a6a6-30c9503ef3a4 rack1
UN 192.168.109.71 240.57 KiB 256 50.4% 3b19bc83-f483-4a60-82e4-109c90c49a14 rack1
[/code]
3: Nodetool info: This returns information about specific node. In output of above command you can see gossip state (whether its active or not), load on node, rack and datacenter where node is placed.
[code]
[root@cassdb04 ~]# nodetool info
ID : c04ac5dd-02db-420c-9933-181b99848c4f
Gossip active : true
Thrift active : false
Native Transport active: true
Load : 124.63 KiB
Generation No : 1484323575
Uptime (seconds) : 1285
Heap Memory (MB) : 65.46 / 1976.00
Off Heap Memory (MB) : 0.00
Data Center : datacenter1
Rack : rack1
Exceptions : 0
Key Cache : entries 11, size 888 bytes, capacity 98 MiB, 323 hits, 370 requests, 0.873 recent hit rate, 14400 save period in seconds
Row Cache : entries 0, size 0 bytes, capacity 0 bytes, 0 hits, 0 requests, NaN recent hit rate, 0 save period in seconds
Counter Cache : entries 0, size 0 bytes, capacity 49 MiB, 0 hits, 0 requests, NaN recent hit rate, 7200 save period in seconds
Chunk Cache : entries 16, size 1 MiB, capacity 462 MiB, 60 misses, 533 requests, 0.887 recent hit rate, 70.149 microseconds miss latency
Percent Repaired : 100.0%
Token : (invoke with -T/–tokens to see all 256 tokens)
[/code]
Note: To query about information of remote node, you can supply -h and -p switch with nodettol info coomand. -h needs ip/fqdn of remote node and -p is the jmx port.
[code]
[root@cassdb04 ~]# nodetool -h 192.168.109.70 -p 7199 info
ID : 14ba62c6-59e4-404b-a6a6-30c9503ef3a4
Gossip active : true
Thrift active : false
Native Transport active: true
Load : 198.57 KiB
Generation No : 1484589468
Uptime (seconds) : 165
Heap Memory (MB) : 91.97 / 1986.00
Off Heap Memory (MB) : 0.00
Data Center : datacenter1
Rack : rack1
Exceptions : 0
Key Cache : entries 17, size 1.37 KiB, capacity 99 MiB, 71 hits, 102 requests, 0.696 recent hit rate, 14400 save period in seconds
Row Cache : entries 0, size 0 bytes, capacity 0 bytes, 0 hits, 0 requests, NaN recent hit rate, 0 save period in seconds
Counter Cache : entries 0, size 0 bytes, capacity 49 MiB, 0 hits, 0 requests, NaN recent hit rate, 7200 save period in seconds
Chunk Cache : entries 12, size 768 KiB, capacity 464 MiB, 78 misses, 230 requests, 0.661 recent hit rate, 412.649 microseconds miss latency
Percent Repaired : 100.0%
Token : (invoke with -T/–tokens to see all 256 tokens)
[/code]
4: Nodetool describecluster: This command will give you name of the cassandra cluster, default partitioner which is used in cluster, type of snitch being used etc.
[code]
[root@cassdb01 ~]# nodetool describecluster
Cluster Information:
Name: Test Cluster
Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
Schema versions:
86afa796-d883-3932-aa73-6b017cef0d19: [192.168.109.72, 192.168.109.73, 192.168.109.70, 192.168.109.71]
[/code]
5: Nodetool ring: This command will tell you which node is responsible for handling which range of tokens. If you are using virtual node concept, each node will be responsible for 256 token ranges. This command will give you a very lengthy output as it will display each and every token associated with each node.
I have snipped the output for sake of this blog post.
[code]
[root@cassdb04 ~]# nodetool ring
Datacenter: datacenter1
==========
Address Rack Status State Load Owns Token
9209474870556602003
192.168.109.70 rack1 Up Normal 240.98 KiB 50.81% -9209386221367757374
192.168.109.73 rack1 Up Normal 124.63 KiB 49.99% -9194836959115518616
192.168.109.73 rack1 Up Normal 124.63 KiB 49.99% -9189566362031437022
192.168.109.71 rack1 Up Normal 240.57 KiB 50.40% -9173836129733051192
192.168.109.71 rack1 Up Normal 240.57 KiB 50.40% -9164925147537642235
192.168.109.71 rack1 Up Normal 240.57 KiB 50.40% -9140745004897827128
192.168.109.72 rack1 Up Normal 175.92 KiB 48.80% -9139635271358393037
192.168.109.73 rack1 Up Normal 124.63 KiB 49.99% -9119385776093381962
192.168.109.73 rack1 Up Normal 124.63 KiB 49.99% -9109674978522278948
192.168.109.72 rack1 Up Normal 175.92 KiB 48.80% -9091325795617772970
192.168.109.71 rack1 Up Normal 240.57 KiB 50.40% -9063930024148859956
192.168.109.71 rack1 Up Normal 240.57 KiB 50.40% -9038394199082806631
192.168.109.72 rack1 Up Normal 175.92 KiB 48.80% -9023437686068220058
192.168.109.73 rack1 Up Normal 124.63 KiB 49.99% -9021385173053652727
192.168.109.71 rack1 Up Normal 240.57 KiB 50.40% -9008429834541495946
192.168.109.70 rack1 Up Normal 240.98 KiB 50.81% -9003901886367509605
192.168.109.73 rack1 Up Normal 124.63 KiB 49.99% -8981251185746444704
192.168.109.72 rack1 Up Normal 175.92 KiB 48.80% -8976243976974462778
192.168.109.72 rack1 Up Normal 175.92 KiB 48.80% -8914749982949440380
192.168.109.71 rack1 Up Normal 240.57 KiB 50.40% -8896728810258422258
192.168.109.72 rack1 Up Normal 175.92 KiB 48.80% -8889132896797497885
192.168.109.73 rack1 Up Normal 124.63 KiB 49.99% -8883470066211443416
192.168.109.72 rack1 Up Normal 175.92 KiB 48.80% -8872886845775707512
192.168.109.72 rack1 Up Normal 175.92 KiB 48.80% -8872853960586482247
192.168.109.72 rack1 Up Normal 175.92 KiB 48.80% -8842804282688091715
192.168.109.71 rack1 Up Normal 240.57 KiB 50.40% -8836328750414937464
192.168.109.70 rack1 Up Normal 240.98 KiB 50.81% -8818194298147545683
[/code]
6: Nodetool cleanup: Nodetool cleanup is used to remove that data from a node for which it is not responsible for.
when a node auto bootstraps, it does not remove the data from the node that had previously been responsible for the data. This is so that,if the new node were to go down shortly after coming online, the data would still exist.
The command to do data cleanup is as below
[root@cassdb01 ~]# nodetool cleanup
WARN 16:54:29 Small cdc volume detected at /opt/apache-cassandra/data/cdc_raw; setting cdc_total_space_in_mb to 613. You can override this in cassandra.yaml
WARN 16:54:29 Only 52.796GiB free across all data volumes. Consider adding more capacity to your cluster or removing obsolete snapshots
Note: To remove data from a remote node, modify cleanup command as shown below
[root@cassdb01 ~]# nodetool -h 192.168.109.72 cleanup
To see what this command do, you can do a watch on nodetool status and you will see load decreasing from that node where cleanup is ran.